vibe-coding-cn/assets/skills/timescaledb/references/llms-full.md

===== PAGE: https://docs.tigerdata.com/getting-started/try-key-features-timescale-products/ =====

Try the key features in Tiger Data products

Tiger Cloud offers managed database services that provide a stable and reliable environment for your applications.

Each Tiger Cloud service is a single optimised Postgres instance extended with innovations such as TimescaleDB in the database engine, in a cloud infrastructure that delivers speed without sacrifice. A radically faster Postgres for transactional, analytical, and agentic workloads at scale.

Tiger Cloud scales Postgres to ingest and query vast amounts of live data. Tiger Cloud provides a range of features and optimizations that supercharge your queries while keeping the costs down. For example:

• The hypercore row-columnar engine in TimescaleDB makes queries up to 350x faster, ingests 44% faster, and reduces storage by 90%.
• Tiered storage in Tiger Cloud seamlessly moves your data from high performance storage for frequently accessed data to low cost bottomless storage for rarely accessed data.

The following figure shows how TimescaleDB optimizes your data for superfast real-time analytics:

This page shows you how to rapidly implement the features in Tiger Cloud that enable you to ingest and query data faster while keeping the costs low.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Optimize time-series data in hypertables with hypercore

Time-series data represents the way a system, process, or behavior changes over time. Hypertables are Postgres tables that help you improve insert and query performance by automatically partitioning your data by time. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table. You can also tune hypertables to increase performance even more.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Hypertables exist alongside regular Postgres tables. You use regular Postgres tables for relational data, and interact with hypertables and regular Postgres tables in the same way.

This section shows you how to create regular tables and hypertables, and import relational and time-series data from external files.

1. Import some time-series data into hypertables

   1. Unzip crypto_sample.zip to a <local folder>.

      This test dataset contains:

      • Second-by-second data for the most-traded crypto-assets. This time-series data is best suited for optimization in a hypertable.
      • A list of asset symbols and company names. This is best suited for a regular relational table.

      To import up to 100 GB of data directly from your current Postgres-based database, migrate with downtime using native Postgres tooling. To seamlessly import 100GB-10TB+ of data, use the live migration tooling supplied by Tiger Data. To add data from non-Postgres data sources, see Import and ingest data.

   2. Upload data into a hypertable:

      To more fully understand how to create a hypertable, how hypertables work, and how to optimize them for performance by tuning chunk intervals and enabling chunk skipping, see the hypertables documentation.

      The Tiger Cloud Console data upload creates hypertables and relational tables from the data you are uploading:

      1. In Tiger Cloud Console, select the service to add data to, then click Actions > Import data > Upload .CSV.

      2. Click to browse, or drag and drop <local folder>/tutorial_sample_tick.csv to upload.

      3. Leave the default settings for the delimiter, skipping the header, and creating a new table.

      4. In Table, provide crypto_ticks as the new table name.

      5. Enable hypertable partition for the time column and click Process CSV file.

         The upload wizard creates a hypertable containing the data from the CSV file.

      6. When the data is uploaded, close Upload .CSV.

         If you want to have a quick look at your data, press Run .

      7. Repeat the process with <local folder>/tutorial_sample_assets.csv and rename to crypto_assets.

         There is no time-series data in this table, so you don't see the hypertable partition option.

      8. In Terminal, navigate to <local folder> and connect to your service.

         psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>"
         

         You use your connection details to fill in this Postgres connection string.

      9. Create tables for the data to import:

         • For the time-series data:

           1. In your sql client, create a hypertable:

              Create a hypertable for your time-series data using CREATE TABLE. For efficient queries, remember to segmentby the column you will use most often to filter your data. For example:

              CREATE TABLE crypto_ticks (
                "time" TIMESTAMPTZ,
                symbol TEXT,
                price DOUBLE PRECISION,
                day_volume NUMERIC
              ) WITH (
                 tsdb.hypertable,
                 tsdb.partition_column='time',
                 tsdb.segmentby = 'symbol'
              );
              

              If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

         • For the relational data:

           In your sql client, create a normal Postgres table:

           CREATE TABLE crypto_assets (
            symbol TEXT NOT NULL,
            name TEXT NOT NULL
           );
           

      10. Speed up data ingestion:

          When you set timescaledb.enable_direct_compress_copy your data gets compressed in memory during ingestion with COPY statements. By writing the compressed batches immediately in the columnstore, the IO footprint is significantly lower. Also, the columnstore policy you set is less important, INSERT already produces compressed chunks.

Please note that this feature is a tech preview and not production-ready. Using this feature could lead to regressed query performance and/or storage ratio, if the ingested batches are not correctly ordered or are of too high cardinality.

To enable in-memory data compression during ingestion:

SET timescaledb.enable_direct_compress_copy=on;

Important facts

• High cardinality use cases do not produce good batches and lead to degreaded query performance.

• The columnstore is optimized to store 1000 records per batch, which is the optimal format for ingestion per segment by.

• WAL records are written for the compressed batches rather than the individual tuples.

• Currently only COPY is support, INSERT will eventually follow.

• Best results are achieved for batch ingestion with 1000 records or more, upper boundary is 10.000 records.

• Continous Aggregates are not supported at the moment.

   3. Upload the dataset to your service:
  
      ```sql
      \COPY crypto_ticks from './tutorial_sample_tick.csv' DELIMITER ',' CSV HEADER;
      ```
  
      ```sql
      \COPY crypto_assets from './tutorial_sample_assets.csv' DELIMITER ',' CSV HEADER;
      ```
  

1. Have a quick look at your data

   You query hypertables in exactly the same way as you would a relational Postgres table. Use one of the following SQL editors to run a query and see the data you uploaded:

   • Data mode: write queries, visualize data, and share your results in Tiger Cloud Console for all your Tiger Cloud services. This feature is not available under the Free pricing plan.
   • SQL editor: write, fix, and organize SQL faster and more accurately in Tiger Cloud Console for a Tiger Cloud service.
   • psql: easily run queries on your Tiger Cloud services or self-hosted TimescaleDB deployment from Terminal.

Enhance query performance for analytics

Hypercore is the TimescaleDB hybrid row-columnar storage engine, designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage. This flexibility enables TimescaleDB to deliver the best of both worlds, solving the key challenges in real-time analytics.

When TimescaleDB converts chunks from the rowstore to the columnstore, multiple records are grouped into a single row. The columns of this row hold an array-like structure that stores all the data. Because a single row takes up less disk space, you can reduce your chunk size by up to 98%, and can also speed up your queries. This helps you save on storage costs, and keeps your queries operating at lightning speed.

hypercore is enabled by default when you call CREATE TABLE. Best practice is to compress data that is no longer needed for highest performance queries, but is still accessed regularly in the columnstore. For example, yesterday's market data.

1. Add a policy to convert chunks to the columnstore at a specific time interval

   For example, yesterday's data:

   CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '1d');
   

   If you have not configured a segmentby column, TimescaleDB chooses one for you based on the data in your hypertable. For more information on how to tune your hypertables for the best performance, see efficient queries.

2. View your data space saving

   When you convert data to the columnstore, as well as being optimized for analytics, it is compressed by more than 90%. This helps you save on storage costs and keeps your queries operating at lightning speed. To see the amount of space saved, click Explorer > public > crypto_ticks.

   

Write fast and efficient analytical queries

Aggregation is a way of combing data to get insights from it. Average, sum, and count are all examples of simple aggregates. However, with large amounts of data, aggregation slows things down, quickly. Continuous aggregates are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background.

You create continuous aggregates on uncompressed data in high-performance storage. They continue to work on data in the columnstore and rarely accessed data in tiered storage. You can even create continuous aggregates on top of your continuous aggregates.

You use time buckets to create a continuous aggregate. Time buckets aggregate data in hypertables by time interval. For example, a 5-minute, 1-hour, or 3-day bucket. The data grouped in a time bucket uses a single timestamp. Continuous aggregates minimize the number of records that you need to look up to perform your query.

This section shows you how to run fast analytical queries using time buckets and continuous aggregate in Tiger Cloud Console. You can also do this using psql.

This feature is not available under the Free pricing plan.

1. Connect to your service

   In Tiger Cloud Console, select your service in the connection drop-down in the top right.

2. Create a continuous aggregate

   For a continuous aggregate, data grouped using a time bucket is stored in a Postgres MATERIALIZED VIEW in a hypertable. timescaledb.continuous ensures that this data is always up to date. In data mode, use the following code to create a continuous aggregate on the real-time data in the crypto_ticks table:

   CREATE MATERIALIZED VIEW assets_candlestick_daily
   WITH (timescaledb.continuous) AS
   SELECT
     time_bucket('1 day', "time") AS day,
     symbol,
     max(price) AS high,
     first(price, time) AS open,
     last(price, time) AS close,
     min(price) AS low
   FROM crypto_ticks srt
   GROUP BY day, symbol;
   

   This continuous aggregate creates the candlestick chart data you use to visualize the price change of an asset.

3. Create a policy to refresh the view every hour

   SELECT add_continuous_aggregate_policy('assets_candlestick_daily',
   start_offset => INTERVAL '3 weeks',
   end_offset => INTERVAL '24 hours',
   schedule_interval => INTERVAL '3 hours');
   

4. Have a quick look at your data

   You query continuous aggregates exactly the same way as your other tables. To query the assets_candlestick_daily continuous aggregate for all assets:

5. In Tiger Cloud Console, select the service you uploaded data to

6. Click Explorer > Continuous Aggregates > Create a Continuous Aggregate next to the crypto_ticks hypertable

7. Create a view called assets_candlestick_daily on the time column with an interval of 1 day, then click Next step

8. Update the view SQL with the following functions, then click Run

   CREATE MATERIALIZED VIEW assets_candlestick_daily
   WITH (timescaledb.continuous) AS
   SELECT
     time_bucket('1 day', "time") AS bucket,
     symbol,
     max(price) AS high,
     first(price, time) AS open,
     last(price, time) AS close,
     min(price) AS low
   FROM "public"."crypto_ticks" srt
   GROUP BY bucket, symbol;
   

9. When the view is created, click Next step

10. Define a refresh policy with the following values:

    • How far back do you want to materialize?: 3 weeks
    • What recent data to exclude?: 24 hours
    • How often do you want the job to run?: 3 hours

11. Click Next step, then click Run

Tiger Cloud creates the continuous aggregate and displays the aggregate ID in Tiger Cloud Console. Click DONE to close the wizard.

To see the change in terms of query time and data returned between a regular query and a continuous aggregate, run the query part of the continuous aggregate ( SELECT ...GROUP BY day, symbol; ) and compare the results.

Slash storage charges

<Availability products={['cloud']} price_plans={['enterprise', 'scale']} />

In the previous sections, you used continuous aggregates to make fast analytical queries, and hypercore to reduce storage costs on frequently accessed data. To reduce storage costs even more, you create tiering policies to move rarely accessed data to the object store. The object store is low-cost bottomless data storage built on Amazon S3. However, no matter the tier, you can query your data when you need. Tiger Cloud seamlessly accesses the correct storage tier and generates the response.

To set up data tiering:

1. Enable data tiering

   1. In Tiger Cloud Console, select the service to modify.

   2. In Explorer, click Storage configuration > Tiering storage, then click Enable tiered storage.

      

      When tiered storage is enabled, you see the amount of data in the tiered object storage.

2. Set the time interval when data is tiered

   In Tiger Cloud Console, click Data to switch to the data mode, then enable data tiering on a hypertable with the following query:

   SELECT add_tiering_policy('assets_candlestick_daily', INTERVAL '3 weeks');
   

3. Query tiered data

   You enable reads from tiered data for each query, for a session or for all future sessions. To run a single query on tiered data:

   1. Enable reads on tiered data:

   set timescaledb.enable_tiered_reads = true
   

   1. Query the data:

   SELECT * FROM crypto_ticks srt LIMIT 10
   

   1. Disable reads on tiered data:

   set timescaledb.enable_tiered_reads = false;
   

   For more information, see Querying tiered data.

Reduce the risk of downtime and data loss

<Availability products={['cloud']} price_plans={['enterprise', 'scale']} />

By default, all Tiger Cloud services have rapid recovery enabled. However, if your app has very low tolerance for downtime, Tiger Cloud offers high-availability replicas. HA replicas are exact, up-to-date copies of your database hosted in multiple AWS availability zones (AZ) within the same region as your primary node. HA replicas automatically take over operations if the original primary data node becomes unavailable. The primary node streams its write-ahead log (WAL) to the replicas to minimize the chances of data loss during failover.

1. In Tiger Cloud Console, select the service to enable replication for.

2. Click Operations, then select High availability.

3. Choose your replication strategy, then click Change configuration.

   

4. In Change high availability configuration, click Change config.

For more information, see High availability.

What next? See the use case tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API.

===== PAGE: https://docs.tigerdata.com/getting-started/start-coding-with-timescale/ =====

Start coding with Tiger Data

Easily integrate your app with Tiger Cloud or self-hosted TimescaleDB. Use your favorite programming language to connect to your Tiger Cloud service, create and manage hypertables, then ingest and query data.

"Quick Start: Ruby and TimescaleDB"

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install Rails.

Connect a Rails app to your service

Every Tiger Cloud service is a 100% Postgres database hosted in Tiger Cloud with Tiger Data extensions such as TimescaleDB. You connect to your Tiger Cloud service from a standard Rails app configured for Postgres.

1. Create a new Rails app configured for Postgres

   Rails creates and bundles your app, then installs the standard Postgres Gems.

   rails new my_app -d=postgresql
   cd my_app
   

2. Install the TimescaleDB gem

   1. Open Gemfile, add the following line, then save your changes:

      gem 'timescaledb'
      

   2. In Terminal, run the following command:

      bundle install
      

3. Connect your app to your Tiger Cloud service

   1. In <my_app_home>/config/database.yml update the configuration to read securely connect to your Tiger Cloud service by adding url: <%= ENV['DATABASE_URL'] %> to the default configuration:

      default: &default
        adapter: postgresql
        encoding: unicode
        pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
        url: <%= ENV['DATABASE_URL'] %>
      

   2. Set the environment variable for DATABASE_URL to the value of Service URL from your connection details

      export DATABASE_URL="value of Service URL"
      

   3. Create the database:

      • Tiger Cloud: nothing to do. The database is part of your Tiger Cloud service.

      • Self-hosted TimescaleDB, create the database for the project:

        rails db:create
        

   4. Run migrations:

      rails db:migrate
      

   5. Verify the connection from your app to your Tiger Cloud service:

      echo "\dx" | rails dbconsole
      

      The result shows the list of extensions in your Tiger Cloud service

   Name	Version	Schema	Description
   pg_buffercache	1.5	public	examine the shared buffer cache
   pg_stat_statements	1.11	public	track planning and execution statistics of all SQL statements executed
   plpgsql	1.0	pg_catalog	PL/pgSQL procedural language
   postgres_fdw	1.1	public	foreign-data wrapper for remote Postgres servers
   timescaledb	2.18.1	public	Enables scalable inserts and complex queries for time-series data (Community Edition)
   timescaledb_toolkit	1.19.0	public	Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities

Optimize time-series data in hypertables

Hypertables are Postgres tables designed to simplify and accelerate data analysis. Anything you can do with regular Postgres tables, you can do with hypertables - but much faster and more conveniently.

In this section, you use the helpers in the TimescaleDB gem to create and manage a hypertable.

1. Generate a migration to create the page loads table

   rails generate migration create_page_loads
   

This creates the <my_app_home>/db/migrate/<migration-datetime>_create_page_loads.rb migration file.

1. Add hypertable options

   Replace the contents of <my_app_home>/db/migrate/<migration-datetime>_create_page_loads.rb with the following:

   class CreatePageLoads < ActiveRecord::Migration[8.0]
     def change
       hypertable_options = {
         time_column: 'created_at',
         chunk_time_interval: '1 day',
         compress_segmentby: 'path',
         compress_orderby: 'created_at',
         compress_after: '7 days',
         drop_after: '30 days'
       }
   
       create_table :page_loads, id: false, primary_key: [:created_at, :user_agent, :path], hypertable: hypertable_options do |t|
         t.timestamptz :created_at, null: false
         t.string :user_agent
         t.string :path
         t.float :performance
       end
     end
   end
   

   The id column is not included in the table. This is because TimescaleDB requires that any UNIQUE or PRIMARY KEY indexes on the table include all partitioning columns. In this case, this is the time column. A new Rails model includes a PRIMARY KEY index for id by default: either remove the column or make sure that the index includes time as part of a "composite key."

   For more information, check the Roby docs around composite primary keys.

2. Create a PageLoad model

   Create a new file called <my_app_home>/app/models/page_load.rb and add the following code:

   class PageLoad < ApplicationRecord
     extend Timescaledb::ActsAsHypertable
     include Timescaledb::ContinuousAggregatesHelper
   
     acts_as_hypertable time_column: "created_at",
       segment_by: "path",
       value_column: "performance"
   
     scope :chrome_users, -> { where("user_agent LIKE ?", "%Chrome%") }
     scope :firefox_users, -> { where("user_agent LIKE ?", "%Firefox%") }
     scope :safari_users, -> { where("user_agent LIKE ?", "%Safari%") }
   
     scope :performance_stats, -> {
       select("stats_agg(#{value_column}) as stats_agg")
     }
   
     scope :slow_requests, -> { where("performance > ?", 1.0) }
     scope :fast_requests, -> { where("performance < ?", 0.1) }
   
     continuous_aggregates scopes: [:performance_stats],
       timeframes: [:minute, :hour, :day],
       refresh_policy: {
         minute: {
           start_offset: '3 minute',
           end_offset: '1 minute',
           schedule_interval: '1 minute'
         },
         hour: {
           start_offset: '3 hours',
           end_offset: '1 hour',
           schedule_interval: '1 minute'
         },
         day: {
           start_offset: '3 day',
           end_offset: '1 day',
           schedule_interval: '1 minute'
         }
       }
   end
   

3. Run the migration

   rails db:migrate
   

Insert data your service

The TimescaleDB gem provides efficient ways to insert data into hypertables. This section shows you how to ingest test data into your hypertable.

1. Create a controller to handle page loads

   Create a new file called <my_app_home>/app/controllers/application_controller.rb and add the following code:

   class ApplicationController < ActionController::Base
     around_action :track_page_load
   
     private
   
     def track_page_load
       start_time = Time.current
       yield
       end_time = Time.current
   
       PageLoad.create(
         path: request.path,
         user_agent: request.user_agent,
         performance: (end_time - start_time)
       )
     end
   end
   

2. Generate some test data

   Use bin/console to join a Rails console session and run the following code to define some random page load access data:

   def generate_sample_page_loads(total: 1000)
     time = 1.month.ago
     paths = %w[/ /about /contact /products /blog]
     browsers = [
       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0",
       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15"
     ]
   
     total.times.map do
       time = time + rand(60).seconds
       {
         path: paths.sample,
         user_agent: browsers.sample,
         performance: rand(0.1..2.0),
         created_at: time,
         updated_at: time
       }
     end
   end
   

3. Insert the generated data into your Tiger Cloud service

   PageLoad.insert_all(generate_sample_page_loads, returning: false)
   

4. Validate the test data in your Tiger Cloud service

PageLoad.count
PageLoad.first

Reference

This section lists the most common tasks you might perform with the TimescaleDB gem.

Query scopes

The TimescaleDB gem provides several convenient scopes for querying your time-series data.

• Built-in time-based scopes:

  PageLoad.last_hour.count
  PageLoad.today.count
  PageLoad.this_week.count
  PageLoad.this_month.count
  

• Browser-specific scopes:

  PageLoad.chrome_users.last_hour.count
  PageLoad.firefox_users.last_hour.count
  PageLoad.safari_users.last_hour.count
  
  PageLoad.slow_requests.last_hour.count
  PageLoad.fast_requests.last_hour.count
  

• Query continuous aggregates:

  This query fetches the average and standard deviation from the performance stats for the /products path over the last day.

  PageLoad::PerformanceStatsPerMinute.last_hour
  PageLoad::PerformanceStatsPerHour.last_day
  PageLoad::PerformanceStatsPerDay.last_month
  
  stats = PageLoad::PerformanceStatsPerHour.last_day.where(path: '/products').select("average(stats_agg) as average, stddev(stats_agg) as stddev").first
  puts "Average: #{stats.average}"
  puts "Standard Deviation: #{stats.stddev}"
  

TimescaleDB features

The TimescaleDB gem provides utility methods to access hypertable and chunk information. Every model that uses the acts_as_hypertable method has access to these methods.

Access hypertable and chunk information

• View chunk or hypertable information:

  PageLoad.chunks.count
  PageLoad.hypertable.detailed_size
  

• Compress/Decompress chunks:

  PageLoad.chunks.uncompressed.first.compress!  # Compress the first uncompressed chunk
  PageLoad.chunks.compressed.first.decompress!  # Decompress the oldest chunk
  PageLoad.hypertable.compression_stats # View compression stats
  
  

Access hypertable stats

You collect hypertable stats using methods that provide insights into your hypertable's structure, size, and compression status:

• Get basic hypertable information:

  hypertable = PageLoad.hypertable
  hypertable.hypertable_name  # The name of your hypertable
  hypertable.schema_name      # The schema where the hypertable is located
  

• Get detailed size information:

  hypertable.detailed_size # Get detailed size information for the hypertable
  hypertable.compression_stats # Get compression statistics
  hypertable.chunks_detailed_size # Get chunk information
  hypertable.approximate_row_count # Get approximate row count
  hypertable.dimensions.map(&:column_name) # Get dimension information
  hypertable.continuous_aggregates.map(&:view_name) # Get continuous aggregate view names
  

Continuous aggregates

The continuous_aggregates method generates a class for each continuous aggregate.

• Get all the continuous aggregate classes:

  PageLoad.descendants # Get all continuous aggregate classes
  

• Manually refresh a continuous aggregate:

  PageLoad.refresh_aggregates
  

• Create or drop a continuous aggregate:

  Create or drop all the continuous aggregates in the proper order to build them hierarchically. See more about how it works in this blog post.

  PageLoad.create_continuous_aggregates
  PageLoad.drop_continuous_aggregates
  

Next steps

Now that you have integrated the ruby gem into your app:

• Learn more about the TimescaleDB gem.
• Check out the official docs.
• Follow the LTTB, Open AI long-term storage, and candlesticks tutorials.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install the psycopg2 library.

For more information, see the psycopg2 documentation.

• Create a Python virtual environment.

Connect to TimescaleDB

In this section, you create a connection to TimescaleDB using the psycopg2 library. This library is one of the most popular Postgres libraries for Python. It allows you to execute raw SQL queries efficiently and safely, and prevents common attacks such as SQL injection.

1. Import the psycogpg2 library:

   import psycopg2
   

2. Locate your TimescaleDB credentials and use them to compose a connection string for psycopg2.

   You'll need:

   • password
   • username
   • host URL
   • port
   • database name

3. Compose your connection string variable as a libpq connection string, using this format:

   CONNECTION = "postgres://username:password@host:port/dbname"
   

   If you're using a hosted version of TimescaleDB, or generally require an SSL connection, use this version instead:

   CONNECTION = "postgres://username:password@host:port/dbname?sslmode=require"
   

   Alternatively you can specify each parameter in the connection string as follows

   CONNECTION = "dbname=tsdb user=tsdbadmin password=secret host=host.com port=5432 sslmode=require"
   

   This method of composing a connection string is for test or development purposes only. For production, use environment variables for sensitive details like your password, hostname, and port number.

4. Use the psycopg2 connect function to create a new database session and create a new cursor object to interact with the database.

   In your main function, add these lines:

   CONNECTION = "postgres://username:password@host:port/dbname"
   with psycopg2.connect(CONNECTION) as conn:
       cursor = conn.cursor()
       # use the cursor to interact with your database
       # cursor.execute("SELECT * FROM table")
   

   Alternatively, you can create a connection object and pass the object around as needed, like opening a cursor to perform database operations:

   CONNECTION = "postgres://username:password@host:port/dbname"
   conn = psycopg2.connect(CONNECTION)
   cursor = conn.cursor()
   # use the cursor to interact with your database
   cursor.execute("SELECT 'hello world'")
   print(cursor.fetchone())
   

Create a relational table

In this section, you create a table called sensors which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called sensor_data which holds the measurements of those sensors. The measurements contain the time, sensor_id, temperature reading, and CPU percentage of the sensors.

1. Compose a string which contains the SQL statement to create a relational table. This example creates a table called sensors, with columns id, type and location:

   query_create_sensors_table = """CREATE TABLE sensors (
                                       id SERIAL PRIMARY KEY,
                                       type VARCHAR(50),
                                       location VARCHAR(50)
                                   );
                                   """
   

2. Open a cursor, execute the query you created in the previous step, and commit the query to make the changes persistent. Afterward, close the cursor to clean up:

   cursor = conn.cursor()
   # see definition in Step 1
   cursor.execute(query_create_sensors_table)
   conn.commit()
   cursor.close()
   

Create a hypertable

When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable.

1. Create a string variable that contains the CREATE TABLE SQL statement for your hypertable. Notice how the hypertable has the compulsory time column:

   # create sensor data hypertable
   query_create_sensordata_table = """CREATE TABLE sensor_data (
                                           time TIMESTAMPTZ NOT NULL,
                                           sensor_id INTEGER,
                                           temperature DOUBLE PRECISION,
                                           cpu DOUBLE PRECISION,
                                           FOREIGN KEY (sensor_id) REFERENCES sensors (id)
                                       );
                                       """
   

2. Formulate a SELECT statement that converts the sensor_data table to a hypertable. You must specify the table name to convert to a hypertable, and the name of the time column as the two arguments. For more information, see the create_hypertable docs:

   query_create_sensordata_hypertable = "SELECT create_hypertable('sensor_data', by_range('time'));"
   

   The by_range dimension builder is an addition to TimescaleDB 2.13.

3. Open a cursor with the connection, execute the statements from the previous steps, commit your changes, and close the cursor:

   cursor = conn.cursor()
   cursor.execute(query_create_sensordata_table)
   cursor.execute(query_create_sensordata_hypertable)
   # commit changes to the database to make changes persistent
   conn.commit()
   cursor.close()
   

Insert rows of data

You can insert data into your hypertables in several different ways. In this section, you can use psycopg2 with prepared statements, or you can use pgcopy for a faster insert.

1. This example inserts a list of tuples, or relational data, called sensors, into the relational table named sensors. Open a cursor with a connection to the database, use prepared statements to formulate the INSERT SQL statement, and then execute that statement:

   sensors = [('a', 'floor'), ('a', 'ceiling'), ('b', 'floor'), ('b', 'ceiling')]
   cursor = conn.cursor()
   for sensor in sensors:
     try:
       cursor.execute("INSERT INTO sensors (type, location) VALUES (%s, %s);",
                   (sensor[0], sensor[1]))
     except (Exception, psycopg2.Error) as error:
       print(error.pgerror)
   conn.commit()
   

2. Alternatively, you can pass variables to the cursor.execute function and separate the formulation of the SQL statement, SQL, from the data being passed with it into the prepared statement, data:

   SQL = "INSERT INTO sensors (type, location) VALUES (%s, %s);"
   sensors = [('a', 'floor'), ('a', 'ceiling'), ('b', 'floor'), ('b', 'ceiling')]
   cursor = conn.cursor()
   for sensor in sensors:
     try:
       data = (sensor[0], sensor[1])
       cursor.execute(SQL, data)
     except (Exception, psycopg2.Error) as error:
       print(error.pgerror)
   conn.commit()
   

If you choose to use pgcopy instead, install the pgcopy package using pip, and then add this line to your list of import statements:

from pgcopy import CopyManager

1. Generate some random sensor data using the generate_series function provided by Postgres. This example inserts a total of 480 rows of data (4 readings, every 5 minutes, for 24 hours). In your application, this would be the query that saves your time-series data into the hypertable:

   # for sensors with ids 1-4
   for id in range(1, 4, 1):
       data = (id,)
       # create random data
       simulate_query = """SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
                               %s as sensor_id,
                               random()*100 AS temperature,
                               random() AS cpu;
                               """
       cursor.execute(simulate_query, data)
       values = cursor.fetchall()
   

2. Define the column names of the table you want to insert data into. This example uses the sensor_data hypertable created earlier. This hypertable consists of columns named time, sensor_id, temperature and cpu. The column names are defined in a list of strings called cols:

   cols = ['time', 'sensor_id', 'temperature', 'cpu']
   

3. Create an instance of the pgcopy CopyManager, mgr, and pass the connection variable, hypertable name, and list of column names. Then use the copy function of the CopyManager to insert the data into the database quickly using pgcopy.

   mgr = CopyManager(conn, 'sensor_data', cols)
   mgr.copy(values)
   

4. Commit to persist changes:

   conn.commit()
   

5. The full sample code to insert data into TimescaleDB using pgcopy, using the example of sensor data from four sensors:

   # insert using pgcopy
   def fast_insert(conn):
       cursor = conn.cursor()
   
       # for sensors with ids 1-4
       for id in range(1, 4, 1):
           data = (id,)
           # create random data
           simulate_query = """SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
                                   %s as sensor_id,
                                   random()*100 AS temperature,
                                   random() AS cpu;
                                   """
           cursor.execute(simulate_query, data)
           values = cursor.fetchall()
   
           # column names of the table you're inserting into
           cols = ['time', 'sensor_id', 'temperature', 'cpu']
   
           # create copy manager with the target table and insert
           mgr = CopyManager(conn, 'sensor_data', cols)
           mgr.copy(values)
   
       # commit after all sensor data is inserted
       # could also commit after each sensor insert is done
       conn.commit()
   

6. You can also check if the insertion worked:

   cursor.execute("SELECT * FROM sensor_data LIMIT 5;")
   print(cursor.fetchall())
   

Execute a query

This section covers how to execute queries against your database.

The first procedure shows a simple SELECT * query. For more complex queries, you can use prepared statements to ensure queries are executed safely against the database.

For more information about properly using placeholders in psycopg2, see the basic module usage document. For more information about how to execute more complex queries in psycopg2, see the psycopg2 documentation.

Execute a query

1. Define the SQL query you'd like to run on the database. This example is a simple SELECT statement querying each row from the previously created sensor_data table.

   query = "SELECT * FROM sensor_data;"
   

2. Open a cursor from the existing database connection, conn, and then execute the query you defined:

   cursor = conn.cursor()
   query = "SELECT * FROM sensor_data;"
   cursor.execute(query)
   

3. To access all resulting rows returned by your query, use one of pyscopg2's results retrieval methods, such as fetchall() or fetchmany(). This example prints the results of the query, row by row. Note that the result of fetchall() is a list of tuples, so you can handle them accordingly:

   cursor = conn.cursor()
   query = "SELECT * FROM sensor_data;"
   cursor.execute(query)
   for row in cursor.fetchall():
       print(row)
   cursor.close()
   

4. If you want a list of dictionaries instead, you can define the cursor using DictCursor:

   cursor = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
   

   Using this cursor, cursor.fetchall() returns a list of dictionary-like objects.

For more complex queries, you can use prepared statements to ensure queries are executed safely against the database.

Execute queries using prepared statements

1. Write the query using prepared statements:

   # query with placeholders
   cursor = conn.cursor()
   query = """
              SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
              FROM sensor_data
              JOIN sensors ON sensors.id = sensor_data.sensor_id
              WHERE sensors.location = %s AND sensors.type = %s
              GROUP BY five_min
              ORDER BY five_min DESC;
              """
   location = "floor"
   sensor_type = "a"
   data = (location, sensor_type)
   cursor.execute(query, data)
   results = cursor.fetchall()
   

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install Node.js.

• Install the Node.js package manager npm.

Connect to TimescaleDB

In this section, you create a connection to TimescaleDB with a common Node.js ORM (object relational mapper) called Sequelize.

1. At the command prompt, initialize a new Node.js app:

   npm init -y
   

   This creates a package.json file in your directory, which contains all of the dependencies for your project. It looks something like this:

   {
     "name": "node-sample",
     "version": "1.0.0",
     "description": "",
     "main": "index.js",
     "scripts": {
       "test": "echo \"Error: no test specified\" && exit 1"
     },
     "keywords": [],
     "author": "",
     "license": "ISC"
   }
   

2. Install Express.js:

   npm install express
   

3. Create a simple web page to check the connection. Create a new file called index.js, with this content:

   const express = require('express')
   const app = express()
   const port = 3000;
   
   app.use(express.json());
   app.get('/', (req, res) => res.send('Hello World!'))
   app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`))
   

4. Test your connection by starting the application:

   node index.js
   

In your web browser, navigate to http://localhost:3000. If the connection is successful, it shows "Hello World!"

1. Add Sequelize to your project:

   npm install sequelize sequelize-cli pg pg-hstore
   

2. Locate your TimescaleDB credentials and use them to compose a connection string for Sequelize.

   You'll need:

   • password
   • username
   • host URL
   • port
   • database name

3. Compose your connection string variable, using this format:

   'postgres://<user>:<password>@<host>:<port>/<dbname>'
   

4. Open the index.js file you created. Require Sequelize in the application, and declare the connection string:

   const Sequelize = require('sequelize')
   const sequelize = new Sequelize('postgres://<user>:<password>@<host>:<port>/<dbname>',
       {
           dialect: 'postgres',
           protocol: 'postgres',
           dialectOptions: {
               ssl: {
                   require: true,
                   rejectUnauthorized: false
               }
           }
       })
   

   Make sure you add the SSL settings in the dialectOptions sections. You can't connect to TimescaleDB using SSL without them.

5. You can test the connection by adding these lines to index.js after the app.get statement:

   sequelize.authenticate().then(() => {
       console.log('Connection has been established successfully.');
   }).catch(err => {
       console.error('Unable to connect to the database:', err);
   });
   

   Start the application on the command line:

   node index.js
   

   If the connection is successful, you'll get output like this:

   Example app listening at http://localhost:3000
   Executing (default): SELECT 1+1 AS result
   Connection has been established successfully.
   

Create a relational table

In this section, you create a relational table called page_loads.

1. Use the Sequelize command line tool to create a table and model called page_loads:

   npx sequelize model:generate --name page_loads \
   --attributes userAgent:string,time:date
   

   The output looks similar to this:

   Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11]
   
   New model was created at <PATH>.
   New migration was created at <PATH>.
   

2. Edit the migration file so that it sets up a migration key:

   'use strict';
   module.exports = {
     up: async (queryInterface, Sequelize) => {
       await queryInterface.createTable('page_loads', {
         userAgent: {
           primaryKey: true,
           type: Sequelize.STRING
         },
         time: {
           primaryKey: true,
           type: Sequelize.DATE
         }
       });
     },
     down: async (queryInterface, Sequelize) => {
       await queryInterface.dropTable('page_loads');
     }
   };
   

3. Migrate the change and make sure that it is reflected in the database:

   npx sequelize db:migrate
   

   The output looks similar to this:

   Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11]
   
   Loaded configuration file "config/config.json".
   Using environment "development".
   == 20200528195725-create-page-loads: migrating =======
   == 20200528195725-create-page-loads: migrated (0.443s)
   

4. Create the PageLoads model in your code. In the index.js file, above the app.use statement, add these lines:

   let PageLoads = sequelize.define('page_loads', {
       userAgent: {type: Sequelize.STRING, primaryKey: true },
       time: {type: Sequelize.DATE, primaryKey: true }
   }, { timestamps: false });
   

5. Instantiate a PageLoads object and save it to the database.

Create a hypertable

When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable.

1. Create a migration to modify the page_loads relational table, and change it to a hypertable by first running the following command:

   npx sequelize migration:generate --name add_hypertable
   

   The output looks similar to this:

   Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11]
   
   migrations folder at <PATH> already exists.
   New migration was created at <PATH>/20200601202912-add_hypertable.js .
   

2. In the migrations folder, there is now a new file. Open the file, and add this content:

   'use strict';
   
   module.exports = {
     up: (queryInterface, Sequelize) => {
       return queryInterface.sequelize.query("SELECT create_hypertable('page_loads', by_range('time'));");
     },
   
     down: (queryInterface, Sequelize) => {
     }
   };
   

   The by_range dimension builder is an addition to TimescaleDB 2.13.

3. At the command prompt, run the migration command:

   npx sequelize db:migrate
   

   The output looks similar to this:

   Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11]
   
   Loaded configuration file "config/config.json".
   Using environment "development".
   == 20200601202912-add_hypertable: migrating =======
   == 20200601202912-add_hypertable: migrated (0.426s)
   

Insert rows of data

This section covers how to insert data into your hypertables.

1. In the index.js file, modify the / route to get the user-agent from the request object (req) and the current timestamp. Then, call the create method on PageLoads model, supplying the user agent and timestamp parameters. The create call executes an INSERT on the database:

   app.get('/', async (req, res) => {
       // get the user agent and current time
       const userAgent = req.get('user-agent');
       const time = new Date().getTime();
   
       try {
           // insert the record
           await PageLoads.create({
               userAgent, time
           });
   
           // send response
           res.send('Inserted!');
       } catch (e) {
           console.log('Error inserting data', e)
       }
   })
   

Execute a query

This section covers how to execute queries against your database. In this example, every time the page is reloaded, all information currently in the table is displayed.

1. Modify the / route in the index.js file to call the Sequelize findAll function and retrieve all data from the page_loads table using the PageLoads model:

   app.get('/', async (req, res) => {
       // get the user agent and current time
       const userAgent = req.get('user-agent');
       const time = new Date().getTime();
   
       try {
           // insert the record
           await PageLoads.create({
               userAgent, time
           });
   
           // now display everything in the table
           const messages = await PageLoads.findAll();
           res.send(messages);
       } catch (e) {
           console.log('Error inserting data', e)
       }
   })
   

Now, when you reload the page, you should see all of the rows currently in the page_loads table.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install Go.
• Install the PGX driver for Go.

Connect to your Tiger Cloud service

In this section, you create a connection to Tiger Cloud using the PGX driver. PGX is a toolkit designed to help Go developers work directly with Postgres. You can use it to help your Go application interact directly with TimescaleDB.

1. Locate your TimescaleDB credentials and use them to compose a connection string for PGX.

   You'll need:

   • password
   • username
   • host URL
   • port number
   • database name

2. Compose your connection string variable as a libpq connection string, using this format:

   connStr := "postgres://username:password@host:port/dbname"
   

   If you're using a hosted version of TimescaleDB, or if you need an SSL connection, use this format instead:

   connStr := "postgres://username:password@host:port/dbname?sslmode=require"
   

3. You can check that you're connected to your database with this hello world program:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5"
   )
   
   //connect to database using a single connection
   func main() {
       /***********************************************/
       /* Single Connection to TimescaleDB/ PostgreSQL */
       /***********************************************/
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       conn, err := pgx.Connect(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer conn.Close(ctx)
   
       //run a simple query to check our connection
       var greeting string
       err = conn.QueryRow(ctx, "select 'Hello, Timescale!'").Scan(&greeting)
       if err != nil {
           fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err)
           os.Exit(1)
       }
       fmt.Println(greeting)
   }
   
   

   If you'd like to specify your connection string as an environment variable, you can use this syntax to access it in place of the connStr variable:

   os.Getenv("DATABASE_CONNECTION_STRING")
   

Alternatively, you can connect to TimescaleDB using a connection pool. Connection pooling is useful to conserve computing resources, and can also result in faster database queries:

1. To create a connection pool that can be used for concurrent connections to your database, use the pgxpool.New() function instead of pgx.Connect(). Also note that this script imports github.com/jackc/pgx/v5/pgxpool, instead of pgx/v5 which was used to create a single connection:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
   
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       //run a simple query to check our connection
       var greeting string
       err = dbpool.QueryRow(ctx, "select 'Hello, Tiger Data (but concurrently)'").Scan(&greeting)
       if err != nil {
           fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err)
           os.Exit(1)
       }
       fmt.Println(greeting)
   }
   

Create a relational table

In this section, you create a table called sensors which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called sensor_data which holds the measurements of those sensors. The measurements contain the time, sensor_id, temperature reading, and CPU percentage of the sensors.

1. Compose a string that contains the SQL statement to create a relational table. This example creates a table called sensors, with columns for ID, type, and location:

   queryCreateTable := `CREATE TABLE sensors (id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50));`
   

2. Execute the CREATE TABLE statement with the Exec() function on the dbpool object, using the arguments of the current context and the statement string you created:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* Create relational table                      */
       /********************************************/
   
       //Create relational table called sensors
       queryCreateTable := `CREATE TABLE sensors (id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50));`
       _, err = dbpool.Exec(ctx, queryCreateTable)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to create SENSORS table: %v\n", err)
           os.Exit(1)
       }
       fmt.Println("Successfully created relational table SENSORS")
   }
   

Generate a hypertable

When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable.

1. Create a variable for the CREATE TABLE SQL statement for your hypertable. Notice how the hypertable has the compulsory time column:

   queryCreateTable := `CREATE TABLE sensor_data (
           time TIMESTAMPTZ NOT NULL,
           sensor_id INTEGER,
           temperature DOUBLE PRECISION,
           cpu DOUBLE PRECISION,
           FOREIGN KEY (sensor_id) REFERENCES sensors (id));
           `
   

2. Formulate the SELECT statement to convert the table into a hypertable. You must specify the table name to convert to a hypertable, and its time column name as the second argument. For more information, see the create_hypertable docs:

   queryCreateHypertable := `SELECT create_hypertable('sensor_data', by_range('time'));`
   

   The by_range dimension builder is an addition to TimescaleDB 2.13.

3. Execute the CREATE TABLE statement and SELECT statement which converts the table into a hypertable. You can do this by calling the Exec() function on the dbpool object, using the arguments of the current context, and the queryCreateTable and queryCreateHypertable statement strings:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* Create Hypertable                        */
       /********************************************/
       // Create hypertable of time-series data called sensor_data
       queryCreateTable := `CREATE TABLE sensor_data (
           time TIMESTAMPTZ NOT NULL,
           sensor_id INTEGER,
           temperature DOUBLE PRECISION,
           cpu DOUBLE PRECISION,
           FOREIGN KEY (sensor_id) REFERENCES sensors (id));
           `
   
       queryCreateHypertable := `SELECT create_hypertable('sensor_data', by_range('time'));`
   
       //execute statement
       _, err = dbpool.Exec(ctx, queryCreateTable+queryCreateHypertable)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to create the `sensor_data` hypertable: %v\n", err)
           os.Exit(1)
       }
       fmt.Println("Successfully created hypertable `sensor_data`")
   }
   

Insert rows of data

You can insert rows into your database in a couple of different ways. Each of these example inserts the data from the two arrays, sensorTypes and sensorLocations, into the relational table named sensors.

The first example inserts a single row of data at a time. The second example inserts multiple rows of data. The third example uses batch inserts to speed up the process.

1. Open a connection pool to the database, then use the prepared statements to formulate an INSERT SQL statement, and execute it:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* INSERT into  relational table            */
       /********************************************/
       //Insert data into relational table
   
       // Slices of sample data to insert
       // observation i has type sensorTypes[i] and location sensorLocations[i]
       sensorTypes := []string{"a", "a", "b", "b"}
       sensorLocations := []string{"floor", "ceiling", "floor", "ceiling"}
   
       for i := range sensorTypes {
           //INSERT statement in SQL
           queryInsertMetadata := `INSERT INTO sensors (type, location) VALUES ($1, $2);`
   
           //Execute INSERT command
           _, err := dbpool.Exec(ctx, queryInsertMetadata, sensorTypes[i], sensorLocations[i])
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to insert data into database: %v\n", err)
               os.Exit(1)
           }
           fmt.Printf("Inserted sensor (%s, %s) into database \n", sensorTypes[i], sensorLocations[i])
       }
       fmt.Println("Successfully inserted all sensors into database")
   }
   

Instead of inserting a single row of data at a time, you can use this procedure to insert multiple rows of data, instead:

1. This example uses Postgres to generate some sample time-series to insert into the sensor_data hypertable. Define the SQL statement to generate the data, called queryDataGeneration. Then use the .Query() function to execute the statement and return the sample data. The data returned by the query is stored in results, a slice of structs, which is then used as a source to insert data into the hypertable:

   package main
   
   import (
       "context"
       "fmt"
       "os"
       "time"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       // Generate data to insert
   
       //SQL query to generate sample data
       queryDataGeneration := `
           SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
           floor(random() * (3) + 1)::int as sensor_id,
           random()*100 AS temperature,
           random() AS cpu
           `
       //Execute query to generate samples for sensor_data hypertable
       rows, err := dbpool.Query(ctx, queryDataGeneration)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   
       fmt.Println("Successfully generated sensor data")
   
       //Store data generated in slice results
       type result struct {
           Time        time.Time
           SensorId    int
           Temperature float64
           CPU         float64
       }
   
       var results []result
       for rows.Next() {
           var r result
           err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
               os.Exit(1)
           }
           results = append(results, r)
       }
   
       // Any errors encountered by rows.Next or rows.Scan are returned here
       if rows.Err() != nil {
           fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
           os.Exit(1)
       }
   
       // Check contents of results slice
       fmt.Println("Contents of RESULTS slice")
       for i := range results {
           var r result
           r = results[i]
           fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU)
       }
   }
   

2. Formulate an SQL insert statement for the sensor_data hypertable:

   //SQL query to generate sample data
   queryInsertTimeseriesData := `
       INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4);
       `
   

3. Execute the SQL statement for each sample in the results slice:

   //Insert contents of results slice into TimescaleDB
   for i := range results {
       var r result
       r = results[i]
       _, err := dbpool.Exec(ctx, queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to insert sample into TimescaleDB %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   }
   fmt.Println("Successfully inserted samples into sensor_data hypertable")
   

4. This example main.go generates sample data and inserts it into the sensor_data hypertable:

   package main
   
   import (
       "context"
       "fmt"
       "os"
       "time"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       /********************************************/
       /* Connect using Connection Pool            */
       /********************************************/
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* Insert data into hypertable              */
       /********************************************/
       // Generate data to insert
   
       //SQL query to generate sample data
       queryDataGeneration := `
           SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
           floor(random() * (3) + 1)::int as sensor_id,
           random()*100 AS temperature,
           random() AS cpu
           `
       //Execute query to generate samples for sensor_data hypertable
       rows, err := dbpool.Query(ctx, queryDataGeneration)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   
       fmt.Println("Successfully generated sensor data")
   
       //Store data generated in slice results
       type result struct {
           Time        time.Time
           SensorId    int
           Temperature float64
           CPU         float64
       }
       var results []result
       for rows.Next() {
           var r result
           err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
               os.Exit(1)
           }
           results = append(results, r)
       }
       // Any errors encountered by rows.Next or rows.Scan are returned here
       if rows.Err() != nil {
           fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
           os.Exit(1)
       }
   
       // Check contents of results slice
       fmt.Println("Contents of RESULTS slice")
       for i := range results {
           var r result
           r = results[i]
           fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU)
       }
   
       //Insert contents of results slice into TimescaleDB
       //SQL query to generate sample data
       queryInsertTimeseriesData := `
           INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4);
           `
   
       //Insert contents of results slice into TimescaleDB
       for i := range results {
           var r result
           r = results[i]
           _, err := dbpool.Exec(ctx, queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to insert sample into TimescaleDB %v\n", err)
               os.Exit(1)
           }
           defer rows.Close()
       }
       fmt.Println("Successfully inserted samples into sensor_data hypertable")
   }
   

Inserting multiple rows of data using this method executes as many insert statements as there are samples to be inserted. This can make ingestion of data slow. To speed up ingestion, you can batch insert data instead.

Here's a sample pattern for how to do so, using the sample data you generated in the previous procedure. It uses the pgx Batch object:

1. This example batch inserts data into the database:

   package main
   
   import (
       "context"
       "fmt"
       "os"
       "time"
   
       "github.com/jackc/pgx/v5"
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       /********************************************/
       /* Connect using Connection Pool            */
       /********************************************/
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       // Generate data to insert
   
       //SQL query to generate sample data
       queryDataGeneration := `
           SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
           floor(random() * (3) + 1)::int as sensor_id,
           random()*100 AS temperature,
           random() AS cpu
           `
   
       //Execute query to generate samples for sensor_data hypertable
       rows, err := dbpool.Query(ctx, queryDataGeneration)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   
       fmt.Println("Successfully generated sensor data")
   
       //Store data generated in slice results
       type result struct {
           Time        time.Time
           SensorId    int
           Temperature float64
           CPU         float64
       }
       var results []result
       for rows.Next() {
           var r result
           err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
               os.Exit(1)
           }
           results = append(results, r)
       }
       // Any errors encountered by rows.Next or rows.Scan are returned here
       if rows.Err() != nil {
           fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
           os.Exit(1)
       }
   
       // Check contents of results slice
       /*fmt.Println("Contents of RESULTS slice")
       for i := range results {
           var r result
           r = results[i]
           fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU)
       }*/
   
       //Insert contents of results slice into TimescaleDB
       //SQL query to generate sample data
       queryInsertTimeseriesData := `
           INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4);
           `
   
       /********************************************/
       /* Batch Insert into TimescaleDB            */
       /********************************************/
       //create batch
       batch := &pgx.Batch{}
       //load insert statements into batch queue
       for i := range results {
           var r result
           r = results[i]
           batch.Queue(queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU)
       }
       batch.Queue("select count(*) from sensor_data")
   
       //send batch to connection pool
       br := dbpool.SendBatch(ctx, batch)
       //execute statements in batch queue
       _, err = br.Exec()
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to execute statement in batch queue %v\n", err)
           os.Exit(1)
       }
       fmt.Println("Successfully batch inserted data")
   
       //Compare length of results slice to size of table
       fmt.Printf("size of results: %d\n", len(results))
       //check size of table for number of rows inserted
       // result of last SELECT statement
       var rowsInserted int
       err = br.QueryRow().Scan(&rowsInserted)
       fmt.Printf("size of table: %d\n", rowsInserted)
   
       err = br.Close()
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to closer batch %v\n", err)
           os.Exit(1)
       }
   }
   

Execute a query

This section covers how to execute queries against your database.

1. Define the SQL query you'd like to run on the database. This example uses a SQL query that combines time-series and relational data. It returns the average CPU values for every 5 minute interval, for sensors located on location ceiling and of type a:

   // Formulate query in SQL
   // Note the use of prepared statement placeholders $1 and $2
   queryTimebucketFiveMin := `
       SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
       FROM sensor_data
       JOIN sensors ON sensors.id = sensor_data.sensor_id
       WHERE sensors.location = $1 AND sensors.type = $2
       GROUP BY five_min
       ORDER BY five_min DESC;
       `
   

2. Use the .Query() function to execute the query string. Make sure you specify the relevant placeholders:

   //Execute query on TimescaleDB
   rows, err := dbpool.Query(ctx, queryTimebucketFiveMin, "ceiling", "a")
   if err != nil {
       fmt.Fprintf(os.Stderr, "Unable to execute query %v\n", err)
       os.Exit(1)
   }
   defer rows.Close()
   
   fmt.Println("Successfully executed query")
   

3. Access the rows returned by .Query(). Create a struct with fields representing the columns that you expect to be returned, then use the rows.Next() function to iterate through the rows returned and fill results with the array of structs. This uses the rows.Scan() function, passing in pointers to the fields that you want to scan for results.

   This example prints out the results returned from the query, but you might want to use those results for some other purpose. Once you've scanned through all the rows returned you can then use the results array however you like.

   //Do something with the results of query
   // Struct for results
   type result2 struct {
       Bucket time.Time
       Avg    float64
   }
   
   // Print rows returned and fill up results slice for later use
   var results []result2
   for rows.Next() {
       var r result2
       err = rows.Scan(&r.Bucket, &r.Avg)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
           os.Exit(1)
       }
       results = append(results, r)
       fmt.Printf("Time bucket: %s | Avg: %f\n", &r.Bucket, r.Avg)
   }
   
   // Any errors encountered by rows.Next or rows.Scan are returned here
   if rows.Err() != nil {
       fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
       os.Exit(1)
   }
   
   // use results here…
   

4. This example program runs a query, and accesses the results of that query:

   package main
   
   import (
       "context"
       "fmt"
       "os"
       "time"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* Execute a query                          */
       /********************************************/
   
       // Formulate query in SQL
       // Note the use of prepared statement placeholders $1 and $2
       queryTimebucketFiveMin := `
           SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
           FROM sensor_data
           JOIN sensors ON sensors.id = sensor_data.sensor_id
           WHERE sensors.location = $1 AND sensors.type = $2
           GROUP BY five_min
           ORDER BY five_min DESC;
           `
   
       //Execute query on TimescaleDB
       rows, err := dbpool.Query(ctx, queryTimebucketFiveMin, "ceiling", "a")
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to execute query %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   
       fmt.Println("Successfully executed query")
   
       //Do something with the results of query
       // Struct for results
       type result2 struct {
           Bucket time.Time
           Avg    float64
       }
   
       // Print rows returned and fill up results slice for later use
       var results []result2
       for rows.Next() {
           var r result2
           err = rows.Scan(&r.Bucket, &r.Avg)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
               os.Exit(1)
           }
           results = append(results, r)
           fmt.Printf("Time bucket: %s | Avg: %f\n", &r.Bucket, r.Avg)
       }
       // Any errors encountered by rows.Next or rows.Scan are returned here
       if rows.Err() != nil {
           fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
           os.Exit(1)
       }
   }
   

Next steps

Now that you're able to connect, read, and write to a TimescaleDB instance from your Go application, be sure to check out these advanced TimescaleDB tutorials:

• Refer to the pgx documentation for more information about pgx.
• Get up and running with TimescaleDB with the Getting Started tutorial.
• Want fast inserts on CSV data? Check out TimescaleDB parallel copy, a tool for fast inserts, written in Go.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install the Java Development Kit (JDK).

• Install the PostgreSQL JDBC driver.

All code in this quick start is for Java 16 and later. If you are working with older JDK versions, use legacy coding techniques.

Connect to your Tiger Cloud service

In this section, you create a connection to your service using an application in a single file. You can use any of your favorite build tools, including gradle or maven.

1. Create a directory containing a text file called Main.java, with this content:

   package com.timescale.java;
   
   public class Main {
   
       public static void main(String... args) {
           System.out.println("Hello, World!");
       }
   }
   

2. From the command line in the current directory, run the application:

   java Main.java
   

   If the command is successful, Hello, World! line output is printed to your console.

3. Import the PostgreSQL JDBC driver. If you are using a dependency manager, include the PostgreSQL JDBC Driver as a dependency.

4. Download the JAR artifact of the JDBC Driver and save it with the Main.java file.

5. Import the JDBC Driver into the Java application and display a list of available drivers for the check:

   package com.timescale.java;
   
   import java.sql.DriverManager;
   
   public class Main {
   
       public static void main(String... args) {
           DriverManager.drivers().forEach(System.out::println);
       }
   }
   

6. Run all the examples:

   java -cp *.jar Main.java
   

If the command is successful, a string similar to org.postgresql.Driver@7f77e91b is printed to your console. This means that you are ready to connect to TimescaleDB from Java.

1. Locate your TimescaleDB credentials and use them to compose a connection string for JDBC.

   You'll need:

   • password
   • username
   • host URL
   • port
   • database name

2. Compose your connection string variable, using this format:

   var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
   

   For more information about creating connection strings, see the JDBC documentation.

   This method of composing a connection string is for test or development purposes only. For production, use environment variables for sensitive details like your password, hostname, and port number.

   package com.timescale.java;
   
   import java.sql.DriverManager;
   import java.sql.SQLException;
   
   public class Main {
   
       public static void main(String... args) throws SQLException {
           var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
           var conn = DriverManager.getConnection(connUrl);
           System.out.println(conn.getClientInfo());
       }
   }
   

3. Run the code:

   java -cp *.jar Main.java
   

   If the command is successful, a string similar to {ApplicationName=PostgreSQL JDBC Driver} is printed to your console.

Create a relational table

In this section, you create a table called sensors which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called sensor_data which holds the measurements of those sensors. The measurements contain the time, sensor_id, temperature reading, and CPU percentage of the sensors.

1. Compose a string which contains the SQL statement to create a relational table. This example creates a table called sensors, with columns id, type and location:

   CREATE TABLE sensors (
       id SERIAL PRIMARY KEY,
       type TEXT NOT NULL,
       location TEXT NOT NULL
   );
   

2. Create a statement, execute the query you created in the previous step, and check that the table was created successfully:

   package com.timescale.java;
   
   import java.sql.DriverManager;
   import java.sql.SQLException;
   
   public class Main {
   
       public static void main(String... args) throws SQLException {
           var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
           var conn = DriverManager.getConnection(connUrl);
   
           var createSensorTableQuery = """
                   CREATE TABLE sensors (
                       id SERIAL PRIMARY KEY,
                       type TEXT NOT NULL,
                       location TEXT NOT NULL
                   )
                   """;
           try (var stmt = conn.createStatement()) {
               stmt.execute(createSensorTableQuery);
           }
   
           var showAllTablesQuery = "SELECT tablename FROM pg_catalog.pg_tables WHERE schemaname = 'public'";
           try (var stmt = conn.createStatement();
                var rs = stmt.executeQuery(showAllTablesQuery)) {
               System.out.println("Tables in the current database: ");
               while (rs.next()) {
                   System.out.println(rs.getString("tablename"));
               }
           }
       }
   }
   

Create a hypertable

When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable.

1. Create a CREATE TABLE SQL statement for your hypertable. Notice how the hypertable has the compulsory time column:

   CREATE TABLE sensor_data (
       time TIMESTAMPTZ NOT NULL,
       sensor_id INTEGER REFERENCES sensors (id),
       value DOUBLE PRECISION
   );
   

2. Create a statement, execute the query you created in the previous step:

   SELECT create_hypertable('sensor_data', by_range('time'));
   

   The by_range and by_hash dimension builder is an addition to TimescaleDB 2.13.

3. Execute the two statements you created, and commit your changes to the database:

   package com.timescale.java;
   
   import java.sql.Connection;
   import java.sql.DriverManager;
   import java.sql.SQLException;
   import java.util.List;
   
   public class Main {
   
       public static void main(String... args) {
           final var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
           try (var conn = DriverManager.getConnection(connUrl)) {
               createSchema(conn);
               insertData(conn);
           } catch (SQLException ex) {
               System.err.println(ex.getMessage());
           }
       }
   
       private static void createSchema(final Connection conn) throws SQLException {
           try (var stmt = conn.createStatement()) {
               stmt.execute("""
                       CREATE TABLE sensors (
                           id SERIAL PRIMARY KEY,
                           type TEXT NOT NULL,
                           location TEXT NOT NULL
                       )
                       """);
           }
   
           try (var stmt = conn.createStatement()) {
               stmt.execute("""
                       CREATE TABLE sensor_data (
                           time TIMESTAMPTZ NOT NULL,
                           sensor_id INTEGER REFERENCES sensors (id),
                           value DOUBLE PRECISION
                       )
                       """);
           }
   
           try (var stmt = conn.createStatement()) {
               stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))");
           }
       }
   }
   

Insert data

You can insert data into your hypertables in several different ways. In this section, you can insert single rows, or insert by batches of rows.

1. Open a connection to the database, use prepared statements to formulate the INSERT SQL statement, then execute the statement:

   final List<Sensor> sensors = List.of(
           new Sensor("temperature", "bedroom"),
           new Sensor("temperature", "living room"),
           new Sensor("temperature", "outside"),
           new Sensor("humidity", "kitchen"),
           new Sensor("humidity", "outside"));
   for (final var sensor : sensors) {
       try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) {
           stmt.setString(1, sensor.type());
           stmt.setString(2, sensor.location());
           stmt.executeUpdate();
       }
   }
   

If you want to insert a batch of rows by using a batching mechanism. In this example, you generate some sample time-series data to insert into the sensor_data hypertable:

1. Insert batches of rows:

   final var sensorDataCount = 100;
   final var insertBatchSize = 10;
   try (var stmt = conn.prepareStatement("""
           INSERT INTO sensor_data (time, sensor_id, value)
           VALUES (
               generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'),
               floor(random() * 4 + 1)::INTEGER,
               random()
           )
           """)) {
       for (int i = 0; i < sensorDataCount; i++) {
           stmt.addBatch();
   
           if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) {
               stmt.executeBatch();
           }
       }
   }
   

Execute a query

This section covers how to execute queries against your database.

Execute queries on TimescaleDB

1. Define the SQL query you'd like to run on the database. This example combines time-series and relational data. It returns the average values for every 15 minute interval for sensors with specific type and location.

   SELECT time_bucket('15 minutes', time) AS bucket, avg(value)
   FROM sensor_data
   JOIN sensors ON sensors.id = sensor_data.sensor_id
   WHERE sensors.type = ? AND sensors.location = ?
   GROUP BY bucket
   ORDER BY bucket DESC;
   

2. Execute the query with the prepared statement and read out the result set for all a-type sensors located on the floor:

   try (var stmt = conn.prepareStatement("""
           SELECT time_bucket('15 minutes', time) AS bucket, avg(value)
           FROM sensor_data
           JOIN sensors ON sensors.id = sensor_data.sensor_id
           WHERE sensors.type = ? AND sensors.location = ?
           GROUP BY bucket
           ORDER BY bucket DESC
           """)) {
       stmt.setString(1, "temperature");
       stmt.setString(2, "living room");
   
       try (var rs = stmt.executeQuery()) {
           while (rs.next()) {
               System.out.printf("%s: %f%n", rs.getTimestamp(1), rs.getDouble(2));
           }
       }
   }
   

   If the command is successful, you'll see output like this:

   2021-05-12 23:30:00.0: 0,508649
   2021-05-12 23:15:00.0: 0,477852
   2021-05-12 23:00:00.0: 0,462298
   2021-05-12 22:45:00.0: 0,457006
   2021-05-12 22:30:00.0: 0,568744
   ...
   

Next steps

Now that you're able to connect, read, and write to a TimescaleDB instance from your Java application, and generate the scaffolding necessary to build a new application from an existing TimescaleDB instance, be sure to check out these advanced TimescaleDB tutorials:

• Continuous Aggregates
• Migrate Your own Data

Complete code samples

This section contains complete code samples.

Complete code sample

package com.timescale.java;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.List;

public class Main {

    public static void main(String... args) {
        final var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
        try (var conn = DriverManager.getConnection(connUrl)) {
            createSchema(conn);
            insertData(conn);
        } catch (SQLException ex) {
            System.err.println(ex.getMessage());
        }
    }

    private static void createSchema(final Connection conn) throws SQLException {
        try (var stmt = conn.createStatement()) {
            stmt.execute("""
                    CREATE TABLE sensors (
                        id SERIAL PRIMARY KEY,
                        type TEXT NOT NULL,
                        location TEXT NOT NULL
                    )
                    """);
        }

        try (var stmt = conn.createStatement()) {
            stmt.execute("""
                    CREATE TABLE sensor_data (
                        time TIMESTAMPTZ NOT NULL,
                        sensor_id INTEGER REFERENCES sensors (id),
                        value DOUBLE PRECISION
                    )
                    """);
        }

        try (var stmt = conn.createStatement()) {
            stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))");
        }
    }

    private static void insertData(final Connection conn) throws SQLException {
        final List<Sensor> sensors = List.of(
                new Sensor("temperature", "bedroom"),
                new Sensor("temperature", "living room"),
                new Sensor("temperature", "outside"),
                new Sensor("humidity", "kitchen"),
                new Sensor("humidity", "outside"));
        for (final var sensor : sensors) {
            try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) {
                stmt.setString(1, sensor.type());
                stmt.setString(2, sensor.location());
                stmt.executeUpdate();
            }
        }

        final var sensorDataCount = 100;
        final var insertBatchSize = 10;
        try (var stmt = conn.prepareStatement("""
                INSERT INTO sensor_data (time, sensor_id, value)
                VALUES (
                    generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'),
                    floor(random() * 4 + 1)::INTEGER,
                    random()
                )
                """)) {
            for (int i = 0; i < sensorDataCount; i++) {
                stmt.addBatch();

                if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) {
                    stmt.executeBatch();
                }
            }
        }
    }

    private record Sensor(String type, String location) {
    }
}

Execute more complex queries

package com.timescale.java;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.List;

public class Main {

    public static void main(String... args) {
        final var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
        try (var conn = DriverManager.getConnection(connUrl)) {
            createSchema(conn);
            insertData(conn);
            executeQueries(conn);
        } catch (SQLException ex) {
            System.err.println(ex.getMessage());
        }
    }

    private static void createSchema(final Connection conn) throws SQLException {
        try (var stmt = conn.createStatement()) {
            stmt.execute("""
                    CREATE TABLE sensors (
                        id SERIAL PRIMARY KEY,
                        type TEXT NOT NULL,
                        location TEXT NOT NULL
                    )
                    """);
        }

        try (var stmt = conn.createStatement()) {
            stmt.execute("""
                    CREATE TABLE sensor_data (
                        time TIMESTAMPTZ NOT NULL,
                        sensor_id INTEGER REFERENCES sensors (id),
                        value DOUBLE PRECISION
                    )
                    """);
        }

        try (var stmt = conn.createStatement()) {
            stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))");
        }
    }

    private static void insertData(final Connection conn) throws SQLException {
        final List<Sensor> sensors = List.of(
                new Sensor("temperature", "bedroom"),
                new Sensor("temperature", "living room"),
                new Sensor("temperature", "outside"),
                new Sensor("humidity", "kitchen"),
                new Sensor("humidity", "outside"));
        for (final var sensor : sensors) {
            try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) {
                stmt.setString(1, sensor.type());
                stmt.setString(2, sensor.location());
                stmt.executeUpdate();
            }
        }

        final var sensorDataCount = 100;
        final var insertBatchSize = 10;
        try (var stmt = conn.prepareStatement("""
                INSERT INTO sensor_data (time, sensor_id, value)
                VALUES (
                    generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'),
                    floor(random() * 4 + 1)::INTEGER,
                    random()
                )
                """)) {
            for (int i = 0; i < sensorDataCount; i++) {
                stmt.addBatch();

                if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) {
                    stmt.executeBatch();
                }
            }
        }
    }

    private static void executeQueries(final Connection conn) throws SQLException {
        try (var stmt = conn.prepareStatement("""
                SELECT time_bucket('15 minutes', time) AS bucket, avg(value)
                FROM sensor_data
                JOIN sensors ON sensors.id = sensor_data.sensor_id
                WHERE sensors.type = ? AND sensors.location = ?
                GROUP BY bucket
                ORDER BY bucket DESC
                """)) {
            stmt.setString(1, "temperature");
            stmt.setString(2, "living room");

            try (var rs = stmt.executeQuery()) {
                while (rs.next()) {
                    System.out.printf("%s: %f%n", rs.getTimestamp(1), rs.getDouble(2));
                }
            }
        }
    }

    private record Sensor(String type, String location) {
    }
}

You are not limited to these languages. Tiger Cloud is based on Postgres, you can interface with TimescaleDB and Tiger Cloud using any Postgres client driver.

===== PAGE: https://docs.tigerdata.com/getting-started/services/ =====

Create your first Tiger Cloud service

Tiger Cloud is the modern Postgres data platform for all your applications. It enhances Postgres to handle time series, events, real-time analytics, and vector search—all in a single database alongside transactional workloads.

You get one system that handles live data ingestion, late and out-of-order updates, and low latency queries, with the performance, reliability, and scalability your app needs. Ideal for IoT, crypto, finance, SaaS, and a myriad other domains, Tiger Cloud allows you to build data-heavy, mission-critical apps while retaining the familiarity and reliability of Postgres.

What is a Tiger Cloud service?

A Tiger Cloud service is a single optimised Postgres instance extended with innovations in the database engine and cloud infrastructure to deliver speed without sacrifice. A Tiger Cloud service is 10-1000x faster at scale! It is ideal for applications requiring strong data consistency, complex relationships, and advanced querying capabilities. Get ACID compliance, extensive SQL support, JSON handling, and extensibility through custom functions, data types, and extensions.

Each service is associated with a project in Tiger Cloud. Each project can have multiple services. Each user is a member of one or more projects.

You create free and standard services in Tiger Cloud Console, depending on your pricing plan. A free service comes at zero cost and gives you limited resources to get to know Tiger Cloud. Once you are ready to try out more advanced features, you can switch to a paid plan and convert your free service to a standard one.

The Free pricing plan and services are currently in beta.

To the Postgres you know and love, Tiger Cloud adds the following capabilities:

• Standard services:

  • Real-time analytics: store and query time-series data at scale for real-time analytics and other use cases. Get faster time-based queries with hypertables, continuous aggregates, and columnar storage. Save money by compressing data into the columnstore, moving cold data to low-cost bottomless storage in Amazon S3, and deleting old data with automated policies.
  • AI-focused: build AI applications from start to scale. Get fast and accurate similarity search with the pgvector and pgvectorscale extensions.
  • Hybrid applications: get a full set of tools to develop applications that combine time-based data and AI.

  All standard Tiger Cloud services include the tooling you expect for production and developer environments: live migration, automatic backups and PITR, high availability, read replicas, data forking, connection pooling, tiered storage, usage-based storage, secure in-Tiger Cloud Console SQL editing, service metrics and insights, streamlined maintenance, and much more. Tiger Cloud continuously monitors your services and prevents common Postgres out-of-memory crashes.

• Free services:

  Postgres with TimescaleDB and vector extensions

  Free services offer limited resources and a basic feature scope, perfect to get to know Tiger Cloud in a development environment.

You manage your Tiger Cloud services and interact with your data in Tiger Cloud Console using the following modes:

Ops mode	Data mode
You use the ops mode to: Ensure data security with high availability and read replicas Save money with columnstore compression and tiered storage Enable Postgres extensions to add extra functionality Increase security using VPCs Perform day-to-day administration	Powered by PopSQL, you use the data mode to: Write queries with autocomplete Visualize data with charts and dashboards Schedule queries and dashboards for alerts or recurring reports Share queries and dashboards Interact with your data on auto-pilot with SQL assistant This feature is not available under the Free pricing plan.

To start using Tiger Cloud for your data:

1. Create a Tiger Data account: register to get access to Tiger Cloud Console as a centralized point to administer and interact with your data.
2. Create a Tiger Cloud service: that is, a Postgres database instance, powered by TimescaleDB, built for production, and extended with cloud features like transparent data tiering to object storage.
3. Connect to your Tiger Cloud service: to run queries, add and migrate your data from other sources.

Create a Tiger Data account

You create a Tiger Data account to manage your services and data in a centralized and efficient manner in Tiger Cloud Console. From there, you can create and delete services, run queries, manage access and billing, integrate other services, contact support, and more.

You create a standalone account to manage Tiger Cloud as a separate unit in your infrastructure, which includes separate billing and invoicing.

To set up Tiger Cloud:

1. Sign up for a 30-day free trial

   Open Sign up for Tiger Cloud and add your details, then click Start your free trial. You receive a confirmation email in your inbox.

2. Confirm your email address

   In the confirmation email, click the link supplied.

3. Select the pricing plan

   You are now logged into Tiger Cloud Console. You can change the pricing plan to better accommodate your growing needs on the Billing page.

To have Tiger Cloud as a part of your AWS infrastructure, you create a Tiger Data account through AWS Marketplace. In this case, Tiger Cloud is a line item in your AWS invoice.

To set up Tiger Cloud via AWS:

1. Open AWS Marketplace and search for Tiger Cloud

   You see two pricing options, pay-as-you-go and annual commit.

2. Select the pricing option that suits you and click View purchase options

3. Review and configure the purchase details, then click Subscribe

4. Click Set up your account at the top of the page

   You are redirected to Tiger Cloud Console.

5. Sign up for a 30-day free trial

   Add your details, then click Start your free trial. If you want to link an existing Tiger Data account to AWS, log in with your existing credentials.

6. Select the pricing plan

   You are now logged into Tiger Cloud Console. You can change the pricing plan later to better accommodate your growing needs on the Billing page.

7. In Confirm AWS Marketplace connection, click Connect

   Your Tiger Cloud and AWS accounts are now connected.

Create a Tiger Cloud service

Now that you have an active Tiger Data account, you create and manage your services in Tiger Cloud Console. When you create a service, you effectively create a blank Postgres database with additional Tiger Cloud features available under your pricing plan. You then add or migrate your data into this database.

To create a free or standard service:

1. In the service creation page, click + New service.

   Follow the wizard to configure your service depending on its type.

2. Click Create service.

   Your service is constructed and ready to use in a few seconds.

3. Click Download the config and store the configuration information you need to connect to this service in a secure location.

   This file contains the passwords and configuration information you need to connect to your service using the Tiger Cloud Console data mode, from the command line, or using third-party database administration tools.

If you choose to go directly to the service overview, Connect to your service shows you how to connect.

Connect to your service

To run queries and perform other operations, connect to your service:

1. Check your service is running correctly

   In Tiger Cloud Console, check that your service is marked as Running.

   

2. Connect to your service

   Connect using data mode or SQL editor in Tiger Cloud Console, or psql in the command line:

   This feature is not available under the Free pricing plan.

   1. In Tiger Cloud Console, toggle Data.

   2. Select your service in the connection drop-down in the top right.

      

   3. Run a test query:

      SELECT CURRENT_DATE;
      

      This query gives you the current date, you have successfully connected to your service.

   And that is it, you are up and running. Enjoy developing with Tiger Data.

   1. In Tiger Cloud Console, select your service.

   2. Click SQL editor.

      

   3. Run a test query:

      SELECT CURRENT_DATE;
      

      This query gives you the current date, you have successfully connected to your service.

   And that is it, you are up and running. Enjoy developing with Tiger Data.

   1. Install psql.

   2. Run the following command in the terminal using the service URL from the config file you have saved during service creation:

      psql "<your-service-url>"
      

   3. Run a test query:

      SELECT CURRENT_DATE;
      

      This query returns the current date. You have successfully connected to your service.

   And that is it, you are up and running. Enjoy developing with Tiger Data.

Quick recap. You:

• Manage your services in the ops mode in Tiger Cloud Console: add read replicas and enable high availability, compress data into the columnstore, change parameters, and so on.
• Analyze your data in the data mode in Tiger Cloud Console: write queries with autocomplete, save them in folders, share them, create charts/dashboards, and much more.
• Store configuration and security information in your config file.

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/getting-started/get-started-devops-as-code/ =====

"DevOps as code with Tiger"

Tiger Data supplies a clean, programmatic control layer for Tiger Cloud. This includes RESTful APIs and CLI commands that enable humans, machines, and AI agents easily provision, configure, and manage Tiger Cloud services programmatically.

Tiger CLI is a command-line interface that you use to manage Tiger Cloud resources including VPCs, services, read replicas, and related infrastructure. Tiger CLI calls Tiger REST API to communicate with Tiger Cloud.

This page shows you how to install and set up secure authentication for Tiger CLI, then create your first service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Data account.

Install and configure Tiger CLI

1. Install Tiger CLI

   Use the terminal to install the CLI:

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   brew install --cask timescale/tap/tiger-cli
   

   curl -fsSL https://cli.tigerdata.com | sh
   

2. Set up API credentials

   1. Log Tiger CLI into your Tiger Data account:

      tiger auth login
      

      Tiger CLI opens Console in your browser. Log in, then click Authorize.

      You can have a maximum of 10 active client credentials. If you get an error, open credentials and delete an unused credential.

   2. Select a Tiger Cloud project:

      Auth URL is: https://console.cloud.timescale.com/oauth/authorize?client_id=lotsOfURLstuff
      Opening browser for authentication...
      Select a project:
      
      > 1. Tiger Project (tgrproject)
      2. YourCompany (Company wide project) (cpnproject)
      3. YourCompany Department (dptproject)
      
      Use ↑/↓ arrows or number keys to navigate, enter to select, q to quit
      

      If only one project is associated with your account, this step is not shown.

      Where possible, Tiger CLI stores your authentication information in the system keychain/credential manager. If that fails, the credentials are stored in ~/.config/tiger/credentials with restricted file permissions (600). By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml.

3. Test your authenticated connection to Tiger Cloud by listing services

   tiger service list
   

   This call returns something like:

   • No services:

     🏜️  No services found! Your project is looking a bit empty.
     🚀 Ready to get started? Create your first service with: tiger service create
     

   • One or more services:

     ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐
     │ SERVICE ID │        NAME         │ STATUS │    TYPE     │    REGION    │     CREATED      │
     ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤
     │ tgrservice │ tiger-agent-service │ READY  │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │
     └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘
     

Create your first Tiger Cloud service

Create a new Tiger Cloud service using Tiger CLI:

1. Submit a service creation request

   By default, Tiger CLI creates a service for you that matches your pricing plan:

   • Free plan: shared CPU/memory and the time-series and ai capabilities
   • Paid plan: 0.5 CPU and 2 GB memory with the time-series capability

   tiger service create
   

   Tiger Cloud creates a Development environment for you. That is, no delete protection, high-availability, spooling or read replication. You see something like:

    🚀 Creating service 'db-11111' (auto-generated name)...
    ✅ Service creation request accepted!
    📋 Service ID: tgrservice
    🔐 Password saved to system keyring for automatic authentication
    🎯 Set service 'tgrservice' as default service.
    ⏳ Waiting for service to be ready (wait timeout: 30m0s)...
    🎉 Service is ready and running!
   🔌 Run 'tiger db connect' to connect to your new service
   ┌───────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
   │     PROPERTY      │                                              VALUE                                               │
   ├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ Service ID        │ tgrservice                                                                                       │
   │ Name              │ db-11111                                                                                         │
   │ Status            │ READY                                                                                            │
   │ Type              │ TIMESCALEDB                                                                                      │
   │ Region            │ us-east-1                                                                                        │
   │ CPU               │ 0.5 cores (500m)                                                                                 │
   │ Memory            │ 2 GB                                                                                             │
   │ Direct Endpoint   │ tgrservice.tgrproject.tsdb.cloud.timescale.com:39004                                             │
   │ Created           │ 2025-10-20 20:33:46 UTC                                                                          │
   │ Connection String │ postgresql://tsdbadmin@tgrservice.tgrproject.tsdb.cloud.timescale.com:0007/tsdb?sslmode=require │
   │ Console URL       │ https://console.cloud.timescale.com/dashboard/services/tgrservice                                │
   └───────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
   

   This service is set as default by the CLI.

2. Check the CLI configuration

   tiger config show
   

   You see something like:

   api_url:     https://console.cloud.timescale.com/public/api/v1
   console_url: https://console.cloud.timescale.com
   gateway_url: https://console.cloud.timescale.com/api
   docs_mcp:       true
   docs_mcp_url:   https://mcp.tigerdata.com/docs
   project_id:  tgrproject
   service_id:  tgrservice
   output:      table
   analytics:   true
   password_storage: keyring
   debug:       false
   config_dir:  /Users/<username>/.config/tiger
   

And that is it, you are ready to use Tiger CLI to manage your services in Tiger Cloud.

Commands

You can use the following commands with Tiger CLI. For more information on each command, use the -h flag. For example: tiger auth login -h

Command	Subcommand	Description
auth		Manage authentication and credentials for your Tiger Data account
	login	Create an authenticated connection to your Tiger Data account
	logout	Remove the credentials used to create authenticated connections to Tiger Cloud
	status	Show your current authentication status and project ID
version		Show information about the currently installed version of Tiger CLI
config		Manage your Tiger CLI configuration
	show	Show the current configuration
	set <key> <value>	Set a specific value in your configuration. For example, tiger config set debug true
	unset <key>	Clear the value of a configuration parameter. For example, tiger config unset debug
	reset	Reset the configuration to the defaults. This also logs you out from the current Tiger Cloud project
service		Manage the Tiger Cloud services in this project
	create	Create a new service in this project. Possible flags are: --name: service name (auto-generated if not provided) --addons: addons to enable (time-series, ai, or none for PostgreSQL-only) --region: region code where the service will be deployed --cpu-memory: CPU/memory allocation combination --replicas: number of high-availability replicas --no-wait: don't wait for the operation to complete --wait-timeout: wait timeout duration (for example, 30m, 1h30m, 90s) --no-set-default: don't set this service as the default service --with-password: include password in output --output, -o: output format (json, yaml, table) Possible cpu-memory combinations are: shared/shared 0.5 CPU/2 GB 1 CPU/4 GB 2 CPU/8 GB 4 CPU/16 GB 8 CPU/32 GB 16 CPU/64 GB 32 CPU/128 GB
	delete <service-id>	Delete a service from this project. This operation is irreversible and requires confirmation by typing the service ID
	fork <service-id>	Fork an existing service to create a new independent copy. Key features are: Timing options: --now, --last-snapshot, --to-timestamp Resource configuration: --cpu-memory Naming: --name <name>. Defaults to {source-service-name}-fork Wait behavior: --no-wait, --wait-timeout Default service: --no-set-default
	get <service-id> (aliases: describe, show)	Show detailed information about a specific service in this project
	list	List all the services in this project
	update-password <service-id>	Update the master password for a service
db		Database operations and management
	connect <service-id>	Connect to a service
	connection-string <service-id>	Retrieve the connection string for a service
	save-password <service-id>	Save the password for a service
	test-connection <service-id>	Test the connectivity to a service
mcp		Manage the Tiger Model Context Protocol Server for AI Assistant integration
	install [client]	Install and configure Tiger Model Context Protocol Server for a specific client (claude-code, cursor, windsurf, or other). If no client is specified, you'll be prompted to select one interactively
	start	Start the Tiger Model Context Protocol Server. This is the same as tiger mcp start stdio
	start stdio	Start the Tiger Model Context Protocol Server with stdio transport (default)
	start http	Start the Tiger Model Context Protocol Server with HTTP transport. Includes flags: --port (default: 8080), --host (default: localhost)

Global flags

You can use the following global flags with Tiger CLI:

Flag	Default	Description
--analytics	true	Set to false to disable usage analytics
--color	true	Set to false to disable colored output
--config-dir string	.config/tiger	Set the directory that holds config.yaml
--debug	No debugging	Enable debug logging
--help	-	Print help about the current command. For example, tiger service --help
--password-storage string	keyring	Set the password storage method. Options are keyring, pgpass, or none
--service-id string	-	Set the Tiger Cloud service to manage
--skip-update-check	-	Do not check if a new version of Tiger CLI is available

Configuration parameters

By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml. The name of these variables matches the flags you use to update them. However, you can override them using the following environmental variables:

• Configuration parameters

  • TIGER_CONFIG_DIR: path to configuration directory (default: ~/.config/tiger)
  • TIGER_API_URL: Tiger REST API base endpoint (default: https://console.cloud.timescale.com/public/api/v1)
  • TIGER_CONSOLE_URL: URL to Tiger Cloud Console (default: https://console.cloud.timescale.com)
  • TIGER_GATEWAY_URL: URL to the Tiger Cloud Console gateway (default: https://console.cloud.timescale.com/api)
  • TIGER_DOCS_MCP: enable/disable docs MCP proxy (default: true)
  • TIGER_DOCS_MCP_URL: URL to the Tiger MCP Server for Tiger Data docs (default: https://mcp.tigerdata.com/docs)
  • TIGER_SERVICE_ID: ID for the service updated when you call CLI commands
  • TIGER_ANALYTICS: enable or disable analytics (default: true)
  • TIGER_PASSWORD_STORAGE: password storage method (keyring, pgpass, or none)
  • TIGER_DEBUG: enable/disable debug logging (default: false)
  • TIGER_COLOR: set to false to disable colored output (default: true)

• Authentication parameters

  To authenticate without using the interactive login, either:

  • Set the following parameters with your client credentials, then login:

    TIGER_PUBLIC_KEY=<public_key> TIGER_SECRET_KEY=<secret_key> TIGER_PROJECT_ID=<project_id>\
    tiger auth login
    

  • Add your client credentials to the login command:

    tiger auth login --public-key=<public_key> --secret-key=<secret-key> --project-id=<project_id>
    

Tiger REST API is a comprehensive RESTful API you use to manage Tiger Cloud resources including VPCs, services, and read replicas.

This page shows you how to set up secure authentication for the Tiger REST API and create your first service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Data account.

• Install curl.

Configure secure authentication

Tiger REST API uses HTTP Basic Authentication with access keys and secret keys. All API requests must include proper authentication headers.

1. Set up API credentials

   1. In Tiger Cloud Console copy your project ID and store it securely using an environment variable:

   export TIGERDATA_PROJECT_ID="your-project-id"
   

   1. In Tiger Cloud Console create your client credentials and store them securely using environment variables:

      export TIGERDATA_ACCESS_KEY="Public key"
      export TIGERDATA_SECRET_KEY="Secret key"
      

2. Configure the API endpoint

   Set the base URL in your environment:

   export API_BASE_URL="https://console.cloud.timescale.com/public/api/v1"
   

3. Test your authenticated connection to Tiger REST API by listing the services in the current Tiger Cloud project

   curl -X GET "${API_BASE_URL}/projects/${TIGERDATA_PROJECT_ID}/services" \
     -u "${TIGERDATA_ACCESS_KEY}:${TIGERDATA_SECRET_KEY}" \
     -H "Content-Type: application/json"
   

   This call returns something like:

   • No services:

     []%
     

   • One or more services:

     [{"service_id":"tgrservice","project_id":"tgrproject","name":"tiger-eon",
     "region_code":"us-east-1","service_type":"TIMESCALEDB",
     "created":"2025-10-20T12:21:28.216172Z","paused":false,"status":"READY",
     "resources":[{"id":"104977","spec":{"cpu_millis":500,"memory_gbs":2,"volume_type":""}}],
     "metadata":{"environment":"DEV"},
     "endpoint":{"host":"tgrservice.tgrproject.tsdb.cloud.timescale.com","port":11111}}]
     

Create your first Tiger Cloud service

Create a new service using the Tiger REST API:

1. Create a service using the POST endpoint

   curl -X POST "${API_BASE_URL}/projects/${TIGERDATA_PROJECT_ID}/services" \
     -u "${TIGERDATA_ACCESS_KEY}:${TIGERDATA_SECRET_KEY}" \
     -H "Content-Type: application/json" \
     -d '{
        "name": "my-first-service",
        "addons": ["time-series"],
        "region_code": "us-east-1",
        "replica_count": 1,
        "cpu_millis": "1000",
        "memory_gbs": "4"
        }'
   

   Tiger Cloud creates a Development environment for you. That is, no delete protection, high-availability, spooling or read replication. You see something like:

    {"service_id":"tgrservice","project_id":"tgrproject","name":"my-first-service",
    "region_code":"us-east-1","service_type":"TIMESCALEDB",
    "created":"2025-10-20T22:29:33.052075713Z","paused":false,"status":"QUEUED",
    "resources":[{"id":"105120","spec":{"cpu_millis":1000,"memory_gbs":4,"volume_type":""}}],
    "metadata":{"environment":"PROD"},
    "endpoint":{"host":"tgrservice.tgrproject.tsdb.cloud.timescale.com","port":00001},
    "initial_password":"notTellingYou",
    "ha_replicas":{"sync_replica_count":0,"replica_count":1}}
   

2. Save service_id from the response to a variable:

   # Extract service_id from the JSON response
   export SERVICE_ID="service_id-from-response"
   

3. Check the configuration for the service

  curl -X GET "${API_BASE_URL}/projects/${TIGERDATA_PROJECT_ID}/services/${SERVICE_ID}" \
    -u "${TIGERDATA_ACCESS_KEY}:${TIGERDATA_SECRET_KEY}" \
    -H "Content-Type: application/json"

You see something like:

  {"service_id":"tgrservice","project_id":"tgrproject","name":"my-first-service",
  "region_code":"us-east-1","service_type":"TIMESCALEDB",
  "created":"2025-10-20T22:29:33.052075Z","paused":false,"status":"READY",
  "resources":[{"id":"105120","spec":{"cpu_millis":1000,"memory_gbs":4,"volume_type":""}}],
  "metadata":{"environment":"DEV"},
  "endpoint":{"host":"tgrservice.tgrproject.tsdb.cloud.timescale.com","port":11111},
  "ha_replicas":{"sync_replica_count":0,"replica_count":1}}

And that is it, you are ready to use the Tiger REST API to manage your services in Tiger Cloud.

Security best practices

Follow these security guidelines when working with the Tiger REST API:

• Credential management

  • Store API credentials as environment variables, not in code
  • Use credential rotation policies for production environments
  • Never commit credentials to version control systems

• Network security

  • Use HTTPS endpoints exclusively for API communication
  • Implement proper certificate validation in your HTTP clients

• Data protection

  • Use secure storage for service connection strings and passwords
  • Implement proper backup and recovery procedures for created services
  • Follow data residency requirements for your region

===== PAGE: https://docs.tigerdata.com/getting-started/run-queries-from-console/ =====

Run your queries from Tiger Cloud Console

As Tiger Cloud is based on Postgres, you can use lots of different tools to connect to your service and interact with your data.

In Tiger Cloud Console you can use the following ways to run SQL queries against your service:

• Data mode: a rich experience powered by PopSQL. You can write queries with autocomplete, save them in folders, share them, create charts/dashboards, and much more.

• SQL Assistant in the data mode: write, fix, and organize SQL faster and more accurately.

• SQL editor in the ops mode: a simple SQL editor in the ops mode that lets you run ad-hoc ephemeral queries. This is useful for quick one-off tasks like creating an index on a small table or inspecting pg_stat_statements.

If you prefer the command line to the ops mode SQL editor in Tiger Cloud Console, use psql.

Data mode

You use the data mode in Tiger Cloud Console to write queries, visualize data, and share your results.

This feature is not available under the Free pricing plan.

Available features are:

• Real-time collaboration: work with your team directly in the data mode query editor with live presence and multiple cursors.
• Schema browser: understand the structure of your service and see usage data on tables and columns.
• SQL Assistant: write, fix, and organize SQL faster and more accurately using AI.
• Autocomplete: get suggestions as you type your queries.
• Version history: access previous versions of a query from the built-in revision history, or connect to a git repo.
• Charts: visualize data from inside the UI rather than switch to Sheets or Excel.
• Schedules: automatically refresh queries and dashboards to create push alerts.
• Query variables: use Liquid to parameterize your queries or use if statements.
• Cross-platform support: work from Tiger Cloud Console or download the desktop app for macOS, Windows, and Linux.
• Easy connection: connect to Tiger Cloud, Postgres, Redshift, Snowflake, BigQuery, MySQL, SQL Server, and more.

Connect to your Tiger Cloud service in the data mode

To connect to a service:

1. Check your service is running correctly

   In Tiger Cloud Console, check that your service is marked as Running:

   

2. Connect to your service

   In the data mode in Tiger Cloud Console, select a service in the connection drop-down:

   

3. Run a test query

   Type SELECT CURRENT_DATE; in Scratchpad and click Run:

   

Quick recap. You:

• Manage your services in the ops mode in Tiger Cloud Console
• Manage your data in the data mode in Tiger Cloud Console
• Store configuration and security information in your config file.

Now you have used the data mode in Tiger Cloud Console, see how to easily do the following:

• Write a query
• Share a query with your teammates
• Create a chart from your data
• Create a dashboard of multiple query results
• Create schedules for your queries

Data mode FAQ

What if my service is within a vpc?

If your Tiger Cloud service runs inside a VPC, do one of the following to enable access for the PopSQL desktop app:

• Use PopSQL's bridge connector.
• Use an SSH tunnel: when you configure the connection in PopSQL, under Advanced Options enable Connect over SSH.
• Add PopSQL's static IPs (23.20.131.72, 54.211.234.135) to your allowlist.

What happens if another member of my Tiger Cloud project uses the data mode?

The number of data mode seats you are allocated depends on your pricing plan.

Will using the data mode affect the performance of my Tiger Cloud service?

There are a few factors to consider:

1. What instance size is your service?
2. How many users are running queries?
3. How computationally intensive are the queries?

If you have a small number of users running performant SQL queries against a service with sufficient resources, then there should be no degradation to performance. However, if you have a large number of users running queries, or if the queries are computationally expensive, best practice is to create a read replica and send analytical queries there.

If you'd like to prevent write operations such as insert or update, instead of using the tsdbadmin user, create a read-only user for your service and use that in the data mode.

SQL Assistant

SQL Assistant in Tiger Cloud Console is a chat-like interface that harnesses the power of AI to help you write, fix, and organize SQL faster and more accurately. Ask SQL Assistant to change existing queries, write new ones from scratch, debug error messages, optimize for query performance, add comments, improve readability—and really, get answers to any questions you can think of.

This feature is not available under the Free pricing plan.

Key capabilities

SQL Assistant offers a range of features to improve your SQL workflow, including:

• Real-time help: SQL Assistant provides in-context help for writing and understanding SQL. Use it to:

  • Understand functions: need to know how functions like LAG() or ROW_NUMBER() work? SQL Assistant explains it with examples.
  • Interpret complex queries: SQL Assistant breaks down dense queries, giving you a clear view of each part.

• Error resolution: SQL Assistant diagnoses errors as they happen, you can resolve issues without leaving your editor. Features include:

  • Error debugging: if your query fails, SQL Assistant identifies the issue and suggests a fix.
  • Performance tuning: for slow queries, SQL Assistant provides optimization suggestions to improve performance immediately.

• Query organization: to keep your query library organized, and help your team understand the purpose of each query, SQL Assistant automatically adds titles and summaries to your queries.

• Agent mode: to get results with minimal involvement from you, SQL Assistant autopilots through complex tasks and troubleshoots its own problems. No need to go step by step, analyze errors, and try out solutions. Simply turn on the agent mode in the LLM picker and watch SQL Assistant do all the work for you. Recommended for use when your database connection is configured with read-only credentials.

Supported LLMs

SQL Assistant supports a large number of LLMs, including:

• GPT-4o mini
• GPT-4o
• GPT-4.1 nano
• GPT-4.1 mini
• GPT-4.1
• o4-mini (low)
• o4-mini
• o4-mini (high)
• o3 (low)
• o3
• o3 (high)
• Claude 3.5 Haiku
• Claud 3.7 Sonnet
• Claud 3.7 Sonnet (extended thinking)
• Llama 3.3 70B Versatile
• Llama 3.3 70B Instruct
• Llama 3.1 405B Instruct
• Llama 4 Scout
• Llama 4 Maverick
• DeepSeek R1 Distill - Llama 3.3. 70B
• DeepSeek R1
• Gemini 2.0 Flash
• Sonnet 4
• Sonnet 4 (extended thinking)
• Opus 4
• Opus 4 (extended thinking)

Choose the LLM based on the particular task at hand. For simpler tasks, try the smaller and faster models like Gemini Flash, Haiku, or o4-mini. For more complex tasks, try the larger reasoning models like Claude Sonnet, Gemini Pro, or o3. We provide a description of each model to help you decide.

Limitations to keep in mind

For best results with SQL Assistant:

• Schema awareness: SQL Assistant references schema data but may need extra context in complex environments. Specify tables, columns, or joins as needed.
• Business logic: SQL Assistant does not inherently know specific business terms such as active user. Define these terms clearly to improve results.

Security, privacy, and data usage

Security and privacy is prioritized in Tiger Cloud Console. In data mode, project members manage SQL Assistant settings under User name > Settings > SQL Assistant.

SQL Assistant settings are:

• Opt-in features: all AI features are off by default. Only members of your Tiger Cloud project can enable them.
• Data protection: your data remains private as SQL Assistant operates with strict security protocols. To provide AI support, Tiger Cloud Console may share your currently open SQL document, some basic metadata about your database, and portions of your database schema. By default, Tiger Cloud Console does not include any data from query results, but you can opt in to include this context to improve the results.
• Sample data: to give the LLM more context so you have better SQL suggestions, enable sample data sharing in the SQL Assistant preferences.
• Telemetry: to improve SQL Assistant, Tiger Data collects telemetry and usage data, including prompts, responses, and query metadata.

Ops mode SQL editor

SQL editor is an integrated secure UI that you use to run queries and see the results for a Tiger Cloud service.

To enable or disable SQL editor in your service, click Operations > Service management, then update the setting for SQL editor.

To use SQL editor:

1. Open SQL editor from Tiger Cloud Console

   In the ops mode in Tiger Cloud Console, select a service, then click SQL editor.

   

2. Run a test query

   Type SELECT CURRENT_DATE; in the UI and click Run. The results appear in the lower window:

   

Cloud SQL editor licenses

• SQL editor in the ops mode: free for anyone with a Tiger Data account.

• Data mode: the number of seats you are allocated depends on your pricing plan.

  SQL Assistant is currently free for all users. In the future, limits or paid options may be introduced as we work to build the best experience.

• PopSQL standalone: there is a free plan available to everyone, as well as paid plans. See PopSQL Pricing for full details.

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/use-timescale/hypertables/ =====

Hypertables

Tiger Cloud supercharges your real-time analytics by letting you run complex queries continuously, with near-zero latency. Under the hood, this is achieved by using hypertables—Postgres tables that automatically partition your time-series data by time and optionally by other dimensions. When you run a query, Tiger Cloud identifies the correct partition, called chunk, and runs the query on it, instead of going through the entire table.

Hypertables offer the following benefits:

• Efficient data management with automated partitioning by time: Tiger Cloud splits your data into chunks that hold data from a specific time range. For example, one day or one week. You can configure this range to better suit your needs.

• Better performance with strategic indexing: an index on time in the descending order is automatically created when you create a hypertable. More indexes are created on the chunk level, to optimize performance. You can create additional indexes, including unique indexes, on the columns you need.

• Faster queries with chunk skipping: Tiger Cloud skips the chunks that are irrelevant in the context of your query, dramatically reducing the time and resources needed to fetch results. Even more—you can enable chunk skipping on non-partitioning columns.

• Advanced data analysis with hyperfunctions: Tiger Cloud enables you to efficiently process, aggregate, and analyze significant volumes of data while maintaining high performance.

To top it all, there is no added complexity—you interact with hypertables in the same way as you would with regular Postgres tables. All the optimization magic happens behind the scenes.

Inheritance is not supported for hypertables and may lead to unexpected behavior.

Partition by time

Each hypertable is partitioned into child hypertables called chunks. Each chunk is assigned a range of time, and only contains data from that range.

Time partitioning

Typically, you partition hypertables on columns that hold time values. Best practice is to use timestamptz column type. However, you can also partition on date, integer, timestamp and UUIDv7 types.

By default, each hypertable chunk holds data for 7 days. You can change this to better suit your needs. For example, if you set chunk_interval to 1 day, each chunk stores data for a single day.

TimescaleDB divides time into potential chunk ranges, based on the chunk_interval. Each hypertable chunk holds data for a specific time range only. When you insert data from a time range that doesn't yet have a chunk, TimescaleDB automatically creates a chunk to store it.

In practice, this means that the start time of your earliest chunk does not necessarily equal the earliest timestamp in your hypertable. Instead, there might be a time gap between the start time and the earliest timestamp. This doesn't affect your usual interactions with your hypertable, but might affect the number of chunks you see when inspecting it.

Best practices for scaling and partitioning

Best practices for maintaining a high performance when scaling include:

• Limit the number of hypertables in your service; having tens of thousands of hypertables is not recommended.
• Choose a strategic chunk size.

Chunk size affects insert and query performance. You want a chunk small enough to fit into memory so you can insert and query recent data without reading from disk. However, having too many small and sparsely filled chunks can affect query planning time and compression. The more chunks in the system, the slower that process becomes, even more so when all those chunks are part of a single hypertable.

Postgres builds the index on the fly during ingestion. That means that to build a new entry on the index, a significant portion of the index needs to be traversed during every row insertion. When the index does not fit into memory, it is constantly flushed to disk and read back. This wastes IO resources which would otherwise be used for writing the heap/WAL data to disk.

The default chunk interval is 7 days. However, best practice is to set chunk_interval so that prior to processing, the indexes for chunks currently being ingested into fit within 25% of main memory. For example, on a system with 64 GB of memory, if index growth is approximately 2 GB per day, a 1-week chunk interval is appropriate. If index growth is around 10 GB per day, use a 1-day interval.

You set chunk_interval when you create a hypertable, or by calling set_chunk_time_interval on an existing hypertable.

For a detailed analysis of how to optimize your chunk sizes, see the blog post on chunk time intervals. To learn how to view and set your chunk time intervals, see Optimize hypertable chunk intervals.

Hypertable indexes

By default, indexes are automatically created when you create a hypertable. The default index is on time, descending. You can prevent index creation by setting the create_default_indexes option to false.

Hypertables have some restrictions on unique constraints and indexes. If you want a unique index on a hypertable, it must include all the partitioning columns for the table. To learn more, see Enforce constraints with unique indexes on hypertables.

You can prevent index creation by setting the create_default_indexes option to false.

Partition by dimension

Partitioning on time is the most common use case for hypertable, but it may not be enough for your needs. For example, you may need to scan for the latest readings that match a certain condition without locking a critical hypertable.

The use case for a partitioning dimension is a multi-tenant setup. You isolate the tenants using the tenant_id space partition. However, you must perform extensive testing to ensure this works as expected, and there is a strong risk of partition explosion.

You add a partitioning dimension at the same time as you create the hypertable, when the table is empty. The good news is that although you select the number of partitions at creation time, as your data grows you can change the number of partitions later and improve query performance. Changing the number of partitions only affects chunks created after the change, not existing chunks. To set the number of partitions for a partitioning dimension, call set_number_partitions. For example:

1. Create the hypertable with the 1-day interval chunk interval

   CREATE TABLE conditions(
      "time"      timestamptz not null,
      device_id   integer,
      temperature float
   )
   WITH(
      timescaledb.hypertable,
      timescaledb.partition_column='time',
      timescaledb.chunk_interval='1 day'
   );
   

2. Add a hash partition on a non-time column

   select * from add_dimension('conditions', by_hash('device_id', 3));
   

   Now use your hypertable as usual, but you can also ingest and query efficiently by the device_id column.

3. Change the number of partitions as you data grows

   select set_number_partitions('conditions', 5, 'device_id');
   

===== PAGE: https://docs.tigerdata.com/use-timescale/hypercore/ =====

Hypercore

Hypercore is a hybrid row-columnar storage engine in TimescaleDB. It is designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage, delivering the best of both worlds:

Hypercore solves the key challenges in real-time analytics:

• High ingest throughput
• Low-latency ingestion
• Fast query performance
• Efficient handling of data updates and late-arriving data
• Streamlined data management

Hypercore’s hybrid approach combines the benefits of row-oriented and column-oriented formats:

• Fast ingest with rowstore: new data is initially written to the rowstore, which is optimized for high-speed inserts and updates. This process ensures that real-time applications easily handle rapid streams of incoming data. Mutability—upserts, updates, and deletes happen seamlessly.

• Efficient analytics with columnstore: as the data cools and becomes more suited for analytics, it is automatically converted to the columnstore. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space.

• Faster queries on compressed data in columnstore: in the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. Combined with chunk skipping, this helps you save on storage costs and keeps your queries operating at lightning speed.

• Fast modification of compressed data in columnstore: just use SQL to add or modify data in the columnstore. TimescaleDB is optimized for superfast INSERT and UPSERT performance.

• Full mutability with transactional semantics: regardless of where data is stored, hypercore provides full ACID support. Like in a vanilla Postgres database, inserts and updates to the rowstore and columnstore are always consistent, and available to queries as soon as they are completed.

For an in-depth explanation of how hypertables and hypercore work, see the Data model.

This section shows the following:

• Optimize your data for real-time analytics
• Improve query and upsert performance using secondary indexes
• Compression methods in hypercore
• Troubleshooting

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/ =====

Continuous aggregates

From real-time dashboards to performance monitoring and historical trend analysis, data aggregation is a must-have for any sort of analytical application. To address this need, TimescaleDB uses continuous aggregates to precompute and store aggregate data for you. Using Postgres materialized views, TimescaleDB incrementally refreshes the aggregation query in the background. When you do run the query, only the data that has changed needs to be computed, not the entire dataset. This means you always have the latest aggregate data at your fingertips—and spend as little resources on it, as possible.

In this section you:

• Learn about continuous aggregates to understand how it works before you begin using it.
• Create a continuous aggregate and query it.
• Create a continuous aggregate on top of another continuous aggregate.
• Add refresh policies to an existing continuous aggregate.
• Manage time in your continuous aggregates.
• Drop data from your continuous aggregates.
• Manage materialized hypertables.
• Use real-time aggregates.
• Convert continuous aggregates to the columnstore.
• Migrate your continuous aggregates from old to new format. Continuous aggregates created in TimescaleDB v2.7 and later are in the new format, unless explicitly created in the old format.
• Troubleshoot continuous aggregates.

===== PAGE: https://docs.tigerdata.com/use-timescale/services/ =====

About Tiger Cloud services

Tiger Cloud is the modern Postgres data platform for all your applications. It enhances Postgres to handle time series, events, real-time analytics, and vector search—all in a single database alongside transactional workloads.

You get one system that handles live data ingestion, late and out-of-order updates, and low latency queries, with the performance, reliability, and scalability your app needs. Ideal for IoT, crypto, finance, SaaS, and a myriad other domains, Tiger Cloud allows you to build data-heavy, mission-critical apps while retaining the familiarity and reliability of Postgres.

A Tiger Cloud service is a single optimised Postgres instance extended with innovations in the database engine and cloud infrastructure to deliver speed without sacrifice. A Tiger Cloud service is 10-1000x faster at scale! It is ideal for applications requiring strong data consistency, complex relationships, and advanced querying capabilities. Get ACID compliance, extensive SQL support, JSON handling, and extensibility through custom functions, data types, and extensions.

Each service is associated with a project in Tiger Cloud. Each project can have multiple services. Each user is a member of one or more projects.

You create free and standard services in Tiger Cloud Console, depending on your pricing plan. A free service comes at zero cost and gives you limited resources to get to know Tiger Cloud. Once you are ready to try out more advanced features, you can switch to a paid plan and convert your free service to a standard one.

The Free pricing plan and services are currently in beta.

To the Postgres you know and love, Tiger Cloud adds the following capabilities:

• Standard services:

  • Real-time analytics: store and query time-series data at scale for real-time analytics and other use cases. Get faster time-based queries with hypertables, continuous aggregates, and columnar storage. Save money by compressing data into the columnstore, moving cold data to low-cost bottomless storage in Amazon S3, and deleting old data with automated policies.
  • AI-focused: build AI applications from start to scale. Get fast and accurate similarity search with the pgvector and pgvectorscale extensions.
  • Hybrid applications: get a full set of tools to develop applications that combine time-based data and AI.

  All standard Tiger Cloud services include the tooling you expect for production and developer environments: live migration, automatic backups and PITR, high availability, read replicas, data forking, connection pooling, tiered storage, usage-based storage, secure in-Tiger Cloud Console SQL editing, service metrics and insights, streamlined maintenance, and much more. Tiger Cloud continuously monitors your services and prevents common Postgres out-of-memory crashes.

• Free services:

  Postgres with TimescaleDB and vector extensions

  Free services offer limited resources and a basic feature scope, perfect to get to know Tiger Cloud in a development environment.

Learn more about Tiger Cloud

Read about Tiger Cloud features in the documentation:

• Create your first hypertable.
• Run your first query using time_bucket().
• Trying more advanced time-series functions, starting with gap filling or real-time aggregates.

Keep testing during your free trial

You're now on your way to a great start with Tiger Cloud.

You have an unthrottled, 30-day free trial with Tiger Cloud to continue to test your use case. Before the end of your trial, make sure you add your credit card information. This ensures a smooth transition after your trial period concludes.

If you have any questions, you can join our community Slack group or contact us directly.

Advanced configuration

Tiger Cloud is a versatile hosting service that provides a growing list of advanced features for your Postgres and time-series data workloads.

For more information about customizing your database configuration, see the Configuration section.

The TimescaleDB Terraform provider provides configuration management resources for Tiger Cloud. You can use it to create, rename, resize, delete, and import services. For more information about the supported service configurations and operations, see the Terraform provider documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/write-data/ =====

Write data

Writing data in TimescaleDB works the same way as writing data to regular Postgres. You can add and modify data in both regular tables and hypertables using INSERT, UPDATE, and DELETE statements.

• Learn about writing data in TimescaleDB
• Insert data into hypertables
• Update data in hypertables
• Upsert data into hypertables
• Delete data from hypertables

For more information about using third-party tools to write data into TimescaleDB, see the Ingest data from other sources section.

===== PAGE: https://docs.tigerdata.com/use-timescale/query-data/ =====

Query data

Hypertables in TimescaleDB are Postgres tables. That means you can query them with standard SQL commands.

• About querying data
• Select data with SELECT
• Get faster DISTINCT queries with SkipScan
• Perform advanced analytic queries

===== PAGE: https://docs.tigerdata.com/use-timescale/time-buckets/ =====

Time buckets

Time buckets enable you to aggregate data in hypertables by time interval. For example, you can group data into 5-minute, 1-hour, and 3-day buckets to calculate summary values.

• Learn how time buckets work
• Use time buckets to aggregate data

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/ =====

Schema management

A database schema defines how the tables and indexes in your database are organized. Using a schema that is appropriate for your workload can result in significant performance improvements.

• Learn about schema management to understand how it works before you begin using it.
• Learn about indexing to understand how it works before you begin using it.
• Learn about tablespaces to understand how they work before you begin using them.
• Learn about constraints to understand how they work before you begin using them.
• Alter a hypertable to modify your schema.
• Create an index to speed up your queries.
• Create triggers to propagate your schema changes to chunks.
• Use JSON and JSONB for semi-structured data.
• Query external databases with foreign data wrappers.
• Troubleshoot your schemas.

===== PAGE: https://docs.tigerdata.com/use-timescale/configuration/ =====

Configuration

By default, Tiger Cloud uses the standard Postgres server configuration settings. However, in some cases, these settings are not appropriate, especially if you have larger servers that use more hardware resources such as CPU, memory, and storage.

This section contains information about tuning your Tiger Cloud service.

===== PAGE: https://docs.tigerdata.com/use-timescale/alerting/ =====

Alerting

Early issue detecting and prevention, ensuring high availability, and performance optimization are only a few of the reasons why alerting plays a major role for modern applications, databases, and services.

There are a variety of different alerting solutions you can use in conjunction with Tiger Cloud that are part of the Postgres ecosystem. Regardless of whether you are creating custom alerts embedded in your applications, or using third-party alerting tools to monitor event data across your organization, there are a wide selection of tools available.

Grafana

Grafana is a great way to visualize your analytical queries, and it has a first-class integration with Tiger Data products. Beyond data visualization, Grafana also provides alerting functionality to keep you notified of anomalies.

Within Grafana, you can define alert rules which are time-based thresholds for your dashboard data (for example, "Average CPU usage greater than 80 percent for 5 minutes"). When those alert rules are triggered, Grafana sends a message via the chosen notification channel. Grafana provides integration with webhooks, email and more than a dozen external services including Slack and PagerDuty.

To get started, first download and install Grafana. Next, add a new Postgres data source that points to your Tiger Cloud service. This data source was built by Tiger Data engineers, and it is designed to take advantage of the database's time-series capabilities. From there, proceed to your dashboard and set up alert rules as described above.

Alerting is only available in Grafana v4.0 and later.

Other alerting tools

Tiger Cloud works with a variety of alerting tools within the Postgres ecosystem. Users can use these tools to set up notifications about meaningful events that signify notable changes to the system.

Some popular alerting tools that work with Tiger Cloud include:

• DataDog
• Nagios
• Zabbix

See the integration guides for details.

===== PAGE: https://docs.tigerdata.com/use-timescale/data-retention/ =====

Data retention

Data retention helps you save on storage costs by deleting old data. You can combine data retention with continuous aggregates to downsample your data.

In this section:

• Learn about data retention before you start using it
• Learn about data retention with continuous aggregates for downsampling data
• Create a data retention policy
• Manually drop chunks of data
• Troubleshoot data retention

===== PAGE: https://docs.tigerdata.com/use-timescale/data-tiering/ =====

Storage in Tiger

Tiered storage is a hierarchical storage management architecture for real-time analytics services you create in Tiger Cloud.

Engineered for infinite low-cost scalability, tiered storage consists of the following:

• High-performance storage tier: stores the most recent and frequently queried data. This tier comes in two types, standard and enhanced, and provides you with up to 64 TB of storage and 32,000 IOPS.

• Object storage tier: stores data that is rarely accessed and has lower performance requirements. For example, old data for auditing or reporting purposes over long periods of time, even forever. The object storage tier is low-cost and bottomless.

No matter the tier your data is stored in, you can query it when you need it. Tiger Cloud seamlessly accesses the correct storage tier and generates the response.

You define tiering policies that automatically migrate data from the high-performance storage tier to the object tier as it ages. You use retention policies to remove very old data from the object storage tier.

With tiered storage you don't need an ETL process, infrastructure changes, or custom-built, bespoke solutions to offload data to secondary storage and fetch it back in when needed. Kick back and relax, we do the work for you.

In this section, you:

• Learn more about storage tiers: understand how the tiers are built and how they differ.
• Manage storage and tiering: configure high-performance storage, object storage, and data tiering.
• Query tiered data: query the data in the object storage.
• Learn about replicas and forks with tiered data: understand how tiered storage works with forks and replicas of your service.

===== PAGE: https://docs.tigerdata.com/use-timescale/metrics-logging/ =====

Metrics and logging

Find metrics and logs for your services in Tiger Cloud Console, or integrate with third-party monitoring services:

• Monitor your services in Tiger Cloud Console.
• Export metrics to Datadog.
• Export metrics to Amazon Cloudwatch.
• Export metrics to Prometheus.

===== PAGE: https://docs.tigerdata.com/use-timescale/ha-replicas/ =====

High availability and read replication

In Tiger Cloud, replicas are copies of the primary data instance in a Tiger Cloud service. If your primary becomes unavailable, Tiger Cloud automatically fails over to your HA replica.

The replication strategies offered by Tiger Cloud are:

• High Availability(HA) replicas: significantly reduce the risk of downtime and data loss due to system failure, and enable services to avoid downtime during routine maintenance.

• Read replicas: safely scale a service to power your read-intensive apps and business intelligence tooling and remove the load from the primary data instance.

For MST, see Failover in Managed Service for TimescaleDB. For self-hosted TimescaleDB, see Replication and high availability.

Rapid recovery

By default, all services have rapid recovery enabled.

Because compute and storage are handled separately in Tiger Cloud, services recover quickly from compute failures, but usually need a full recovery from backup for storage failures.

• Compute failure: the most common cause of database failure. Compute failures can be caused by hardware failing, or through things like unoptimized queries, causing increased load that maxes out the CPU usage. In these cases, data on disk is unaffected and only the compute and memory needs replacing. Tiger Cloud recovery immediately provisions new compute infrastructure for the service and mounts the existing storage to the new node. Any WAL that was in memory then replays. This process typically only takes thirty seconds. However, depending on the amount of WAL that needs replaying this may take up to twenty minutes. Even in the worst-case scenario, Tiger Cloud recovery is an order of magnitude faster than a standard recovery from backup.

• Storage failure: in the rare occurrence of disk failure, Tiger Cloud automatically performs a full recovery from backup.

If CPU usage for a service runs high for long periods of time, issues such as WAL archiving getting queued behind other processes can occur. This can cause a failure and could result in a larger data loss. To avoid data loss, services are monitored for this kind of scenario.

===== PAGE: https://docs.tigerdata.com/use-timescale/upgrades/ =====

Maintenance and upgrades

Tiger Cloud offers managed database services that provide a stable and reliable environment for your applications. Each service is based on a specific version of the Postgres database and the TimescaleDB extension. To ensure that you benefit from the latest features, performance and security improvements, it is important that your Tiger Cloud service is kept up to date with the latest versions of TimescaleDB and Postgres.

Tiger Cloud has the following upgrade policies:

• Minor software upgrades: handled automatically, you do not need to do anything.

  Upgrades are performed on your Tiger Cloud service during a maintenance window that you define to suit your workload. You can also manually upgrade TimescaleDB.

• Critical security upgrades: installed outside normal maintenance windows when necessary, and sometimes require a short outage.

  Downtime is usually between 30 seconds and 5 minutes. Tiger Data aims to notify you by email if downtime is required, so that you can plan accordingly. However, in some cases this is not possible.

• Major upgrades: such as a new version of Postgres are performed manually by you, or automatically by Tiger Cloud.

After a maintenance upgrade, the DNS name remains the same. However, the IP address often changes.

Minor software upgrades

If you do not manually upgrade TimescaleDB for non-critical upgrades, Tiger Cloud performs upgrades automatically in the next available maintenance window. The upgrade is first applied to your services tagged #dev, and three weeks later to those tagged #prod. Subscribe to get an email notification before your #prod services are upgraded. You can upgrade your #prod services manually sooner, if needed.

Most upgrades that occur during your maintenance windows do not require any downtime. This means that there is no service outage during the upgrade. However, all connections and transactions in progress during the upgrade are reset. Usually, the service connection is automatically restored after the reset.

Some minor upgrades do require some downtime. This is usually between 30 seconds and 5 minutes. If downtime is required for an upgrade, Tiger Data endeavors to notify you by email ahead of the upgrade. However, in some cases, we might not be able to do so. Best practice is to schedule your maintenance window so that any downtime disrupts your workloads as little as possible and minimize downtime with replicas. If there are no pending upgrades available during a regular maintenance window, no changes are performed.

To track the status of maintenance events, see the Tiger Cloud status page.

Minimize downtime with replicas

Maintenance upgrades require up to two automatic failovers. Each failover takes less than a few seconds. Tiger Cloud services with high-availability replicas and read replicas require minimal write downtime during maintenance, read-only queries keep working throughout.

During a maintenance event, services with replicas perform maintenance on each node independently. When maintenance is complete on the primary node, it is restarted:

• If the restart takes more than a minute, a replica node is promoted to primary, given that the replica has no replication lag. Maintenance now proceeds on the newly promoted replica, following the same sequence. If the newly promoted replica takes more than a minute to restart, the former primary is promoted back. In total, the process may result in up to two minutes of write downtime and two failover events.
• If the maintenance on the primary node is completed within a minute and it comes back online, the replica remains the replica.

Manually upgrade TimescaleDB for non-critical upgrades

Non-critical upgrades are available before the upgrade is performed automatically by Tiger Cloud. To upgrade TimescaleDB manually:

1. Connect to your service

   In Tiger Cloud Console, select the service you want to upgrade.

2. Upgrade TimescaleDB

   Either:

   • Click SQL Editor, then run ALTEREXTENSION timescaledb UPDATE.
   • Click ⋮, then Pause and Resume the service.

Upgrading to a newer version of Postgres allows you to take advantage of new features, enhancements, and security fixes. It also ensures that you are using a version of Postgres that's compatible with the newest version of TimescaleDB, allowing you to take advantage of everything it has to offer. For more information about feature changes between versions, see the Tiger Cloud release notes, supported systems, and the Postgres release notes.

Deprecations

To ensure you benefit from the latest features, optimal performance, enhanced security, and full compatibility with TimescaleDB, Tiger Cloud supports a defined set of Postgres major versions. To reduce the maintenance burden and continue providing a high-quality managed experience, as Postgres and TimescaleDB evolve, Tiger Data periodically deprecates older Postgres versions.

Tiger Data provides advance notification to allow you ample time to plan and perform your upgrade. The timeline deprecation is as follows:

• Deprecation notice period begins: you receive email notification of the deprecation and the timeline for the upgrade.
• Customer self-service upgrade window: best practice is to manually upgrade to a new Postgres version in this time.
• Automatic upgrade deadline: Tiger Cloud performs an automatic upgrade of your service.

Manually upgrade Postgres for a service

Upgrading to a newer version of Postgres enables you to take advantage of new features, enhancements, and security fixes. It also ensures that you are using a version of Postgres that's compatible with the newest version of TimescaleDB.

For a smooth upgrade experience, make sure you:

• Plan ahead: upgrades cause downtime, so ideally perform an upgrade during a low traffic time.
• Run a test upgrade: fork your service, then try out the upgrade on the fork before running it on your production system. This gives you a good idea of what happens during the upgrade, and how long it might take.
• Keep a copy of your service: if you're worried about losing your data, fork your service without upgrading, and keep this duplicate of your service. To reduce cost, you can immediately pause this fork and only pay for storage until you are comfortable deleting it after the upgrade is complete.

Tiger Cloud services with replicas cannot be upgraded. To upgrade a service with a replica, you must first delete the replica and then upgrade the service.

The following table shows you the compatible versions of Postgres and TimescaleDB.

TimescaleDB version	Postgres 17	Postgres 16	Postgres 15	Postgres 14	Postgres 13	Postgres 12	Postgres 11	Postgres 10
2.22.x	✅	✅	✅	❌	❌	❌	❌	❌
2.21.x	✅	✅	✅	❌	❌	❌	❌	❌
2.20.x	✅	✅	✅	❌	❌	❌	❌	❌
2.17 - 2.19	✅	✅	✅	✅	❌	❌	❌	❌
2.16.x	❌	✅	✅	✅	❌	❌	❌	❌
2.13 - 2.15	❌	✅	✅	✅	✅	❌	❌	❌
2.12.x	❌	❌	✅	✅	✅	❌	❌	❌
2.10.x	❌	❌	✅	✅	✅	✅	❌	❌
2.5 - 2.9	❌	❌	❌	✅	✅	✅	❌	❌
2.4	❌	❌	❌	❌	✅	✅	❌	❌
2.1 - 2.3	❌	❌	❌	❌	✅	✅	✅	❌
2.0	❌	❌	❌	❌	❌	✅	✅	❌
1.7	❌	❌	❌	❌	❌	✅	✅	✅

We recommend not using TimescaleDB with Postgres 17.1, 16.5, 15.9, 14.14, 13.17, 12.21. These minor versions introduced a breaking binary interface change that, once identified, was reverted in subsequent minor Postgres versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with Postgres 17.2, 16.6, etc and higher. Users of Tiger Cloud and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected.

For more information about feature changes between versions, see the Postgres release notes and TimescaleDB release notes.

Your Tiger Cloud service is unavailable until the upgrade is complete. This can take up to 20 minutes. Best practice is to test on a fork first, so you can estimate how long the upgrade will take.

To upgrade your service to a newer version of Postgres:

1. Connect to your service

   In Tiger Cloud Console, select the service you want to upgrade.

2. Disable high-availability replicas

   1. Click Operations > High Availability, then click Change configuaration.
   2. Select Non-production (No replica), then click Change configuration.

3. Disable read replicas

   1. Click Operations > Read scaling, then click the trash icon next to all replica sets.

4. Upgrade Postgres

   1. Click Operations > Service Upgrades.
   2. Click Upgrade service, then confirm that you are ready to start the upgrade.

   Your Tiger Cloud service is unavailable until the upgrade is complete. This normally takes up to 20 minutes. However, it can take longer if you have a large or complex service.

   When the upgrade is finished, your service automatically resumes normal operations. If the upgrade is unsuccessful, the service returns to the state it was in before you started the upgrade.

5. Enable high-availability replicas and replace your read replicas

Automatic Postgres upgrades for a service

If you do not manually upgrade your services within the customer self-service upgrade window, Tiger Cloud performs an automatic upgrade. Automatic upgrades can result in downtime, best practice is to manually upgrade your services during a low-traffic period for your application.

During an automatic upgrade:

1. Any configured high-availability replicas or read replicas are temporarily removed.
2. The primary service is upgraded.
3. High-availability replicas and read replicas are added back to the service.

Define your maintenance window

When you are considering your maintenance window schedule, best practice is to choose a day and time that usually has very low activity, such as during the early hours of the morning, or over the weekend. This helps minimize the impact of a short service interruption. Alternatively, you might prefer to have your maintenance window occur during office hours, so that you can monitor your system during the upgrade.

To change your maintenance window:

1. Connect to your service

   In Tiger Cloud Console, select the service you want to manage.

2. Set your maintenance window

   1. Click Operations > Environment, then click Change maintenance window.
   2. Select the maintence window start time, then click Apply.

   Maintenance windows can run for up to four hours.

===== PAGE: https://docs.tigerdata.com/use-timescale/extensions/ =====

Postgres extensions

The following Postgres extensions are installed with each Tiger Cloud service:

• Tiger Data extensions
• Postgres built-in extensions
• Third-party extensions

Tiger Data extensions

Extension	Description	Enabled by default
pgai	Helper functions for AI workflows	For AI-focused services
pg_textsearch	BM25-based full-text search	Currently early access. For development and staging environments only
pgvector	Vector similarity search for Postgres	For AI-focused services
pgvectorscale	Advanced indexing for vector data	For AI-focused services
timescaledb_toolkit	TimescaleDB Toolkit	For Real-time analytics services
timescaledb	TimescaleDB	For all services

Postgres built-in extensions

Extension	Description	Enabled by default
autoinc	Functions for autoincrementing fields	-
amcheck	Functions for verifying relation integrity	-
bloom	Bloom access method - signature file-based index	-
bool_plperl	Transform between bool and plperl	-
btree_gin	Support for indexing common datatypes in GIN	-
btree_gist	Support for indexing common datatypes in GiST	-
citext	Data type for case-insensitive character strings	-
cube	Data type for multidimensional cubes	-
dict_int	Text search dictionary template for integers	-
dict_xsyn	Text search dictionary template for extended synonym processing	-
earthdistance	Calculate great-circle distances on the surface of the Earth	-
fuzzystrmatch	Determine similarities and distance between strings	-
hstore	Data type for storing sets of (key, value) pairs	-
hstore_plperl	Transform between hstore and plperl	-
insert_username	Functions for tracking who changed a table	-
intagg	Integer aggregator and enumerator (obsolete)	-
intarray	Functions, operators, and index support for 1-D arrays of integers	-
isn	Data types for international product numbering standards	-
jsonb_plperl	Transform between jsonb and plperl	-
lo	Large object maintenance	-
ltree	Data type for hierarchical tree-like structures	-
moddatetime	Functions for tracking last modification time	-
old_snapshot	Utilities in support of old_snapshot_threshold	-
pgcrypto	Cryptographic functions	-
pgrowlocks	Show row-level locking information	-
pgstattuple	Obtain tuple-level statistics	-
pg_freespacemap	Examine the free space map (FSM)	-
pg_prewarm	Prewarm relation data	-
pg_stat_statements	Track execution statistics of all SQL statements executed	For all services
pg_trgm	Text similarity measurement and index searching based on trigrams	-
pg_visibility	Examine the visibility map (VM) and page-level visibility info	-
plperl	PL/Perl procedural language	-
plpgsql	SQL procedural language	For all services
postgres_fdw	Foreign data wrappers	For all services
refint	Functions for implementing referential integrity (obsolete)	-
seg	Data type for representing line segments or floating-point intervals	-
sslinfo	Information about SSL certificates	-
tablefunc	Functions that manipulate whole tables, including crosstab	-
tcn	Trigger change notifications	-
tsm_system_rows	TABLESAMPLE method which accepts the number of rows as a limit	-
tsm_system_time	TABLESAMPLE method which accepts the time in milliseconds as a limit	-
unaccent	Text search dictionary that removes accents	-
uuid-ossp	Generate universally unique identifiers (UUIDs)	-

Third-party extensions

Extension	Description	Enabled by default
h3	H3 bindings for Postgres	-
pgaudit	Detailed session and/or object audit logging	-
pgpcre	Perl-compatible RegEx	-
pg_cron	SQL commands that you can schedule and run directly inside the database	Contact us to enable
pg_repack	Table reorganization in Postgres with minimal locks	-
pgrouting	Geospatial routing functionality	-
postgis	PostGIS geometry and geography spatial types and functions	-
postgis_raster	PostGIS raster types and functions	-
postgis_sfcgal	PostGIS SFCGAL functions	-
postgis_tiger_geocoder	PostGIS Tiger Cloud geocoder and reverse geocoder	-
postgis_topology	PostGIS topology spatial types and functions	-
unit	SI units for Postgres	-

===== PAGE: https://docs.tigerdata.com/use-timescale/backup-restore/ =====

Back up and recover your Tiger Cloud services

Tiger Cloud provides comprehensive backup and recovery solutions to protect your data, including automatic daily backups, cross-region protection, and point-in-time recovery.

Automatic backups

Tiger Cloud automatically handles backup for your Tiger Cloud services using the pgBackRest tool. You don't need to perform backups manually. What's more, with cross-region backup, you are protected when an entire AWS region goes down.

Tiger Cloud automatically creates one full backup every week, and incremental backups every day in the same region as your service. Additionally, all Write-Ahead Log (WAL) files are retained back to the oldest full backup. This means that you always have a full backup available for the current and previous week:

On Scale and Performance pricing plans, you can check the list of backups for the previous 14 days in Tiger Cloud Console. To do so, select your service, then click Operations > Backup and restore > Backup history.

In the event of a storage failure, a service automatically recovers from a backup to the point of failure. If the whole availability zone goes down, your Tiger Cloud services are recovered in a different zone. In the event of a user error, you can create a point-in-time recovery fork.

Enable cross-region backup

<Availability products={['cloud']} price_plans={['enterprise']} />

For added reliability, you can enable cross-region backup. This protects your data when an entire AWS region goes down. In this case, you have two identical backups of your service at any time, but one of them is in a different AWS region. Cross-region backups are updated daily and weekly in the same way as a regular backup. You can have one cross-region backup for a service.

You enable cross-region backup when you create a service, or configure it for an existing service in Tiger Cloud Console:

1. In Console, select your service and click Operations > Backup & restore.

2. In Cross-region backup, select the region in the dropdown and click Enable backup.

   

   You can now see the backup, its region, and creation date in a list.

You can have one cross-region backup per service. To change the region of your backup:

1. In Console, select your service and click Operations > Backup & restore.

2. Click the trash icon next to the existing backup to disable it.

   

3. Create a new backup in a different region.

Create a point-in-time recovery fork

<Availability products={['cloud']} />

To recover your service from a destructive or unwanted action, create a point-in-time recovery fork. You can recover a service to any point within the period defined by your pricing plan. The provision time for the recovery fork is typically less than twenty minutes, but can take longer depending on the amount of WAL to be replayed. The original service stays untouched to avoid losing data created since the time of recovery.

All tiered data remains recoverable during the PITR period. When restoring to any point-in-time recovery fork, your service contains all data that existed at that moment - whether it was stored in high-performance or low-cost storage.

When you restore a recovery fork:

• Data restored from a PITR point is placed into high-performance storage
• The tiered data, as of that point in time, remains in tiered storage

To avoid paying for compute for the recovery fork and the original service, pause the original to only pay storage costs.

You initiate a point-in-time recovery from a same-region or cross-region backup in Tiger Cloud Console:

1. In Tiger Cloud Console, from the Services list, ensure the service you want to recover has a status of Running or Paused.

2. Navigate to Operations > Service management and click Create recovery fork.

3. Select the recovery point, ensuring the correct time zone (UTC offset).

4. Configure the fork.

   

   You can configure the compute resources, add an HA replica, tag your fork, and add a connection pooler. Best practice is to match the same configuration you had at the point you want to recover to.

5. Confirm by clicking Create recovery fork.

   A fork of the service is created. The recovered service shows in Services with a label specifying which service it has been forked from.

6. Update the connection strings in your app

   Since the point-in-time recovery is done in a fork, to migrate your application to the point of recovery, change the connection strings in your application to use the fork.

Contact us, and we will assist in recovering your service.

Create a service fork

To manage development forks:

1. Install Tiger CLI

   Use the terminal to install the CLI:

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   brew install --cask timescale/tap/tiger-cli
   

   curl -fsSL https://cli.tigerdata.com | sh
   

2. Set up API credentials

   1. Log Tiger CLI into your Tiger Data account:

      tiger auth login
      

      Tiger CLI opens Console in your browser. Log in, then click Authorize.

      You can have a maximum of 10 active client credentials. If you get an error, open credentials and delete an unused credential.

   2. Select a Tiger Cloud project:

      Auth URL is: https://console.cloud.timescale.com/oauth/authorize?client_id=lotsOfURLstuff
      Opening browser for authentication...
      Select a project:
      
      > 1. Tiger Project (tgrproject)
      2. YourCompany (Company wide project) (cpnproject)
      3. YourCompany Department (dptproject)
      
      Use ↑/↓ arrows or number keys to navigate, enter to select, q to quit
      

      If only one project is associated with your account, this step is not shown.

      Where possible, Tiger CLI stores your authentication information in the system keychain/credential manager. If that fails, the credentials are stored in ~/.config/tiger/credentials with restricted file permissions (600). By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml.

3. Test your authenticated connection to Tiger Cloud by listing services

   tiger service list
   

   This call returns something like:

   • No services:

     🏜️  No services found! Your project is looking a bit empty.
     🚀 Ready to get started? Create your first service with: tiger service create
     

   • One or more services:

     ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐
     │ SERVICE ID │        NAME         │ STATUS │    TYPE     │    REGION    │     CREATED      │
     ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤
     │ tgrservice │ tiger-agent-service │ READY  │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │
     └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘
     

4. Fork the service

    tiger service fork tgrservice --now --no-wait --name bob
   

   By default a fork matches the resource of the parent Tiger Cloud services. For paid plans specify --cpu and/or --memory for dedicated resources.

   You see something like:

   🍴 Forking service 'tgrservice' to create 'bob' at current state...
   ✅ Fork request accepted!
   📋 New Service ID: <service_id>
   🔐 Password saved to system keyring for automatic authentication
   🎯 Set service '<service_id>' as default service.
   ⏳ Service is being forked. Use 'tiger service list' to check status.
   ┌───────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
   │     PROPERTY      │                                              VALUE                                               │
   ├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ Service ID        │ <service_id>                                                                                       │
   │ Name              │ bob                                                                                              │
   │ Status            │                                                                                                  │
   │ Type              │ TIMESCALEDB                                                                                      │
   │ Region            │ eu-central-1                                                                                     │
   │ CPU               │ 0.5 cores (500m)                                                                                 │
   │ Memory            │ 2 GB                                                                                             │
   │ Direct Endpoint   │ <service-id>.<project-id>.tsdb.cloud.timescale.com:<port>                                             │
   │ Created           │ 2025-10-08 13:58:07 UTC                                                                          │
   │ Connection String │ postgresql://tsdbadmin@<service-id>.<project-id>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require │
   └───────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
   

5. When you are done, delete your forked service

   1. Use the CLI to request service delete:

      tiger service delete <service_id>
      

   2. Validate the service delete:

      Are you sure you want to delete service '<service_id>'? This operation cannot be undone.
      Type the service ID '<service_id>' to confirm:
      <service_id>
      

      You see something like:

      🗑️  Delete request accepted for service '<service_id>'.
      ✅ Service '<service_id>' has been successfully deleted.
      

===== PAGE: https://docs.tigerdata.com/use-timescale/fork-services/ =====

Fork services

Modern development is highly iterative. Developers and AI agents need safe spaces to test changes before deploying them to production. Forkable services make this natural and easy. Spin up a branch, run your test, throw it away, or merge it back.

A fork is an exact copy of a service at a specific point in time, with its own independent data and configuration, including:

• The database data and schema
• Configuration
• An admin tsdbadmin user with a new password

Forks are fully independent. Changes to the fork don't affect the parent service. You can query them, run migrations, add indexes, or test new features against the fork without affecting the original service.

Forks are a powerful way to share production-scale data safely. Testing, BI and data science teams often need access to real datasets to build models or generate insights. With forkable services, you easily create fast, zero-copy branches of a production service that are isolated from production, but contain all the data needed for analysis. Rapid fork creation dramatically reduces friction getting insights from live data.

Understand service forks

You can use service forks for disaster recovery, CI/CD automation, and testing and development. For example, you can automatically test a major Postgres upgrade on a fork before applying it to your production service.

Tiger Cloud offers the following fork strategies:

• now: create a fresh fork of your database at the current time. Use when:

  • You need the absolute latest data
  • Recent changes must be included in the fork

• last-snapshot: fork from the most recent automatic backup or snapshot. Use when:

  • You want the fastest possible fork creation
  • Slightly behind current data is acceptable

• timestamp: fork from a specific point in time within your [retention period][pricing]. Use when:

  • Disaster recovery from a known-good state
  • Investigating issues that occurred at a specific time
  • Testing "what-if" scenarios from historical data

The retention period for point-in-time recovery and forking depends on your pricing plan.

Fork creation speed

Fork creation speed depends on your type of service you want to create:

• Free: ~30-90 seconds. Uses a Copy-on-Write storage architecture with zero-copy between a fork and the parent.
• Paid: varies with the size of your service, typically 5-20+ minutes. Uses tradional storage architecture with backup restore + WAL replay.

Billing

You can fork a free service to a free or a paid service. However, you cannot fork a paid service to a free service.

Billing on storage works in the following way:

• High-performance storage:
  • Copy-on-Write: you are only billed for storage for the chunks that diverge from the parent service.
  • Traditional: you are billed for storage for the whole service.
• Object storage tier:
  • Tiered data is shared across forks using copy-on-write and traditional storage:
  • Chunks in tiered storage are only billed once, regardless of the number of forks
  • Only new or modified chunks in a fork incur additional costs

For details, see Replicas and forks with tiered data.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Manage forks using Tiger CLI

To manage development forks:

1. Install Tiger CLI

   Use the terminal to install the CLI:

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   brew install --cask timescale/tap/tiger-cli
   

   curl -fsSL https://cli.tigerdata.com | sh
   

2. Set up API credentials

   1. Log Tiger CLI into your Tiger Data account:

      tiger auth login
      

      Tiger CLI opens Console in your browser. Log in, then click Authorize.

      You can have a maximum of 10 active client credentials. If you get an error, open credentials and delete an unused credential.

   2. Select a Tiger Cloud project:

      Auth URL is: https://console.cloud.timescale.com/oauth/authorize?client_id=lotsOfURLstuff
      Opening browser for authentication...
      Select a project:
      
      > 1. Tiger Project (tgrproject)
      2. YourCompany (Company wide project) (cpnproject)
      3. YourCompany Department (dptproject)
      
      Use ↑/↓ arrows or number keys to navigate, enter to select, q to quit
      

      If only one project is associated with your account, this step is not shown.

      Where possible, Tiger CLI stores your authentication information in the system keychain/credential manager. If that fails, the credentials are stored in ~/.config/tiger/credentials with restricted file permissions (600). By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml.

3. Test your authenticated connection to Tiger Cloud by listing services

   tiger service list
   

   This call returns something like:

   • No services:

     🏜️  No services found! Your project is looking a bit empty.
     🚀 Ready to get started? Create your first service with: tiger service create
     

   • One or more services:

     ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐
     │ SERVICE ID │        NAME         │ STATUS │    TYPE     │    REGION    │     CREATED      │
     ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤
     │ tgrservice │ tiger-agent-service │ READY  │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │
     └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘
     

4. Fork the service

    tiger service fork tgrservice --now --no-wait --name bob
   

   By default a fork matches the resource of the parent Tiger Cloud services. For paid plans specify --cpu and/or --memory for dedicated resources.

   You see something like:

   🍴 Forking service 'tgrservice' to create 'bob' at current state...
   ✅ Fork request accepted!
   📋 New Service ID: <service_id>
   🔐 Password saved to system keyring for automatic authentication
   🎯 Set service '<service_id>' as default service.
   ⏳ Service is being forked. Use 'tiger service list' to check status.
   ┌───────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
   │     PROPERTY      │                                              VALUE                                               │
   ├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ Service ID        │ <service_id>                                                                                       │
   │ Name              │ bob                                                                                              │
   │ Status            │                                                                                                  │
   │ Type              │ TIMESCALEDB                                                                                      │
   │ Region            │ eu-central-1                                                                                     │
   │ CPU               │ 0.5 cores (500m)                                                                                 │
   │ Memory            │ 2 GB                                                                                             │
   │ Direct Endpoint   │ <service-id>.<project-id>.tsdb.cloud.timescale.com:<port>                                             │
   │ Created           │ 2025-10-08 13:58:07 UTC                                                                          │
   │ Connection String │ postgresql://tsdbadmin@<service-id>.<project-id>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require │
   └───────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
   

5. When you are done, delete your forked service

   1. Use the CLI to request service delete:

      tiger service delete <service_id>
      

   2. Validate the service delete:

      Are you sure you want to delete service '<service_id>'? This operation cannot be undone.
      Type the service ID '<service_id>' to confirm:
      <service_id>
      

      You see something like:

      🗑️  Delete request accepted for service '<service_id>'.
      ✅ Service '<service_id>' has been successfully deleted.
      

Manage forks using Console

To manage development forks:

1. In Tiger Cloud Console, from the Services list, ensure the service you want to recover has a status of Running or Paused.

2. Navigate to Operations > Service Management and click Fork service.

3. Configure the fork, then click Fork service.

   A fork of the service is created. The forked service shows in Services with a label specifying which service it has been forked from.

   

4. Update the connection strings in your app to use the fork.

Integrate service forks in your CI/CD pipeline

To fork your Tiger Cloud service using GitHub actions:

1. Store your Tiger Cloud API key as a GitHub Actions secret

   1. In Tiger Cloud Console, click Create credentials.
   2. Save the Public key and Secret key locally, then click Done.
   3. In your GitHub repository, click Settings, open Secrets and variables, then click Actions.
   4. Click New repository secret, then set Name to TIGERDATA_API_KEY
   5. Set Secret to your Tiger Cloud API key in the following format <Public key>:<Secret key>, then click Add secret.

2. Add the GitHub Actions Marketplace to your workflow YAML files

   For example, the following workflow forks a service when a pull request is opened, running tests against the fork, then automatically cleans up.

   name: Test on a service fork
   on: pull_request
   
   jobs:
     test:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
   
         - name: Fork Database
           id: fork
           uses: timescale/fork-service@v1
           with:
             project_id: ${{ secrets.TIGERDATA_PROJECT_ID }}
             service_id: ${{ secrets.TIGERDATA_SERVICE_ID }}
             api_key: ${{ secrets.TIGERDATA_API_KEY }}
             fork_strategy: last-snapshot
             cleanup: true
             name: pr-${{ github.event.pull_request.number }}
   
         - name: Run Integration Tests
           env:
             DATABASE_URL: postgresql://tsdbadmin:${{ steps.fork.outputs.initial_password }}@${{ steps.fork.outputs.host }}:${{ steps.fork.outputs.port }}/tsdb?sslmode=require
           run: |
             npm install
             npm test          
         - name: Run Migrations
           env:
             DATABASE_URL: postgresql://tsdbadmin:${{ steps.fork.outputs.initial_password }}@${{ steps.fork.outputs.host }}:${{ steps.fork.outputs.port }}/tsdb?sslmode=require
           run: npm run migrate
   

   For the full list of inputs, outputs, and configuration options, see the Tiger Data - Fork Service in GitHub marketplace.

===== PAGE: https://docs.tigerdata.com/use-timescale/jobs/ =====

Jobs in TimescaleDB

TimescaleDB natively includes some job-scheduling policies, such as:

• Continuous aggregate policies to automatically refresh continuous aggregates
• Hypercore policies to optimize and compress historical data
• Retention policies to drop historical data
• Reordering policies to reorder data within chunks

If these don't cover your use case, you can create and schedule custom-defined jobs to run within your database. They help you automate periodic tasks that aren't covered by the native policies.

In this section, you see how to:

• Create and manage jobs
• Set up a generic data retention policy that applies across all hypertables
• Implement automatic moving of chunks between tablespaces
• Automatically downsample and compress older chunks

===== PAGE: https://docs.tigerdata.com/use-timescale/security/ =====

Security

Learn how Tiger Cloud protects your data and privacy.

• Learn about security in Tiger Cloud
• Restrict access to your project
• Restrict access to the data in your service
• Set up multifactor and SAML authentication
• Generate multiple client credentials instead of using your username and password
• Connect with a stricter SSL mode
• Secure your services with VPC peering
• Connect to your services from any cloud with AWS Transit Gateway
• Restrict access with an IP address allow list

===== PAGE: https://docs.tigerdata.com/use-timescale/limitations/ =====

Limitations

While TimescaleDB generally offers capabilities that go beyond what Postgres offers, there are some limitations to using hypertables.

Hypertable limitations

• Time dimensions (columns) used for partitioning cannot have NULL values.
• Unique indexes must include all columns that are partitioning dimensions.
• UPDATE statements that move values between partitions (chunks) are not supported. This includes upserts (INSERT ... ON CONFLICT UPDATE).
• Foreign key constraints from a hypertable referencing another hypertable are not supported.

===== PAGE: https://docs.tigerdata.com/use-timescale/tigerlake/ =====

Integrate data lakes with Tiger Cloud

Tiger Lake enables you to build real-time applications alongside efficient data pipeline management within a single system. Tiger Lake unifies the Tiger Cloud operational architecture with data lake architectures.

Tiger Lake is a native integration enabling synchronization between hypertables and relational tables running in Tiger Cloud services to Iceberg tables running in Amazon S3 Tables in your AWS account.

Tiger Lake is currently in private beta. Please contact us to request access.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details.

Integrate a data lake with your Tiger Cloud service

To connect a Tiger Cloud service to your data lake:

1. Set the AWS region to host your table bucket

   1. In AWS CloudFormation, select the current AWS region at the top-right of the page.
   2. Set it to the Region you want to create your table bucket in.

   This must match the region your Tiger Cloud service is running in: if the regions do not match AWS charges you for cross-region data transfer.

2. Create your CloudFormation stack

   1. Click Create stack, then select With new resources (standard).

   2. In Amazon S3 URL, paste the following URL, then click Next.

      https://tigerlake.s3.us-east-1.amazonaws.com/tigerlake-connect-cloudformation.yaml
      

   3. In Specify stack details, enter the following details, then click Next:

      • Stack Name: a name for this CloudFormation stack
      • BucketName: a name for this S3 table bucket
      • ProjectID and ServiceID: enter the connection details for your Tiger Lake service

   4. In Configure stack options check I acknowledge that AWS CloudFormation might create IAM resources, then click Next.

   5. In Review and create, click Submit, then wait for the deployment to complete. AWS deploys your stack and creates the S3 table bucket and IAM role.

   6. Click Outputs, then copy all four outputs.

3. Connect your service to the data lake

   1. In Tiger Cloud Console, select the service you want to integrate with AWS S3 Tables, then click Connectors.

   2. Select the Apache Iceberg connector and supply the:

      • ARN of the S3Table bucket
      • ARN of a role with permissions to write to the table bucket

   Provisioning takes a couple of minutes.

1. Create your CloudFormation stack

   Replace the following values in the command, then run it from the terminal:

   • Region: region of the S3 table bucket
   • StackName: the name for this CloudFormation stack
   • BucketName: the name of the S3 table bucket to create
   • ProjectID: enter your Tiger Cloud service connection details
   • ServiceID: enter your Tiger Cloud service connection details

   aws cloudformation create-stack \
    --capabilities CAPABILITY_IAM \
    --template-url https://tigerlake.s3.us-east-1.amazonaws.com/tigerlake-connect-cloudformation.yaml \
    --region <Region> \
    --stack-name <StackName> \
    --parameters \
      ParameterKey=BucketName,ParameterValue="<BucketName>" \
      ParameterKey=ProjectID,ParameterValue="<ProjectID>" \
      ParameterKey=ServiceID,ParameterValue="<ServiceID>"
   

Setting up the integration through Tiger Cloud Console in Tiger Cloud, provides a convenient copy-paste option with the placeholders populated.

1. Connect your service to the data lake

   1. In Tiger Cloud Console, select the service you want to integrate with AWS S3 Tables, then click Connectors.

   2. Select the Apache Iceberg connector and supply the:

      • ARN of the S3Table bucket
      • ARN of a role with permissions to write to the table bucket

   Provisioning takes a couple of minutes.

1. Create a S3 Bucket

   1. Set the AWS region to host your table bucket

      1. In Amazon S3 console, select the current AWS region at the top-right of the page.
      2. Set it to the Region your you want to create your table bucket in.

      This must match the region your Tiger Cloud service is running in: if the regions do not match AWS charges you for cross-region data transfer.

   2. In the left navigation pane, click Table buckets, then click Create table bucket.

   3. Enter Table bucket name, then click Create table bucket.

   4. Copy the Amazon Resource Name (ARN) for your table bucket.

2. Create an ARN role

   1. In IAM Dashboard, click Roles then click Create role

   2. In Select trusted entity, click Custom trust policy, replace the Custom trust policy code block with the following:

      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::142548018081:root"
                  },
                  "Action": "sts:AssumeRole",
                  "Condition": {
                      "StringEquals": {
                          "sts:ExternalId": "<ProjectID>/<ServiceID>"
                      }
                  }
              }
          ]
      }
      

      "Principal": { "AWS": "arn:aws:iam::123456789012:root" } does not mean root access. This delegates permissions to the entire AWS account, not just the root user.

   3. Replace <ProjectID> and <ServiceID> with the the connection details for your Tiger Lake service, then click Next.

   4. In Permissions policies. click Next.

   5. In Role details, enter Role name, then click Create role.

   6. In Roles, select the role you just created, then click Add Permissions > Create inline policy.

   7. Select JSON then replace the Policy editor code block with the following:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "BucketOps",
            "Effect": "Allow",
            "Action": [
              "s3tables:*"
            ],
            "Resource": "<S3TABLE_BUCKET_ARN>"
          },
          {
            "Sid": "BucketTableOps",
            "Effect": "Allow",
            "Action": [
              "s3tables:*"
            ],
            "Resource": "<S3TABLE_BUCKET_ARN>/table/*"
          }
        ]
      }
      

   8. Replace <S3TABLE_BUCKET_ARN> with the Amazon Resource Name (ARN) for the table bucket you just created.

   9. Click Next, then give the inline policy a name and click Create policy.

3. Connect your service to the data lake

   1. In Tiger Cloud Console, select the service you want to integrate with AWS S3 Tables, then click Connectors.

   2. Select the Apache Iceberg connector and supply the:

      • ARN of the S3Table bucket
      • ARN of a role with permissions to write to the table bucket

   Provisioning takes a couple of minutes.

Stream data from your Tiger Cloud service to your data lake

When you start streaming, all data in the table is synchronized to Iceberg. Records are imported in time order, from oldest to youngest. The write throughput is approximately 40.000 records / second. For larger tables, a full import can take some time.

For Iceberg to perform update or delete statements, your hypertable or relational table must have a primary key. This includes composite primary keys.

To stream data from a Postgres relational table, or a hypertable in your Tiger Cloud service to your data lake, run the following statement:

ALTER TABLE  SET (
  tigerlake.iceberg_sync = true | false,
  tigerlake.iceberg_partitionby = '<partition_specification>',
  tigerlake.iceberg_namespace = '<namespace>',
  tigerlake.iceberg_table = ''
)

• tigerlake.iceberg_sync: boolean, set to true to start streaming, or false to stop the stream. A stream cannot resume after being stopped.
• tigerlake.iceberg_partitionby: optional property to define a partition specification in Iceberg. By default the Iceberg table is partitioned as day(<time-column of hypertable>). This default behavior is only applicable to hypertables. For more information, see partitioning.
• tigerlake.iceberg_namespace: optional property to set a namespace, the default is timescaledb.
• tigerlake.iceberg_table: optional property to specify a different table name. If no name is specified the Postgres table name is used.

Partitioning intervals

By default, the partition interval for an Iceberg table is one day(time-column) for a hypertable. Postgres table sync does not enable any partitioning in Iceberg for non-hypertables. You can set it using tigerlake.iceberg_partitionby. The following partition intervals and specifications are supported:

Interval	Description	Source types
hour	Extract a date or timestamp day, as days from epoch. Epoch is 1970-01-01.	date, timestamp, timestamptz
day	Extract a date or timestamp day, as days from epoch.	date, timestamp, timestamptz
month	Extract a date or timestamp day, as days from epoch.	date, timestamp, timestamptz
year	Extract a date or timestamp day, as days from epoch.	date, timestamp, timestamptz
truncate[W]	Value truncated to width W, see options

These partitions define the behavior using the Iceberg partition specification:

Sample code

The following samples show you how to tune data sync from a hypertable or a Postgres relational table to your data lake:

• Sync a hypertable with the default one-day partitioning interval on the ts_column column

  To start syncing data from a hypertable to your data lake using the default one-day chunk interval as the partitioning scheme to the Iceberg table, run the following statement:

  ALTER TABLE my_hypertable SET (tigerlake.iceberg_sync = true);
  

  This is equivalent to day(ts_column).

• Specify a custom partitioning scheme for a hypertable

  You use the tigerlake.iceberg_partitionby property to specify a different partitioning scheme for the Iceberg table at sync start. For example, to enforce an hourly partition scheme from the chunks on ts_column on a hypertable, run the following statement:

  ALTER TABLE my_hypertable SET (
    tigerlake.iceberg_sync = true,
    tigerlake.iceberg_partitionby = 'hour(ts_column)'
  );
  

• Set the partition to sync relational tables

  Postgres relational tables do not forward a partitioning scheme to Iceberg, you must specify the partitioning scheme using tigerlake.iceberg_partitionby when you start the sync. For example, for a standard Postgres table to sync to the Iceberg table with daily partitioning , run the following statement:

  ALTER TABLE my_postgres_table SET (
    tigerlake.iceberg_sync = true,
    tigerlake.iceberg_partitionby = 'day(timestamp_col)'
  );
  

• Stop sync to an Iceberg table for a hypertable or a Postgres relational table

  ALTER TABLE my_hypertable SET (tigerlake.iceberg_sync = false);
  

• Update or add the partitioning scheme of an Iceberg table

  To change the partitioning scheme of an Iceberg table, you specify the desired partitioning scheme using the tigerlake.iceberg_partitionby property. For example. if the samples table has an hourly (hour(ts)) partition on the ts timestamp column, to change to daily partitioning, call the following statement:

  ALTER TABLE samples SET (tigerlake.iceberg_partitionby = 'day(ts)');
  

  This statement is also correct for Iceberg tables without a partitioning scheme. When you change the partition, you do not have to pause the sync to Iceberg. Apache Iceberg handles the partitioning operation in function of the internal implementation.

Specify a different namespace

By default, tables are created in the the timescaledb namespace. To specify a different namespace when you start the sync, use the tigerlake.iceberg_namespace property. For example:

ALTER TABLE my_hypertable SET (
  tigerlake.iceberg_sync = true,
  tigerlake.iceberg_namespace = 'my_namespace'
);

Specify a different Iceberg table name

The table name in Iceberg is the same as the source table in Tiger Cloud. Some services do not allow mixed case, or have other constraints for table names. To define a different table name for the Iceberg table at sync start, use the tigerlake.iceberg_table property. For example:

ALTER TABLE Mixed_CASE_TableNAME SET (
  tigerlake.iceberg_sync = true,
  tigerlake.iceberg_table = 'my_table_name'
);

Limitations

• Service requires Postgres 17.6 and above is supported.
• Consistent ingestion rates of over 30000 records / second can lead to a lost replication slot. Burst can be feathered out over time.
• Amazon S3 Tables Iceberg REST catalog only is supported.
• In order to collect deletes made to data in the columstore, certain columnstore optimizations are disabled for hypertables.
• Direct Compress is not supported.
• The TRUNCATE statement is not supported, and does not truncate data in the corresponding Iceberg table.
• Data in a hypertable that has been moved to the low-cost object storage tier is not synced.
• Writing to the same S3 table bucket from multiple services is not supported, bucket-to-service mapping is one-to-one.
• Iceberg snapshots are pruned automatically if the amount exceeds 2500.

===== PAGE: https://docs.tigerdata.com/use-timescale/troubleshoot-timescaledb/ =====

Troubleshooting TimescaleDB

If you run into problems when using TimescaleDB, there are a few things that you can do. There are some solutions to common errors in this section as well as ways to output diagnostic information about your setup. If you need more guidance, you can join the community Slack group or post an issue on the TimescaleDB GitHub.

Common errors

Error updating TimescaleDB when using a third-party Postgres administration tool

The ALTER EXTENSION timescaledb UPDATE command must be the first command executed upon connection to a database. Some administration tools execute commands before this, which can disrupt the process. You might need to manually update the database with psql. See the update docs for details.

Log error: could not access file "timescaledb"

If your Postgres logs have this error preventing it from starting up, you should double-check that the TimescaleDB files have been installed to the correct location. The installation methods use pg_config to get Postgres's location. However, if you have multiple versions of Postgres installed on the same machine, the location pg_config points to may not be for the version you expect. To check which version of TimescaleDB is used:

$ pg_config --version
PostgreSQL 12.3

If that is the correct version, double-check that the installation path is the one you'd expect. For example, for Postgres 11.0 installed via Homebrew on macOS it should be /usr/local/Cellar/postgresql/11.0/bin:

$ pg_config --bindir
/usr/local/Cellar/postgresql/11.0/bin

If either of those steps is not the version you are expecting, you need to either uninstall the incorrect version of Postgres if you can, or update your PATH environmental variable to have the correct path of pg_config listed first, that is, by prepending the full path:

export PATH = /usr/local/Cellar/postgresql/11.0/bin:$PATH

Then, reinstall TimescaleDB and it should find the correct installation path.

ERROR: could not access file "timescaledb-<version>": No such file or directory

If the error occurs immediately after updating your version of TimescaleDB and the file mentioned is from the previous version, it is probably due to an incomplete update process. Within the greater Postgres server instance, each database that has TimescaleDB installed needs to be updated with the SQL command ALTER EXTENSION timescaledb UPDATE; while connected to that database. Otherwise, the database looks for the previous version of the timescaledb files.

See our update docs for more info.

Scheduled jobs stop running

Your scheduled jobs might stop running for various reasons. On self-hosted TimescaleDB, you can fix this by restarting background workers:

SELECT _timescaledb_internal.restart_background_workers();

On Tiger Cloud and Managed Service for TimescaleDB, restart background workers by doing one of the following:

• Run SELECT timescaledb_pre_restore(), followed by SELECT timescaledb_post_restore().
• Power the service off and on again. This might cause a downtime of a few minutes while the service restores from backup and replays the write-ahead log.

Failed to start a background worker

You might see this error message in the logs if background workers aren't properly configured:

"<TYPE_OF_BACKGROUND_JOB>": failed to start a background worker

To fix this error, make sure that max_worker_processes, max_parallel_workers, and timescaledb.max_background_workers are properly set. timescaledb.max_background_workers should equal the number of databases plus the number of concurrent background workers. max_worker_processes should equal the sum of timescaledb.max_background_workers and max_parallel_workers.

For more information, see the worker configuration docs.

Cannot compress chunk

You might see this error message when trying to compress a chunk if the permissions for the compressed hypertable are corrupt.

tsdb=> SELECT compress_chunk('_timescaledb_internal._hyper_65_587239_chunk');
ERROR: role 149910 was concurrently dropped

This can be caused if you dropped a user for the hypertable before TimescaleDB 2.5. For this case, the user would be removed from pg_authid but not revoked from the compressed table.

As a result, the compressed table contains permission items that refer to numerical values rather than existing users (see below for how to find the compressed hypertable from a normal hypertable):

tsdb=> \dp _timescaledb_internal._compressed_hypertable_2
                                 Access privileges
 Schema |     Name     | Type  |  Access privileges  | Column privileges | Policies
--------+--------------+-------+---------------------+-------------------+----------
 public | transactions | table | mats=arwdDxt/mats  +|                   |
        |              |       | wizard=arwdDxt/mats+|                   |
        |              |       | 149910=r/mats       |                   |
(1 row)

This means that the relacl column of pg_class needs to be updated and the offending user removed, but it is not possible to drop a user by numerical value. Instead, you can use the internal function repair_relation_acls in the _timescaledb_function schema:

tsdb=> CALL _timescaledb_functions.repair_relation_acls();

This requires superuser privileges (since you're modifying the pg_class table) and that it removes any user not present in pg_authid from all tables, so use with caution.

The permissions are usually corrupted for the hypertable as well, but not always, so it is better to look at the compressed hypertable to see if the problem is present. To find the compressed hypertable for an associated hypertable (readings in this case):

tsdb=> select ht.table_name,
tsdb->        (select format('%I.%I', schema_name, table_name)::regclass
tsdb->           from _timescaledb_catalog.hypertable
tsdb->			where ht.compressed_hypertable_id = id) as compressed_table
tsdb->   from _timescaledb_catalog.hypertable ht
tsdb->  where table_name = 'readings';
  format  |                     format
----------+------------------------------------------------
 readings | _timescaledb_internal._compressed_hypertable_2
(1 row)

Getting more information

EXPLAINing query performance

Postgres's EXPLAIN feature allows users to understand the underlying query plan that Postgres uses to execute a query. There are multiple ways that Postgres can execute a query: for example, a query might be fulfilled using a slow sequence scan or a much more efficient index scan. The choice of plan depends on what indexes are created on the table, the statistics that Postgres has about your data, and various planner settings. The EXPLAIN output let's you know which plan Postgres is choosing for a particular query. Postgres has a in-depth explanation of this feature.

To understand the query performance on a hypertable, we suggest first making sure that the planner statistics and table maintenance is up-to-date on the hypertable by running VACUUM ANALYZE <your-hypertable>;. Then, we suggest running the following version of EXPLAIN:

EXPLAIN (ANALYZE on, BUFFERS on) <original query>;

If you suspect that your performance issues are due to slow IOs from disk, you can get even more information by enabling the track_io_timing variable with SET track_io_timing = 'on'; before running the above EXPLAIN.

Dump TimescaleDB meta data

To help when asking for support and reporting bugs, TimescaleDB includes a SQL script that outputs metadata from the internal TimescaleDB tables as well as version information. The script is available in the source distribution in scripts/ but can also be downloaded separately. To use it, run:

psql [your connect flags] -d your_timescale_db < dump_meta_data.sql > dumpfile.txt

and then inspect dump_file.txt before sending it together with a bug report or support question.

Debugging background jobs

By default, background workers do not print a lot of information about execution. The reason for this is to avoid writing a lot of debug information to the Postgres log unless necessary.

To aid in debugging the background jobs, it is possible to increase the log level of the background workers without having to restart the server by setting the timescaledb.bgw_log_level GUC and reloading the configuration.

ALTER SYSTEM SET timescaledb.bgw_log_level TO 'DEBUG1';
SELECT pg_reload_conf();

This variable is set to the value of log_min_messages by default, which typically is WARNING. If the value of log_min_messages is changed in the configuration file, it is used for timescaledb.bgw_log_level when starting the workers.

Both ALTER SYSTEM and pg_reload_conf() require superuser privileges by default. Grant EXECUTE permissions to pg_reload_conf() and ALTER SYSTEM privileges to timescaledb.bgw_log_level if you want this to work for a non-superuser.

Since ALTER SYSTEM privileges only exist on Postgres 15 and later, the necessary grants for executing these statements only exist on Tiger Cloud for Postgres 15 or later.

Debug level 1

The amount of information printed at each level varies between jobs, but the information printed at DEBUG1 is currently shown below.

Source	Event
All jobs	Job exit with runtime information
All jobs	Job scheduled for fast restart
Custom job	Execution started
Recompression job	Recompression job completed
Reorder job	Chunk reorder completed
Reorder job	Chunk reorder started
Scheduler	New jobs discovered and added to scheduled jobs list
Scheduler	Scheduling job for launch

Debug level 2

The amount of information printed at each level varies between jobs, but the information printed at DEBUG2 is currently shown below.

Note that all messages at level DEBUG1 are also printed when you set the log level to DEBUG2, which is normal Postgres behaviour.

Source	Event
All jobs	Job found in jobs table
All jobs	Job starting execution
Scheduler	Scheduled jobs list update started
Scheduler	Scheduler dispatching job

Debug level 5

Source	Event
Scheduler	Scheduled wake up
Scheduler	Scheduler delayed in dispatching job

hypertable chunks are not discoverable by the Postgres CDC service

hypertables require special handling for CDC support. Newly created chunks are not not published, which means they are not discoverable by the CDC service. To fix this problem, use the following trigger to automatically publishe newly created chunks on the replication slot. Please be aware that TimescaleDB does not provide full CDC support.

CREATE OR REPLACE FUNCTION ddl_end_trigger_func() RETURNS EVENT_TRIGGER AS
$$
DECLARE
    r RECORD;
    pub NAME;
BEGIN
    FOR r IN SELECT * FROM pg_event_trigger_ddl_commands()
    LOOP
        SELECT pubname INTO pub
        FROM pg_inherits
        JOIN _timescaledb_catalog.hypertable ht
            ON inhparent = format('%I.%I', ht.schema_name, ht.table_name)::regclass
        JOIN pg_publication_tables
            ON schemaname = ht.schema_name AND tablename = ht.table_name
        WHERE inhrelid = r.objid;

        IF NOT pub IS NULL THEN
            EXECUTE format('ALTER PUBLICATION %s ADD TABLE %s', pub, r.objid::regclass);
        END IF;
    END LOOP;
END;
$$ LANGUAGE plpgsql;

CREATE EVENT TRIGGER ddl_end_trigger
ON ddl_command_end WHEN TAG IN ('CREATE TABLE') EXECUTE FUNCTION ddl_end_trigger_func();

===== PAGE: https://docs.tigerdata.com/use-timescale/compression/ =====

Compression

Old API since TimescaleDB v2.18.0 Replaced by hypercore.

Time-series data can be compressed to reduce the amount of storage required, and increase the speed of some queries. This is a cornerstone feature of TimescaleDB. When new data is added to your database, it is in the form of uncompressed rows. TimescaleDB uses a built-in job scheduler to convert this data to the form of compressed columns. This occurs across chunks of TimescaleDB hypertables.

===== PAGE: https://docs.tigerdata.com/tutorials/real-time-analytics-transport/ =====

Analytics on transport and geospatial data

Real-time analytics refers to the process of collecting, analyzing, and interpreting data instantly as it is generated. This approach enables you track and monitor activity, and make decisions based on real-time insights on data stored in a Tiger Cloud service.

This page shows you how to integrate Grafana with a Tiger Cloud service and make insights based on visualization of data optimized for size and speed in the columnstore.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install and run self-managed Grafana, or sign up for Grafana Cloud.

Optimize time-series data in hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Import time-series data into a hypertable

   1. Unzip nyc_data.tar.gz to a <local folder>.

      This test dataset contains historical data from New York's yellow taxi network.

      To import up to 100GB of data directly from your current Postgres-based database, migrate with downtime using native Postgres tooling. To seamlessly import 100GB-10TB+ of data, use the live migration tooling supplied by Tiger Data. To add data from non-Postgres data sources, see Import and ingest data.

   2. In Terminal, navigate to <local folder> and update the following string with your connection details to connect to your service.

      psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>?sslmode=require"
      

   3. Create an optimized hypertable for your time-series data:

      1. Create a hypertable with hypercore enabled by default for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data.

         In your sql client, run the following command:

         CREATE TABLE "rides"(
           vendor_id TEXT,
           pickup_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
           dropoff_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
           passenger_count NUMERIC,
           trip_distance NUMERIC,
           pickup_longitude  NUMERIC,
           pickup_latitude   NUMERIC,
           rate_code         INTEGER,
           dropoff_longitude NUMERIC,
           dropoff_latitude  NUMERIC,
           payment_type INTEGER,
           fare_amount NUMERIC,
           extra NUMERIC,
           mta_tax NUMERIC,
           tip_amount NUMERIC,
           tolls_amount NUMERIC,
           improvement_surcharge NUMERIC,
           total_amount NUMERIC
         ) WITH (
           tsdb.hypertable,
           tsdb.partition_column='pickup_datetime',
           tsdb.create_default_indexes=false,
           tsdb.segmentby='vendor_id',
           tsdb.orderby='pickup_datetime DESC'
         );
         

         If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

      2. Add another dimension to partition your hypertable more efficiently:

         SELECT add_dimension('rides', by_hash('payment_type', 2));
         

      3. Create an index to support efficient queries by vendor, rate code, and passenger count:

         CREATE INDEX ON rides (vendor_id, pickup_datetime DESC);
         CREATE INDEX ON rides (rate_code, pickup_datetime DESC);
         CREATE INDEX ON rides (passenger_count, pickup_datetime DESC);
         

   4. Create Postgres tables for relational data:

      1. Add a table to store the payment types data:

         CREATE TABLE IF NOT EXISTS "payment_types"(
           payment_type INTEGER,
           description TEXT
         );
         INSERT INTO payment_types(payment_type, description) VALUES
           (1, 'credit card'),
           (2, 'cash'),
           (3, 'no charge'),
           (4, 'dispute'),
           (5, 'unknown'),
           (6, 'voided trip');
         

      2. Add a table to store the rates data:

         CREATE TABLE IF NOT EXISTS "rates"(
          rate_code   INTEGER,
          description TEXT
         );
         INSERT INTO rates(rate_code, description) VALUES
          (1, 'standard rate'),
          (2, 'JFK'),
          (3, 'Newark'),
          (4, 'Nassau or Westchester'),
          (5, 'negotiated fare'),
          (6, 'group ride');
         

   5. Upload the dataset to your service

      \COPY rides FROM nyc_data_rides.csv CSV;
      

2. Have a quick look at your data

   You query hypertables in exactly the same way as you would a relational Postgres table. Use one of the following SQL editors to run a query and see the data you uploaded:

   • Data mode: write queries, visualize data, and share your results in Tiger Cloud Console for all your Tiger Cloud services.
   • SQL editor: write, fix, and organize SQL faster and more accurately in Tiger Cloud Console for a Tiger Cloud service.
   • psql: easily run queries on your Tiger Cloud services or self-hosted TimescaleDB deployment from Terminal.

   For example:

   • Display the number of rides for each fare type:

     SELECT rate_code, COUNT(vendor_id) AS num_trips
     FROM rides
     WHERE pickup_datetime < '2016-01-08'
     GROUP BY rate_code
     ORDER BY rate_code;
     

     This simple query runs in 3 seconds. You see something like:

     rate_code	num_trips
     1	2266401
     2	54832
     3	4126
     4	967
     5	7193
     6	17
     99	42

   • To select all rides taken in the first week of January 2016, and return the total number of trips taken for each rate code:

     SELECT rates.description, COUNT(vendor_id) AS num_trips
     FROM rides
     JOIN rates ON rides.rate_code = rates.rate_code
     WHERE pickup_datetime < '2016-01-08'
     GROUP BY rates.description
     ORDER BY LOWER(rates.description);
     

     On this large amount of data, this analytical query on data in the rowstore takes about 59 seconds. You see something like:

     description	num_trips
     group ride	17
     JFK	54832
     Nassau or Westchester	967
     negotiated fare	7193
     Newark	4126
     standard rate	2266401

Optimize your data for real-time analytics

When TimescaleDB converts a chunk to the columnstore, it automatically creates a different schema for your data. TimescaleDB creates and uses custom indexes to incorporate the segmentby and orderby parameters when you write to and read from the columstore.

To increase the speed of your analytical queries by a factor of 10 and reduce storage costs by up to 90%, convert data to the columnstore:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed. You can also connect to your serviceusing psql.

2. Add a policy to convert chunks to the columnstore at a specific time interval

   For example, convert data older than 8 days old to the columstore:

   CALL add_columnstore_policy('rides', INTERVAL '8 days');
   

   See add_columnstore_policy.

   The data you imported for this tutorial is from 2016, it was already added to the columnstore by default. However, you get the idea. To see the space savings in action, follow Try the key Tiger Data features.

Just to hit this one home, by converting cooling data to the columnstore, you have increased the speed of your analytical queries by a factor of 10, and reduced storage by up to 90%.

Connect Grafana to Tiger Cloud

To visualize the results of your queries, enable Grafana to read the data in your service:

1. Log in to Grafana

   In your browser, log in to either:

   • Self-hosted Grafana: at http://localhost:3000/. The default credentials are admin, admin.
   • Grafana Cloud: use the URL and credentials you set when you created your account.

2. Add your service as a data source

   1. Open Connections > Data sources, then click Add new data source.

   2. Select PostgreSQL from the list.

   3. Configure the connection:

      • Host URL, Database name, Username, and Password

        Configure using your connection details. Host URL is in the format <host>:<port>.

      • TLS/SSL Mode: select require.

      • PostgreSQL options: enable TimescaleDB.

      • Leave the default setting for all other fields.

   4. Click Save & test.

   Grafana checks that your details are set correctly.

Monitor performance over time

A Grafana dashboard represents a view into the performance of a system, and each dashboard consists of one or more panels, which represent information about a specific metric related to that system.

To visually monitor the volume of taxi rides over time:

1. Create the dashboard

   1. On the Dashboards page, click New and select New dashboard.

   2. Click Add visualization.

   3. Select the data source that connects to your Tiger Cloud service. The Time series visualization is chosen by default.

   4. In the Queries section, select Code, then select Time series in Format.

   5. Select the data range for your visualization: the data set is from 2016. Click the date range above the panel and set:

      • From: 2016-01-01 01:00:00
      • To: 2016-01-30 01:00:00

2. Combine TimescaleDB and Grafana functionality to analyze your data

   Combine a TimescaleDB time_bucket, with the Grafana _timefilter() function to set the pickup_datetime column as the filtering range for your visualizations.

   SELECT
     time_bucket('1 day', pickup_datetime) AS "time",
     COUNT(*)
   FROM rides
   WHERE _timeFilter(pickup_datetime)
   GROUP BY time
   ORDER BY time;
   

   This query groups the results by day and orders them by time.

   

3. Click Save dashboard

Optimize revenue potential

Having all this data is great but how do you use it? Monitoring data is useful to check what has happened, but how can you analyse this information to your advantage? This section explains how to create a visualization that shows how you can maximize potential revenue.

Set up your data for geospatial queries

To add geospatial analysis to your ride count visualization, you need geospatial data to work out which trips originated where. As TimescaleDB is compatible with all Postgres extensions, use PostGIS to slice data by time and location.

1. Connect to your Tiger Cloud service and add the PostGIS extension:

   CREATE EXTENSION postgis;
   

2. Add geometry columns for pick up and drop off locations:

   ALTER TABLE rides ADD COLUMN pickup_geom geometry(POINT,2163);
   ALTER TABLE rides ADD COLUMN dropoff_geom geometry(POINT,2163);
   

3. Convert the latitude and longitude points into geometry coordinates that work with PostGIS:

   UPDATE rides SET pickup_geom = ST_Transform(ST_SetSRID(ST_MakePoint(pickup_longitude,pickup_latitude),4326),2163),
      dropoff_geom = ST_Transform(ST_SetSRID(ST_MakePoint(dropoff_longitude,dropoff_latitude),4326),2163);
   

   This updates 10,906,860 rows of data on both columns, it takes a while. Coffee is your friend.

Visualize the area where you can make the most money

In this section you visualize a query that returns rides longer than 5 miles for trips taken within 2 km of Times Square. The data includes the distance travelled and is GROUP BY trip_distance and location so that Grafana can plot the data properly.

This enables you to see where a taxi driver is most likely to pick up a passenger who wants a longer ride, and make more money.

1. Create a geolocalization dashboard

   1. In Grafana, create a new dashboard that is connected to your Tiger Cloud service data source with a Geomap visualization.

   2. In the Queries section, select Code, then select the Time series Format.

      

   3. To find rides longer than 5 miles in Manhattan, paste the following query:

      SELECT time_bucket('5m', rides.pickup_datetime) AS time,
             rides.trip_distance AS value,
             rides.pickup_latitude AS latitude,
             rides.pickup_longitude AS longitude
      FROM rides
      WHERE rides.pickup_datetime BETWEEN '2016-01-01T01:41:55.986Z' AND '2016-01-01T07:41:55.986Z' AND
        ST_Distance(pickup_geom,
                    ST_Transform(ST_SetSRID(ST_MakePoint(-73.9851,40.7589),4326),2163)
        ) < 2000
      GROUP BY time,
               rides.trip_distance,
               rides.pickup_latitude,
               rides.pickup_longitude
      ORDER BY time
      LIMIT 500;
      

      You see a world map with a dot on New York.

   4. Zoom into your map to see the visualization clearly.

2. Customize the visualization

   1. In the Geomap options, under Map Layers, click + Add layer and select Heatmap. You now see the areas where a taxi driver is most likely to pick up a passenger who wants a longer ride, and make more money.

      

You have integrated Grafana with a Tiger Cloud service and made insights based on visualization of your data.

===== PAGE: https://docs.tigerdata.com/tutorials/real-time-analytics-energy-consumption/ =====

Real-time analytics with Tiger Cloud and Grafana

Energy providers understand that customers tend to lose patience when there is not enough power for them to complete day-to-day activities. Task one is keeping the lights on. If you are transitioning to renewable energy, it helps to know when you need to produce energy so you can choose a suitable energy source.

Real-time analytics refers to the process of collecting, analyzing, and interpreting data instantly as it is generated. This approach enables you to track and monitor activity, make the decisions based on real-time insights on data stored in a Tiger Cloud service and keep those lights on.

Grafana is a popular data visualization tool that enables you to create customizable dashboards and effectively monitor your systems and applications.

This page shows you how to integrate Grafana with a Tiger Cloud service and make insights based on visualization of data optimized for size and speed in the columnstore.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install and run self-managed Grafana, or sign up for Grafana Cloud.

Optimize time-series data in hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Import time-series data into a hypertable

   1. Unzip metrics.csv.gz to a <local folder>.

      This test dataset contains energy consumption data.

      To import up to 100GB of data directly from your current Postgres based database, migrate with downtime using native Postgres tooling. To seamlessly import 100GB-10TB+ of data, use the live migration tooling supplied by Tiger Data. To add data from non-Postgres data sources, see Import and ingest data.

   2. In Terminal, navigate to <local folder> and update the following string with your connection details to connect to your service.

      psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>?sslmode=require"
      

   3. Create an optimized hypertable for your time-series data:

      1. Create a hypertable with hypercore enabled by default for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data.

         In your sql client, run the following command:

         CREATE TABLE "metrics"(
           created timestamp with time zone default now() not null,
           type_id integer                                not null,
           value   double precision                       not null
         ) WITH (
           tsdb.hypertable,
           tsdb.partition_column='created',
           tsdb.segmentby = 'type_id',
           tsdb.orderby = 'created DESC'
         );
         

         If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

   4. Upload the dataset to your service

      \COPY metrics FROM metrics.csv CSV;
      

2. Have a quick look at your data

   You query hypertables in exactly the same way as you would a relational Postgres table. Use one of the following SQL editors to run a query and see the data you uploaded:

   • Data mode: write queries, visualize data, and share your results in Tiger Cloud Console for all your Tiger Cloud services.
   • SQL editor: write, fix, and organize SQL faster and more accurately in Tiger Cloud Console for a Tiger Cloud service.
   • psql: easily run queries on your Tiger Cloud services or self-hosted TimescaleDB deployment from Terminal.

   SELECT time_bucket('1 day', created, 'Europe/Berlin') AS "time",
   round((last(value, created) - first(value, created)) * 100.) / 100. AS value
   FROM metrics
   WHERE type_id = 5
   GROUP BY 1;
   

   On this amount of data, this query on data in the rowstore takes about 3.6 seconds. You see something like:

   Time	value
   2023-05-29 22:00:00+00	23.1
   2023-05-28 22:00:00+00	19.5
   2023-05-30 22:00:00+00	25
   2023-05-31 22:00:00+00	8.1

Optimize your data for real-time analytics

When TimescaleDB converts a chunk to the columnstore, it automatically creates a different schema for your data. TimescaleDB creates and uses custom indexes to incorporate the segmentby and orderby parameters when you write to and read from the columstore.

To increase the speed of your analytical queries by a factor of 10 and reduce storage costs by up to 90%, convert data to the columnstore:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed. You can also connect to your service using psql.

2. Add a policy to convert chunks to the columnstore at a specific time interval

   For example, 60 days after the data was added to the table:

   CALL add_columnstore_policy('metrics', INTERVAL '8 days');
   

   See add_columnstore_policy.

3. Faster analytical queries on data in the columnstore

   Now run the analytical query again:

    SELECT time_bucket('1 day', created, 'Europe/Berlin') AS "time",
    round((last(value, created) - first(value, created)) * 100.) / 100. AS value
    FROM metrics
    WHERE type_id = 5
    GROUP BY 1;
   

   On this amount of data, this analytical query on data in the columnstore takes about 250ms.

Just to hit this one home, by converting cooling data to the columnstore, you have increased the speed of your analytical queries by a factor of 10, and reduced storage by up to 90%.

Write fast analytical queries

Aggregation is a way of combining data to get insights from it. Average, sum, and count are all examples of simple aggregates. However, with large amounts of data aggregation slows things down, quickly. Continuous aggregates are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background.

By default, querying continuous aggregates provides you with real-time data. Pre-aggregated data from the materialized view is combined with recent data that hasn't been aggregated yet. This gives you up-to-date results on every query.

You create continuous aggregates on uncompressed data in high-performance storage. They continue to work on data in the columnstore and rarely accessed data in tiered storage. You can even create continuous aggregates on top of your continuous aggregates.

1. Monitor energy consumption on a day-to-day basis

   1. Create a continuous aggregate kwh_day_by_day for energy consumption:

      CREATE MATERIALIZED VIEW kwh_day_by_day(time, value)
         with (timescaledb.continuous) as
      SELECT time_bucket('1 day', created, 'Europe/Berlin') AS "time",
             round((last(value, created) - first(value, created)) * 100.) / 100. AS value
      FROM metrics
      WHERE type_id = 5
      GROUP BY 1;
      

   2. Add a refresh policy to keep kwh_day_by_day up-to-date:

      SELECT add_continuous_aggregate_policy('kwh_day_by_day',
         start_offset => NULL,
         end_offset => INTERVAL '1 hour',
         schedule_interval => INTERVAL '1 hour');
      

2. Monitor energy consumption on an hourly basis

   1. Create a continuous aggregate kwh_hour_by_hour for energy consumption:

      CREATE MATERIALIZED VIEW kwh_hour_by_hour(time, value)
        with (timescaledb.continuous) as
      SELECT time_bucket('01:00:00', metrics.created, 'Europe/Berlin') AS "time",
             round((last(value, created) - first(value, created)) * 100.) / 100. AS value
      FROM metrics
      WHERE type_id = 5
      GROUP BY 1;
      

   2. Add a refresh policy to keep the continuous aggregate up-to-date:

   SELECT add_continuous_aggregate_policy('kwh_hour_by_hour',
    start_offset => NULL,
       end_offset => INTERVAL '1 hour',
       schedule_interval => INTERVAL '1 hour');
   

3. Analyze your data

   Now you have made continuous aggregates, it could be a good idea to use them to perform analytics on your data. For example, to see how average energy consumption changes during weekdays over the last year, run the following query:

     WITH per_day AS (
      SELECT
        time,
        value
      FROM kwh_day_by_day
      WHERE "time" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year'
      ORDER BY 1
     ), daily AS (
         SELECT
            to_char(time, 'Dy') as day,
            value
         FROM per_day
     ), percentile AS (
         SELECT
             day,
             approx_percentile(0.50, percentile_agg(value)) as value
         FROM daily
         GROUP BY 1
         ORDER BY 1
     )
     SELECT
         d.day,
         d.ordinal,
         pd.value
     FROM unnest(array['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']) WITH ORDINALITY AS d(day, ordinal)
     LEFT JOIN percentile pd ON lower(pd.day) = lower(d.day);
   

   You see something like:

   day	ordinal	value
   Mon	2	23.08078714975423
   Sun	1	19.511430831944395
   Tue	3	25.003118897837307
   Wed	4	8.09300571759772

Connect Grafana to Tiger Cloud

To visualize the results of your queries, enable Grafana to read the data in your service:

1. Log in to Grafana

   In your browser, log in to either:

   • Self-hosted Grafana: at http://localhost:3000/. The default credentials are admin, admin.
   • Grafana Cloud: use the URL and credentials you set when you created your account.

2. Add your service as a data source

   1. Open Connections > Data sources, then click Add new data source.

   2. Select PostgreSQL from the list.

   3. Configure the connection:

      • Host URL, Database name, Username, and Password

        Configure using your connection details. Host URL is in the format <host>:<port>.

      • TLS/SSL Mode: select require.

      • PostgreSQL options: enable TimescaleDB.

      • Leave the default setting for all other fields.

   4. Click Save & test.

   Grafana checks that your details are set correctly.

Visualize energy consumption

A Grafana dashboard represents a view into the performance of a system, and each dashboard consists of one or more panels, which represent information about a specific metric related to that system.

To visually monitor the volume of energy consumption over time:

1. Create the dashboard

   1. On the Dashboards page, click New and select New dashboard.

   2. Click Add visualization, then select the data source that connects to your Tiger Cloud service and the Bar chart visualization.

      

   3. In the Queries section, select Code, then run the following query based on your continuous aggregate:

      WITH per_hour AS (
      SELECT
      time,
      value
      FROM kwh_hour_by_hour
      WHERE "time" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year'
      ORDER BY 1
      ), hourly AS (
       SELECT
            extract(HOUR FROM time) * interval '1 hour' as hour,
            value
       FROM per_hour
      )
      SELECT
          hour,
          approx_percentile(0.50, percentile_agg(value)) as median,
          max(value) as maximum
      FROM hourly
      GROUP BY 1
      ORDER BY 1;
      

      This query averages the results for households in a specific time zone by hour and orders them by time. Because you use a continuous aggregate, this data is always correct in real time.

      

      You see that energy consumption is highest in the evening and at breakfast time. You also know that the wind drops off in the evening. This data proves that you need to supply a supplementary power source for peak times, or plan to store energy during the day for peak times.

2. Click Save dashboard

You have integrated Grafana with a Tiger Cloud service and made insights based on visualization of your data.

===== PAGE: https://docs.tigerdata.com/tutorials/simulate-iot-sensor-data/ =====

Simulate an IoT sensor dataset

The Internet of Things (IoT) describes a trend where computing capabilities are embedded into IoT devices. That is, physical objects, ranging from light bulbs to oil wells. Many IoT devices collect sensor data about their environment and generate time-series datasets with relational metadata.

It is often necessary to simulate IoT datasets. For example, when you are testing a new system. This tutorial shows how to simulate a basic dataset in your Tiger Cloud service, and then run simple queries on it.

To simulate a more advanced dataset, see Time-series Benchmarking Suite (TSBS).

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Simulate a dataset

To simulate a dataset, run the following queries:

1. Create the sensors table:

   CREATE TABLE sensors(
     id SERIAL PRIMARY KEY,
     type VARCHAR(50),
     location VARCHAR(50)
   );
   

2. Create the sensor_data hypertable

   CREATE TABLE sensor_data (
     time TIMESTAMPTZ NOT NULL,
     sensor_id INTEGER,
     temperature DOUBLE PRECISION,
     cpu DOUBLE PRECISION,
     FOREIGN KEY (sensor_id) REFERENCES sensors (id)
   ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='time'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

3. Populate the sensors table:

   INSERT INTO sensors (type, location) VALUES
   ('a','floor'),
   ('a', 'ceiling'),
   ('b','floor'),
   ('b', 'ceiling');
   

4. Verify that the sensors have been added correctly:

   SELECT * FROM sensors;
   

   Sample output:

    id | type | location
   ----+------+----------
     1 | a    | floor
     2 | a    | ceiling
     3 | b    | floor
     4 | b    | ceiling
   (4 rows)
   

5. Generate and insert a dataset for all sensors:

   INSERT INTO sensor_data (time, sensor_id, cpu, temperature)
   SELECT
     time,
     sensor_id,
     random() AS cpu,
     random()*100 AS temperature
   FROM generate_series(now() - interval '24 hour', now(), interval '5 minute') AS g1(time), generate_series(1,4,1) AS g2(sensor_id);
   

6. Verify the simulated dataset:

   SELECT * FROM sensor_data ORDER BY time;
   

   Sample output:

                time              | sensor_id |    temperature     |         cpu
   -------------------------------+-----------+--------------------+---------------------
    2020-03-31 15:56:25.843575+00 |         1 |   6.86688972637057 |   0.682070567272604
    2020-03-31 15:56:40.244287+00 |         2 |    26.589260622859 |   0.229583469685167
    2030-03-31 15:56:45.653115+00 |         3 |   79.9925176426768 |   0.457779890391976
    2020-03-31 15:56:53.560205+00 |         4 |   24.3201029952615 |   0.641885648947209
    2020-03-31 16:01:25.843575+00 |         1 |   33.3203678019345 |  0.0159163917414844
    2020-03-31 16:01:40.244287+00 |         2 |   31.2673618085682 |   0.701185956597328
    2020-03-31 16:01:45.653115+00 |         3 |   85.2960689924657 |   0.693413889966905
    2020-03-31 16:01:53.560205+00 |         4 |   79.4769988860935 |   0.360561791341752
   ...
   

Run basic queries

After you simulate a dataset, you can run some basic queries on it. For example:

• Average temperature and CPU by 30-minute windows:

  SELECT
    time_bucket('30 minutes', time) AS period,
    AVG(temperature) AS avg_temp,
    AVG(cpu) AS avg_cpu
  FROM sensor_data
  GROUP BY period;
  

  Sample output:

           period         |     avg_temp     |      avg_cpu
  ------------------------+------------------+-------------------
   2020-03-31 19:00:00+00 | 49.6615830013373 | 0.477344429974134
   2020-03-31 22:00:00+00 | 58.8521540844037 | 0.503637770501276
   2020-03-31 16:00:00+00 | 50.4250325243144 | 0.511075591299838
   2020-03-31 17:30:00+00 | 49.0742547437549 | 0.527267253802468
   2020-04-01 14:30:00+00 | 49.3416377226822 | 0.438027751864865
   ...
  

• Average and last temperature, average CPU by 30-minute windows:

  SELECT
    time_bucket('30 minutes', time) AS period,
    AVG(temperature) AS avg_temp,
    last(temperature, time) AS last_temp,
    AVG(cpu) AS avg_cpu
  FROM sensor_data
  GROUP BY period;
  

  Sample output:

           period         |     avg_temp     |    last_temp     |      avg_cpu
  ------------------------+------------------+------------------+-------------------
   2020-03-31 19:00:00+00 | 49.6615830013373 | 84.3963081017137 | 0.477344429974134
   2020-03-31 22:00:00+00 | 58.8521540844037 | 76.5528806950897 | 0.503637770501276
   2020-03-31 16:00:00+00 | 50.4250325243144 | 43.5192013625056 | 0.511075591299838
   2020-03-31 17:30:00+00 | 49.0742547437549 |  22.740753274411 | 0.527267253802468
   2020-04-01 14:30:00+00 | 49.3416377226822 | 59.1331578791142 | 0.438027751864865
  ...
  

• Query the metadata:

  SELECT
    sensors.location,
    time_bucket('30 minutes', time) AS period,
    AVG(temperature) AS avg_temp,
    last(temperature, time) AS last_temp,
    AVG(cpu) AS avg_cpu
  FROM sensor_data JOIN sensors on sensor_data.sensor_id = sensors.id
  GROUP BY period, sensors.location;
  

  Sample output:

   location |         period         |     avg_temp     |     last_temp     |      avg_cpu
  ----------+------------------------+------------------+-------------------+-------------------
   ceiling  | 20120-03-31 15:30:00+00 | 25.4546818090603 |  24.3201029952615 | 0.435734559316188
   floor    | 2020-03-31 15:30:00+00 | 43.4297036845237 |  79.9925176426768 |  0.56992522883229
   ceiling  | 2020-03-31 16:00:00+00 | 53.8454438598516 |  43.5192013625056 | 0.490728285357666
   floor    | 2020-03-31 16:00:00+00 | 47.0046211887772 |  23.0230117216706 |  0.53142289724201
   ceiling  | 2020-03-31 16:30:00+00 | 58.7817596504465 |  63.6621567420661 | 0.488188337767497
   floor    | 2020-03-31 16:30:00+00 |  44.611586847653 |  2.21919436007738 | 0.434762630766879
   ceiling  | 2020-03-31 17:00:00+00 | 35.7026890735142 |  42.9420990403742 | 0.550129583687522
   floor    | 2020-03-31 17:00:00+00 | 62.2794370166957 |  52.6636955793947 | 0.454323202022351
  ...
  

You have now successfully simulated and run queries on an IoT dataset.

===== PAGE: https://docs.tigerdata.com/tutorials/cookbook/ =====

Tiger Data cookbook

This page contains suggestions from the Tiger Data Community about how to resolve common issues. Use these code examples as guidance to work with your own data.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Hypertable recipes

This section contains recipes about hypertables.

Remove duplicates from an existing hypertable

Looking to remove duplicates from an existing hypertable? One method is to run a PARTITION BY query to get ROW_NUMBER() and then the ctid of rows where row_number>1. You then delete these rows. However, you need to check tableoid and ctid. This is because ctid is not unique and might be duplicated in different chunks. The following code example took 17 hours to process a table with 40 million rows:

CREATE OR REPLACE FUNCTION deduplicate_chunks(ht_name TEXT, partition_columns TEXT, bot_id INT DEFAULT NULL)
    RETURNS TABLE
            (
                chunk_schema  name,
                chunk_name    name,
                deleted_count INT
            )
AS
$$
DECLARE
    chunk         RECORD;
    where_clause  TEXT := '';
    deleted_count INT;
BEGIN
    IF bot_id IS NOT NULL THEN
        where_clause := FORMAT('WHERE bot_id = %s', bot_id);
    END IF;

    FOR chunk IN
        SELECT c.chunk_schema, c.chunk_name
        FROM timescaledb_information.chunks c
        WHERE c.hypertable_name = ht_name
        LOOP
            EXECUTE FORMAT('
            WITH cte AS (
                SELECT ctid,
                       ROW_NUMBER() OVER (PARTITION BY %s ORDER BY %s ASC) AS row_num,
                       *
                FROM %I.%I
                %s
            )
            DELETE FROM %I.%I
            WHERE ctid IN (
                SELECT ctid
                FROM cte
                WHERE row_num > 1
            )
            RETURNING 1;
        ', partition_columns, partition_columns, chunk.chunk_schema, chunk.chunk_name, where_clause, chunk.chunk_schema,
                           chunk.chunk_name)
                INTO deleted_count;

            RETURN QUERY SELECT chunk.chunk_schema, chunk.chunk_name, COALESCE(deleted_count, 0);
        END LOOP;
END
$$ LANGUAGE plpgsql;


SELECT *
FROM deduplicate_chunks('nudge_events', 'bot_id, session_id, nudge_id, time', 2540);

Shoutout to Mathias Ose and Christopher Piggott for this recipe.

Get faster JOIN queries with Common Table Expressions

Imagine there is a query that joins a hypertable to another table on a shared key:

    SELECT timestamp,
      FROM hypertable as h
      JOIN related_table as rt
        ON rt.id = h.related_table_id
     WHERE h.timestamp BETWEEN '2024-10-10 00:00:00' AND '2024-10-17 00:00:00'

If you run EXPLAIN on this query, you see that the query planner performs a NestedJoin between these two tables, which means querying the hypertable multiple times. Even if the hypertable is well indexed, if it is also large, the query will be slow. How do you force a once-only lookup? Use materialized Common Table Expressions (CTEs).

If you split the query into two parts using CTEs, you can materialize the hypertable lookup and force Postgres to perform it only once.

WITH cached_query AS materialized (
  SELECT *
    FROM hypertable
   WHERE BETWEEN '2024-10-10 00:00:00' AND '2024-10-17 00:00:00'
)
  SELECT *
    FROM cached_query as c
    JOIN related_table as rt
      ON rt.id = h.related_table_id

Now if you run EXPLAIN once again, you see that this query performs only one lookup. Depending on the size of your hypertable, this could result in a multi-hour query taking mere seconds.

Shoutout to Rowan Molony for this recipe.

IoT recipes

This section contains recipes for IoT issues:

Work with columnar IoT data

Narrow and medium width tables are a great way to store IoT data. A lot of reasons are outlined in Designing Your Database Schema: Wide vs. Narrow Postgres Tables.

One of the key advantages of narrow tables is that the schema does not have to change when you add new sensors. Another big advantage is that each sensor can sample at different rates and times. This helps support things like hysteresis, where new values are written infrequently unless the value changes by a certain amount.

Narrow table format example

Working with narrow table data structures presents a few challenges. In the IoT world one concern is that many data analysis approaches - including machine learning as well as more traditional data analysis - require that your data is resampled and synchronized to a common time basis. Fortunately, TimescaleDB provides you with hyperfunctions and other tools to help you work with this data.

An example of a narrow table format is:

ts	sensor_id	value
2024-10-31 11:17:30.000	1007	23.45

Typically you would couple this with a sensor table:

sensor_id	sensor_name	units
1007	temperature	degreesC
1012	heat_mode	on/off
1013	cooling_mode	on/off
1041	occupancy	number of people in room

A medium table retains the generic structure but adds columns of various types so that you can use the same table to store float, int, bool, or even JSON (jsonb) data:

ts	sensor_id	d	i	b	t	j
2024-10-31 11:17:30.000	1007	23.45	null	null	null	null
2024-10-31 11:17:47.000	1012	null	null	TRUE	null	null
2024-10-31 11:18:01.000	1041	null	4	null	null	null

To remove all-null entries, use an optional constraint such as:

    CONSTRAINT at_least_one_not_null
        CHECK ((d IS NOT NULL) OR (i IS NOT NULL) OR (b IS NOT NULL) OR (j IS NOT NULL) OR (t IS NOT NULL))

Get the last value of every sensor

There are several ways to get the latest value of every sensor. The following examples use the structure defined in Narrow table format example as a reference:

• SELECT DISTINCT ON
• JOIN LATERAL

SELECT DISTINCT ON

If you have a list of sensors, the easy way to get the latest value of every sensor is to use SELECT DISTINCT ON:

WITH latest_data AS (
    SELECT DISTINCT ON (sensor_id) ts, sensor_id, d
    FROM iot_data
    WHERE d is not null
      AND ts > CURRENT_TIMESTAMP - INTERVAL '1 week'  -- important
    ORDER BY sensor_id, ts DESC
)
SELECT
    sensor_id, sensors.name, ts, d
FROM latest_data
LEFT OUTER JOIN sensors ON latest_data.sensor_id = sensors.id
WHERE latest_data.d is not null
ORDER BY sensor_id, ts; -- Optional, for displaying results ordered by sensor_id

The common table expression (CTE) used above is not strictly necessary. However, it is an elegant way to join to the sensor list to get a sensor name in the output. If this is not something you care about, you can leave it out:

SELECT DISTINCT ON (sensor_id) ts, sensor_id, d
    FROM iot_data
    WHERE d is not null
      AND ts > CURRENT_TIMESTAMP - INTERVAL '1 week'  -- important
    ORDER BY sensor_id, ts DESC

It is important to take care when down-selecting this data. In the previous examples, the time that the query would scan back was limited. However, if there any sensors that have either not reported in a long time or in the worst case, never reported, this query devolves to a full table scan. In a database with 1000+ sensors and 41 million rows, an unconstrained query takes over an hour.

JOIN LATERAL

An alternative to SELECT DISTINCT ON is to use a JOIN LATERAL. By selecting your entire sensor list from the sensors table rather than pulling the IDs out using SELECT DISTINCT, JOIN LATERAL can offer some improvements in performance:

SELECT sensor_list.id, latest_data.ts, latest_data.d
FROM sensors sensor_list
    -- Add a WHERE clause here to downselect the sensor list, if you wish
LEFT JOIN LATERAL (
    SELECT ts, d
    FROM iot_data raw_data
    WHERE sensor_id = sensor_list.id
    ORDER BY ts DESC
    LIMIT 1
) latest_data ON true
WHERE latest_data.d is not null -- only pulling out float values ("d" column) in this example
  AND latest_data.ts > CURRENT_TIMESTAMP - interval '1 week' -- important
ORDER BY sensor_list.id, latest_data.ts;

Limiting the time range is important, especially if you have a lot of data. Best practice is to use these kinds of queries for dashboards and quick status checks. To query over a much larger time range, encapsulate the previous example into a materialized query that refreshes infrequently, perhaps once a day.

Shoutout to Christopher Piggott for this recipe.

===== PAGE: https://docs.tigerdata.com/tutorials/blockchain-query/ =====

Query the Bitcoin blockchain

The financial industry is extremely data-heavy and relies on real-time and historical data for decision-making, risk assessment, fraud detection, and market analysis. Tiger Data simplifies management of these large volumes of data, while also providing you with meaningful analytical insights and optimizing storage costs.

In this tutorial, you use Tiger Cloud to ingest, store, and analyze transactions on the Bitcoin blockchain.

Blockchains are, at their essence, a distributed database. The transactions in a blockchain are an example of time-series data. You can use TimescaleDB to query transactions on a blockchain, in exactly the same way as you might query time-series transactions in any other database.

Steps in this tutorial

This tutorial covers:

1. Ingest data into a service: set up and connect to a Tiger Cloud service, create tables and hypertables, and ingest data.
2. Query your data: obtain information, including finding the most recent transactions on the blockchain, and gathering information about the transactions using aggregation functions.
3. Compress your data using hypercore: compress data that is no longer needed for highest performance queries, but is still accessed regularly for real-time analytics.

When you've completed this tutorial, you can use the same dataset to Analyze the Bitcoin data, using TimescaleDB hyperfunctions.

===== PAGE: https://docs.tigerdata.com/tutorials/blockchain-analyze/ =====

Analyze the Bitcoin blockchain

The financial industry is extremely data-heavy and relies on real-time and historical data for decision-making, risk assessment, fraud detection, and market analysis. Tiger Data simplifies management of these large volumes of data, while also providing you with meaningful analytical insights and optimizing storage costs.

In this tutorial, you use Tiger Cloud to ingest, store, and analyze transactions on the Bitcoin blockchain.

Blockchains are, at their essence, a distributed database. The transactions in a blockchain are an example of time-series data. You can use TimescaleDB to query transactions on a blockchain, in exactly the same way as you might query time-series transactions in any other database.

Prerequisites

Before you begin, make sure you have:

• Signed up for a free Tiger Data account.
• Signed up for a Grafana account to graph your queries.

Steps in this tutorial

This tutorial covers:

1. Setting up your dataset
2. Querying your dataset

About analyzing the Bitcoin blockchain with Tiger Cloud

This tutorial uses a sample Bitcoin dataset to show you how to aggregate blockchain transaction data, and construct queries to analyze information from the aggregations. The queries in this tutorial help you determine if a cryptocurrency has a high transaction fee, shows any correlation between transaction volumes and fees, or if it's expensive to mine.

It starts by setting up and connecting to a Tiger Cloud service, create tables, and load data into the tables using psql. If you have already completed the beginner blockchain tutorial, then you already have the dataset loaded, and you can skip straight to the queries.

You then learn how to conduct analysis on your dataset using Timescale hyperfunctions. It walks you through creating a series of continuous aggregates, and querying the aggregates to analyze the data. You can also use those queries to graph the output in Grafana.

===== PAGE: https://docs.tigerdata.com/tutorials/financial-tick-data/ =====

Analyze financial tick data with TimescaleDB

The financial industry is extremely data-heavy and relies on real-time and historical data for decision-making, risk assessment, fraud detection, and market analysis. Tiger Data simplifies management of these large volumes of data, while also providing you with meaningful analytical insights and optimizing storage costs.

To analyze financial data, you can chart the open, high, low, close, and volume (OHLCV) information for a financial asset. Using this data, you can create candlestick charts that make it easier to analyze the price changes of financial assets over time. You can use candlestick charts to examine trends in stock, cryptocurrency, or NFT prices.

In this tutorial, you use real raw financial data provided by Twelve Data, create an aggregated candlestick view, query the aggregated data, and visualize the data in Grafana.

OHLCV data and candlestick charts

The financial sector regularly uses candlestick charts to visualize the price change of an asset. Each candlestick represents a time period, such as one minute or one hour, and shows how the asset's price changed during that time.

Candlestick charts are generated from the open, high, low, close, and volume data for each financial asset during the time period. This is often abbreviated as OHLCV:

• Open: opening price
• High: highest price
• Low: lowest price
• Close: closing price
• Volume: volume of transactions

TimescaleDB is well suited to storing and analyzing financial candlestick data, and many Tiger Data community members use it for exactly this purpose. Check out these stories from some Tiger Datacommunity members:

• How Trading Strategy built a data stack for crypto quant trading
• How Messari uses data to open the cryptoeconomy to everyone
• How I power a (successful) crypto trading bot with TimescaleDB

Steps in this tutorial

This tutorial shows you how to ingest real-time time-series data into a Tiger Cloud service:

1. Ingest data into a service: load data from Twelve Data into your TimescaleDB database.
2. Query your dataset: create candlestick views, query the aggregated data, and visualize the data in Grafana.
3. Compress your data using hypercore: learn how to store and query your financial tick data more efficiently using compression feature of TimescaleDB.

To create candlestick views, query the aggregated data, and visualize the data in Grafana, see the ingest real-time websocket data section.

===== PAGE: https://docs.tigerdata.com/tutorials/financial-ingest-real-time/ =====

Ingest real-time financial data using WebSocket

The financial industry is extremely data-heavy and relies on real-time and historical data for decision-making, risk assessment, fraud detection, and market analysis. Tiger Data simplifies management of these large volumes of data, while also providing you with meaningful analytical insights and optimizing storage costs.

This tutorial shows you how to ingest real-time time-series data into TimescaleDB using a websocket connection. The tutorial sets up a data pipeline to ingest real-time data from our data partner, Twelve Data. Twelve Data provides a number of different financial APIs, including stock, cryptocurrencies, foreign exchanges, and ETFs. It also supports websocket connections in case you want to update your database frequently. With websockets, you need to connect to the server, subscribe to symbols, and you can start receiving data in real-time during market hours.

When you complete this tutorial, you'll have a data pipeline set up that ingests real-time financial data into your Tiger Cloud.

This tutorial uses Python and the API wrapper library provided by Twelve Data.

Prerequisites

Before you begin, make sure you have:

• Signed up for a free Tiger Data account.
• Installed Python 3
• Signed up for Twelve Data. The free tier is perfect for this tutorial.
• Made a note of your Twelve Data API key.

Steps in this tutorial

This tutorial covers:

1. Setting up your dataset: Load data from Twelve Data into your TimescaleDB database.

2. Querying your dataset: Create candlestick views, query the aggregated data, and visualize the data in Grafana.

   This tutorial shows you how to ingest real-time time-series data into a Tiger Cloud service using a websocket connection. To create candlestick views, query the aggregated data, and visualize the data in Grafana.

About OHLCV data and candlestick charts

The financial sector regularly uses candlestick charts to visualize the price change of an asset. Each candlestick represents a time period, such as one minute or one hour, and shows how the asset's price changed during that time.

Candlestick charts are generated from the open, high, low, close, and volume data for each financial asset during the time period. This is often abbreviated as OHLCV:

• Open: opening price
• High: highest price
• Low: lowest price
• Close: closing price
• Volume: volume of transactions

TimescaleDB is well suited to storing and analyzing financial candlestick data, and many Tiger Datacommunity members use it for exactly this purpose.

===== PAGE: https://docs.tigerdata.com/api/hypertable/ =====

Hypertables and chunks

Tiger Cloud supercharges your real-time analytics by letting you run complex queries continuously, with near-zero latency. Under the hood, this is achieved by using hypertables—Postgres tables that automatically partition your time-series data by time and optionally by other dimensions. When you run a query, Tiger Cloud identifies the correct partition, called chunk, and runs the query on it, instead of going through the entire table.

Hypertables offer the following benefits:

• Efficient data management with automated partitioning by time: Tiger Cloud splits your data into chunks that hold data from a specific time range. For example, one day or one week. You can configure this range to better suit your needs.

• Better performance with strategic indexing: an index on time in the descending order is automatically created when you create a hypertable. More indexes are created on the chunk level, to optimize performance. You can create additional indexes, including unique indexes, on the columns you need.

• Faster queries with chunk skipping: Tiger Cloud skips the chunks that are irrelevant in the context of your query, dramatically reducing the time and resources needed to fetch results. Even more—you can enable chunk skipping on non-partitioning columns.

• Advanced data analysis with hyperfunctions: Tiger Cloud enables you to efficiently process, aggregate, and analyze significant volumes of data while maintaining high performance.

To top it all, there is no added complexity—you interact with hypertables in the same way as you would with regular Postgres tables. All the optimization magic happens behind the scenes.

Inheritance is not supported for hypertables and may lead to unexpected behavior.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

The hypertable workflow

Best practice for using a hypertable is to:

1. Create a hypertable

   Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data. For example:

   CREATE TABLE conditions (
      time        TIMESTAMPTZ       NOT NULL,
      location    TEXT              NOT NULL,
      device      TEXT              NOT NULL,
      temperature DOUBLE PRECISION  NULL,
      humidity    DOUBLE PRECISION  NULL
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time',
      tsdb.segmentby = 'device',
      tsdb.orderby = 'time DESC'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

2. Set the columnstore policy

   CALL add_columnstore_policy('conditions', after => INTERVAL '1d');
   

===== PAGE: https://docs.tigerdata.com/api/hypercore/ =====

Hypercore

Hypercore is a hybrid row-columnar storage engine in TimescaleDB. It is designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage, delivering the best of both worlds:

Hypercore solves the key challenges in real-time analytics:

• High ingest throughput
• Low-latency ingestion
• Fast query performance
• Efficient handling of data updates and late-arriving data
• Streamlined data management

Hypercore’s hybrid approach combines the benefits of row-oriented and column-oriented formats:

• Fast ingest with rowstore: new data is initially written to the rowstore, which is optimized for high-speed inserts and updates. This process ensures that real-time applications easily handle rapid streams of incoming data. Mutability—upserts, updates, and deletes happen seamlessly.

• Efficient analytics with columnstore: as the data cools and becomes more suited for analytics, it is automatically converted to the columnstore. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space.

• Faster queries on compressed data in columnstore: in the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. Combined with chunk skipping, this helps you save on storage costs and keeps your queries operating at lightning speed.

• Fast modification of compressed data in columnstore: just use SQL to add or modify data in the columnstore. TimescaleDB is optimized for superfast INSERT and UPSERT performance.

• Full mutability with transactional semantics: regardless of where data is stored, hypercore provides full ACID support. Like in a vanilla Postgres database, inserts and updates to the rowstore and columnstore are always consistent, and available to queries as soon as they are completed.

For an in-depth explanation of how hypertables and hypercore work, see the Data model.

Since TimescaleDB v2.18.0

Hypercore workflow

Best practice for using hypercore is to:

1. Enable columnstore

   Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data. For example:

   • Use CREATE TABLE for a hypertable

     CREATE TABLE crypto_ticks (
        "time" TIMESTAMPTZ,
        symbol TEXT,
        price DOUBLE PRECISION,
        day_volume NUMERIC
     ) WITH (
       tsdb.hypertable,
       tsdb.partition_column='time',
       tsdb.segmentby='symbol',
       tsdb.orderby='time DESC'
     );
     

     If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

   • Use ALTER MATERIALIZED VIEW for a continuous aggregate

     ALTER MATERIALIZED VIEW assets_candlestick_daily set (
        timescaledb.enable_columnstore = true,
        timescaledb.segmentby = 'symbol' );
     

2. Add a policy to move chunks to the columnstore at a specific time interval

   For example, 7 days after the data was added to the table:

   CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '7d');
   

   See add_columnstore_policy.

3. View the policies that you set or the policies that already exist

   SELECT * FROM timescaledb_information.jobs
   WHERE proc_name='policy_compression';
   

   See timescaledb_information.jobs.

You can also convert_to_columnstore and convert_to_rowstore manually for more fine-grained control over your data.

Limitations

Chunks in the columnstore have the following limitations:

• ROW LEVEL SECURITY is not supported on chunks in the columnstore.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/ =====

Continuous aggregates

In modern applications, data usually grows very quickly. This means that aggregating it into useful summaries can become very slow. If you are collecting data very frequently, you might want to aggregate your data into minutes or hours instead. For example, if an IoT device takes temperature readings every second, you might want to find the average temperature for each hour. Every time you run this query, the database needs to scan the entire table and recalculate the average. TimescaleDB makes aggregating data lightning fast, accurate, and easy with continuous aggregates.

Continuous aggregates in TimescaleDB are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background.

Continuous aggregates have a much lower maintenance burden than regular Postgres materialized views, because the whole view is not created from scratch on each refresh. This means that you can get on with working your data instead of maintaining your database.

Because continuous aggregates are based on hypertables, you can query them in exactly the same way as your other tables. This includes continuous aggregates in the rowstore, compressed into the columnstore, or tiered to object storage. You can even create continuous aggregates on top of your continuous aggregates, for an even more fine-tuned aggregation.

Real-time aggregation enables you to combine pre-aggregated data from the materialized view with the most recent raw data. This gives you up-to-date results on every query. In TimescaleDB v2.13 and later, real-time aggregates are DISABLED by default. In earlier versions, real-time aggregates are ENABLED by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data.

For more information about using continuous aggregates, see the documentation in Use Tiger Data products.

===== PAGE: https://docs.tigerdata.com/api/data-retention/ =====

Data retention

An intrinsic part of time-series data is that new data is accumulated and old data is rarely, if ever, updated. This means that the relevance of the data diminishes over time. It is therefore often desirable to delete old data to save disk space.

With TimescaleDB, you can manually remove old chunks of data or implement policies using these APIs.

For more information about creating a data retention policy, see the data retention section.

===== PAGE: https://docs.tigerdata.com/api/jobs-automation/ =====

Jobs

Jobs allow you to run functions and procedures implemented in a language of your choice on a schedule within Timescale. This allows automatic periodic tasks that are not covered by existing policies and even enhancing existing policies with additional functionality.

The following APIs and views allow you to manage the jobs that you create and get details around automatic jobs used by other TimescaleDB functions like continuous aggregation refresh policies and data retention policies. To view the policies that you set or the policies that already exist, see informational views.

===== PAGE: https://docs.tigerdata.com/api/uuid-functions/ =====

UUIDv7 functions

UUIDv7 is a time-ordered UUID that includes a Unix timestamp (with millisecond precision) in its first 48 bits. Like other UUIDs, it uses 6 bits for version and variant info, and the remaining 74 bits are random.

UUIDv7 is ideal anywhere you create lots of records over time, not only observability. Advantages are:

• No extra column required to partition by time with sortability: you can sort UUIDv7 instances by their value. This is useful for ordering records by creation time without the need for a separate timestamp column.
• Indexing performance: UUIDv7s increase with time, so new rows append near the end of a B-tree instead of This results in fewer page splits, less fragmentation, faster inserts, and efficient time-range scans.
• Easy keyset pagination: WHERE id > :cursor and natural sharding.
• UUID: safe across services, replicas, and unique across distributed systems.

UUIDv7 also increases query speed by reducing the number of chunks scanned during queries. For example, in a database with 25 million rows, the following query runs in 25 seconds:

WITH ref AS (SELECT now() AS t0)
SELECT count(*) AS cnt_ts_filter
FROM events e, ref
WHERE uuid_timestamp(e.event_id) >= ref.t0 - INTERVAL '2 days';

Using UUIDv7 excludes chunks at startup and reduces the query time to 550ms:

WITH ref AS (SELECT now() AS t0)
SELECT count(*) AS cnt_boundary_filter
FROM events e, ref
WHERE e.event_id >= to_uuidv7_boundary(ref.t0 - INTERVAL '2 days')

You use UUIDvs for events, orders, messages, uploads, runs, jobs, spans, and more.

Examples

• High-rate event logs for observability and metrics:

  UUIDv7 gives you globally unique IDs (for traceability) and time windows (“last hour”), without the need for a separate created_at column. UUIDv7 create less churn because inserts land at the end of the index, and you can filter by time using UUIDv7 objects.

  • Last hour:

    SELECT count(*) FROM logs WHERE id >= to_uuidv7_boundary(now() - interval '1 hour');
    

  • Keyset pagination

    SELECT * FROM logs WHERE id > to_uuidv7($last_seen'::timestamptz, true) ORDER BY id LIMIT 1000;
    

• Workflow / durable execution runs:

  Each run needs a stable ID for joins and retries, and you often ask “what started since X?”. UUIDs help by serving both as the primary key and a time cursor across services. For example:

  SELECT run_id, status
  FROM runs
  WHERE run_id >= to_uuidv7_boundary(now() - interval '5 minutes')
  

• Orders / activity feeds / messages (SaaS apps):

  Human-readable timestamps are not mandatory in a table. However, you still need time-ordered pages and day/week ranges. UUIDv7 enables clean date windows and cursor pagination with just the ID. For example:

  SELECT * FROM orders
  WHERE id >= to_uuidv7('2025-08-01'::timestamptz, true)
  AND id <  to_uuidv7('2025-08-02'::timestamptz, true)
  ORDER BY id;
  

Functions

• generate_uuidv7(): generate a version 7 UUID based on current time
• to_uuidv7(): create a version 7 UUID from a PostgreSQL timestamp
• to_uuidv7_boundary(): create a version 7 "boundary" UUID from a PostgreSQL timestamp
• uuid_timestamp(): extract a PostgreSQL timestamp from a version 7 UUID
• uuid_timestamp_micros(): extract a PostgreSQL timestamp with microsecond precision from a version 7 UUID
• uuid_version(): extract the version of a UUID

===== PAGE: https://docs.tigerdata.com/api/approximate_row_count/ =====

approximate_row_count()

Get approximate row count for hypertable, distributed hypertable, or regular Postgres table based on catalog estimates. This function supports tables with nested inheritance and declarative partitioning.

The accuracy of approximate_row_count depends on the database having up-to-date statistics about the table or hypertable, which are updated by VACUUM, ANALYZE, and a few DDL commands. If you have auto-vacuum configured on your table or hypertable, or changes to the table are relatively infrequent, you might not need to explicitly ANALYZE your table as shown below. Otherwise, if your table statistics are too out-of-date, running this command updates your statistics and yields more accurate approximation results.

Samples

Get the approximate row count for a single hypertable.

ANALYZE conditions;

SELECT * FROM approximate_row_count('conditions');

The expected output:

approximate_row_count
----------------------
               240000

Required arguments

Name	Type	Description
relation	REGCLASS	Hypertable or regular Postgres table to get row count for.

===== PAGE: https://docs.tigerdata.com/api/first/ =====

first()

The first aggregate allows you to get the value of one column as ordered by another. For example, first(temperature, time) returns the earliest temperature value based on time within an aggregate group.

The last and first commands do not use indexes, they perform a sequential scan through the group. They are primarily used for ordered selection within a GROUP BY aggregate, and not as an alternative to an ORDER BY time DESC LIMIT 1 clause to find the latest value, which uses indexes.

Samples

Get the earliest temperature by device_id:

SELECT device_id, first(temp, time)
FROM metrics
GROUP BY device_id;

This example uses first and last with an aggregate filter, and avoids null values in the output:

SELECT
   TIME_BUCKET('5 MIN', time_column) AS interv,
   AVG(temperature) as avg_temp,
   first(temperature,time_column) FILTER(WHERE time_column IS NOT NULL) AS beg_temp,
   last(temperature,time_column) FILTER(WHERE time_column IS NOT NULL) AS end_temp
FROM sensors
GROUP BY interv

Required arguments

Name	Type	Description
value	TEXT	The value to return
time	TIMESTAMP or INTEGER	The timestamp to use for comparison

===== PAGE: https://docs.tigerdata.com/api/last/ =====

last()

The last aggregate allows you to get the value of one column as ordered by another. For example, last(temperature, time) returns the latest temperature value based on time within an aggregate group.

The last and first commands do not use indexes, they perform a sequential scan through the group. They are primarily used for ordered selection within a GROUP BY aggregate, and not as an alternative to an ORDER BY time DESC LIMIT 1 clause to find the latest value, which uses indexes.

Samples

Get the temperature every 5 minutes for each device over the past day:

SELECT device_id, time_bucket('5 minutes', time) AS interval,
  last(temp, time)
FROM metrics
WHERE time > now () - INTERVAL '1 day'
GROUP BY device_id, interval
ORDER BY interval DESC;

This example uses first and last with an aggregate filter, and avoids null values in the output:

SELECT
   TIME_BUCKET('5 MIN', time_column) AS interv,
   AVG(temperature) as avg_temp,
   first(temperature,time_column) FILTER(WHERE time_column IS NOT NULL) AS beg_temp,
   last(temperature,time_column) FILTER(WHERE time_column IS NOT NULL) AS end_temp
FROM sensors
GROUP BY interv

Required arguments

Name	Type	Description
value	ANY ELEMENT	The value to return
time	TIMESTAMP or INTEGER	The timestamp to use for comparison

===== PAGE: https://docs.tigerdata.com/api/histogram/ =====

histogram()

The histogram() function represents the distribution of a set of values as an array of equal-width buckets. It partitions the dataset into a specified number of buckets (nbuckets) ranging from the inputted min and max values.

The return value is an array containing nbuckets+2 buckets, with the middle nbuckets bins for values in the stated range, the first bucket at the head of the array for values under the lower min bound, and the last bucket for values greater than or equal to the max bound. Each bucket is inclusive on its lower bound, and exclusive on its upper bound. Therefore, values equal to the min are included in the bucket starting with min, but values equal to the max are in the last bucket.

Samples

A simple bucketing of device's battery levels from the readings dataset:

SELECT device_id, histogram(battery_level, 20, 60, 5)
FROM readings
GROUP BY device_id
LIMIT 10;

The expected output:

 device_id  |          histogram
------------+------------------------------
 demo000000 | {0,0,0,7,215,206,572}
 demo000001 | {0,12,173,112,99,145,459}
 demo000002 | {0,0,187,167,68,229,349}
 demo000003 | {197,209,127,221,106,112,28}
 demo000004 | {0,0,0,0,0,39,961}
 demo000005 | {12,225,171,122,233,80,157}
 demo000006 | {0,78,176,170,8,40,528}
 demo000007 | {0,0,0,126,239,245,390}
 demo000008 | {0,0,311,345,116,228,0}
 demo000009 | {295,92,105,50,8,8,442}

Required arguments

Name	Type	Description
value	ANY VALUE	A set of values to partition into a histogram
min	NUMERIC	The histogram's lower bound used in bucketing (inclusive)
max	NUMERIC	The histogram's upper bound used in bucketing (exclusive)
nbuckets	INTEGER	The integer value for the number of histogram buckets (partitions)

===== PAGE: https://docs.tigerdata.com/api/time_bucket/ =====

time_bucket()

The time_bucket function is similar to the standard Postgres date_bin function. Unlike date_bin, it allows for arbitrary time intervals of months or longer. The return value is the bucket's start time.

Buckets are aligned to start at midnight in UTC+0. The time bucket size (bucket_width) can be set as INTERVAL or INTEGER. For INTERVAL-type bucket_width, you can change the time zone with the optional timezone parameter. In this case, the buckets are realigned to start at midnight in the time zone you specify.

Note that during shifts to and from daylight savings, the amount of data aggregated into the corresponding buckets can be irregular. For example, if the bucket_width is 2 hours, the number of bucketed hours is either three hours or one hour.

Samples

Simple five-minute averaging:

SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
FROM metrics
GROUP BY five_min
ORDER BY five_min DESC LIMIT 10;

To report the middle of the bucket, instead of the left edge:

SELECT time_bucket('5 minutes', time) + '2.5 minutes'
  AS five_min, avg(cpu)
FROM metrics
GROUP BY five_min
ORDER BY five_min DESC LIMIT 10;

For rounding, move the alignment so that the middle of the bucket is at the five-minute mark, and report the middle of the bucket:

SELECT time_bucket('5 minutes', time, '-2.5 minutes'::INTERVAL) + '2.5 minutes'
  AS five_min, avg(cpu)
FROM metrics
GROUP BY five_min
ORDER BY five_min DESC LIMIT 10;

In this example, add the explicit cast to ensure that Postgres chooses the correct function.

To shift the alignment of the buckets, you can use the origin parameter passed as a timestamp, timestamptz, or date type. This example shifts the start of the week to a Sunday, instead of the default of Monday:

SELECT time_bucket('1 week', timetz, TIMESTAMPTZ '2017-12-31')
  AS one_week, avg(cpu)
FROM metrics
GROUP BY one_week
WHERE time > TIMESTAMPTZ '2017-12-01'  AND time < TIMESTAMPTZ '2018-01-03'
ORDER BY one_week DESC LIMIT 10;

The value of the origin parameter in this example is 2017-12-31, a Sunday within the period being analyzed. However, the origin provided to the function can be before, during, or after the data being analyzed. All buckets are calculated relative to this origin. So, in this example, any Sunday could have been used. Note that because time < TIMESTAMPTZ '2018-01-03' is used in this example, the last bucket would have only 4 days of data. This cast to TIMESTAMP converts the time to local time according to the server's time zone setting.

SELECT time_bucket(INTERVAL '2 hours', timetz::TIMESTAMP)
  AS five_min, avg(cpu)
FROM metrics
GROUP BY five_min
ORDER BY five_min DESC LIMIT 10;

Bucket temperature values to calculate the average monthly temperature. Set the time zone to 'Europe/Berlin' so bucket start and end times are aligned to midnight in Berlin.

SELECT time_bucket('1 month', ts, 'Europe/Berlin') AS month_bucket,
  avg(temperature) AS avg_temp
FROM weather
GROUP BY month_bucket
ORDER BY month_bucket DESC LIMIT 10;

Required arguments for interval time inputs

Name	Type	Description
bucket_width	INTERVAL	A Postgres time interval for how long each bucket is
ts	DATE, TIMESTAMP, or TIMESTAMPTZ	The timestamp to bucket

If you use months as an interval for bucket_width, you cannot combine it with a non-month component. For example, 1 month and 3 months are both valid bucket widths, but 1 month 1 day and 3 months 2 weeks are not.

Optional arguments for interval time inputs

Name	Type	Description
timezone	TEXT	The time zone for calculating bucket start and end times. Can only be used with TIMESTAMPTZ. Defaults to UTC+0.
origin	DATE, TIMESTAMP, or TIMESTAMPTZ	Buckets are aligned relative to this timestamp. Defaults to midnight on January 3, 2000, for buckets that don't include a month or year interval, and to midnight on January 1, 2000, for month, year, and century buckets.
offset	INTERVAL	The time interval to offset all time buckets by. A positive value shifts bucket start and end times later. A negative value shifts bucket start and end times earlier. offset must be surrounded with double quotes when used as a named argument, because it is a reserved key word in Postgres.

Required arguments for integer time inputs

Name	Type	Description
bucket_width	INTEGER	The bucket width
ts	INTEGER	The timestamp to bucket

Optional arguments for integer time inputs

Name	Type	Description
offset	INTEGER	The amount to offset all buckets by. A positive value shifts bucket start and end times later. A negative value shifts bucket start and end times earlier. offset must be surrounded with double quotes when used as a named argument, because it is a reserved key word in Postgres.

===== PAGE: https://docs.tigerdata.com/api/time_bucket_ng/ =====

timescaledb_experimental.time_bucket_ng()

The time_bucket_ng() function is an experimental version of the time_bucket() function. It introduced some new capabilities, such as monthly buckets and timezone support. Those features are now part of the regular time_bucket() function.

This section describes a feature that is deprecated. We strongly recommend that you do not use this feature in a production environment. If you need more information, contact us.

The time_bucket() and time_bucket_ng() functions are similar, but not completely compatible. There are two main differences.

Firstly, time_bucket_ng() doesn't work with timestamps prior to origin, while time_bucket() does.

Secondly, the default origin values differ. time_bucket() uses an origin date of January 3, 2000, for buckets shorter than a month. time_bucket_ng() uses an origin date of January 1, 2000, for all bucket sizes.

Samples

In this example, time_bucket_ng() is used to create bucket data in three month intervals:

SELECT timescaledb_experimental.time_bucket_ng('3 month', date '2021-08-01');
 time_bucket_ng
----------------
 2021-07-01
(1 row)

This example uses time_bucket_ng() to bucket data in one year intervals:

SELECT timescaledb_experimental.time_bucket_ng('1 year', date '2021-08-01');
 time_bucket_ng
----------------
 2021-01-01
(1 row)

To split time into buckets, time_bucket_ng() uses a starting point in time called origin. The default origin is 2000-01-01. time_bucket_ng cannot use timestamps earlier than origin:

SELECT timescaledb_experimental.time_bucket_ng('100 years', timestamp '1988-05-08');
ERROR:  origin must be before the given date

Going back in time from origin isn't usually possible, especially when you consider timezones and daylight savings time (DST). Note also that there is no reasonable way to split time in variable-sized buckets (such as months) from an arbitrary origin, so origin defaults to the first day of the month.

To bypass named limitations, you can override the default origin:

-- working with timestamps before 2000-01-01
SELECT timescaledb_experimental.time_bucket_ng('100 years', timestamp '1988-05-08', origin => '1900-01-01');
   time_bucket_ng
---------------------
 1900-01-01 00:00:00

-- unlike the default origin, which is Saturday, 2000-01-03 is Monday
SELECT timescaledb_experimental.time_bucket_ng('1 week', timestamp '2021-08-26', origin => '2000-01-03');
   time_bucket_ng
---------------------
 2021-08-23 00:00:00

This example shows how time_bucket_ng() is used to bucket data by months in a specified timezone:

-- note that timestamptz is displayed differently depending on the session parameters
SET TIME ZONE 'Europe/Moscow';
SET

SELECT timescaledb_experimental.time_bucket_ng('1 month', timestamptz '2001-02-03 12:34:56 MSK', timezone => 'Europe/Moscow');
     time_bucket_ng
------------------------
 2001-02-01 00:00:00+03

You can use time_bucket_ng() with continuous aggregates. This example tracks the temperature in Moscow over seven day intervals:

CREATE TABLE conditions(
  day DATE NOT NULL,
  city text NOT NULL,
  temperature INT NOT NULL);

SELECT create_hypertable(
  'conditions', by_range('day', INTERVAL '1 day')
);

INSERT INTO conditions (day, city, temperature) VALUES
  ('2021-06-14', 'Moscow', 26),
  ('2021-06-15', 'Moscow', 22),
  ('2021-06-16', 'Moscow', 24),
  ('2021-06-17', 'Moscow', 24),
  ('2021-06-18', 'Moscow', 27),
  ('2021-06-19', 'Moscow', 28),
  ('2021-06-20', 'Moscow', 30),
  ('2021-06-21', 'Moscow', 31),
  ('2021-06-22', 'Moscow', 34),
  ('2021-06-23', 'Moscow', 34),
  ('2021-06-24', 'Moscow', 34),
  ('2021-06-25', 'Moscow', 32),
  ('2021-06-26', 'Moscow', 32),
  ('2021-06-27', 'Moscow', 31);

CREATE MATERIALIZED VIEW conditions_summary_weekly
WITH (timescaledb.continuous) AS
SELECT city,
       timescaledb_experimental.time_bucket_ng('7 days', day) AS bucket,
       MIN(temperature),
       MAX(temperature)
FROM conditions
GROUP BY city, bucket;

SELECT to_char(bucket, 'YYYY-MM-DD'), city, min, max
FROM conditions_summary_weekly
ORDER BY bucket;

  to_char   |  city  | min | max
------------+--------+-----+-----
 2021-06-12 | Moscow |  22 |  27
 2021-06-19 | Moscow |  28 |  34
 2021-06-26 | Moscow |  31 |  32
(3 rows)

The by_range dimension builder is an addition to TimescaleDB 2.13. For simpler cases, like this one, you can also create the hypertable using the old syntax:

SELECT create_hypertable('', '<time column name>');

For more information, see the continuous aggregates documentation.

While time_bucket_ng() supports months and timezones, continuous aggregates cannot always be used with monthly buckets or buckets with timezones.

This table shows which time_bucket_ng() functions can be used in a continuous aggregate:

Function	Available in continuous aggregate	TimescaleDB version
Buckets by seconds, minutes, hours, days, and weeks	✅	2.4.0 - 2.14.2
Buckets by months and years	✅	2.6.0 - 2.14.2
Timezones support	✅	2.6.0 - 2.14.2
Specify custom origin	✅	2.7.0 - 2.14.2

Required arguments

Name	Type	Description
bucket_width	INTERVAL	A Postgres time interval for how long each bucket is
ts	DATE, TIMESTAMP or TIMESTAMPTZ	The timestamp to bucket

Optional arguments

Name	Type	Description
origin	Should be the same as ts	Buckets are aligned relative to this timestamp
timezone	TEXT	The name of the timezone. The argument can be specified only if the type of ts is TIMESTAMPTZ

For backward compatibility with time_bucket() the timezone argument is optional. However, it is required for time buckets that are less than 24 hours.

If you call the TIMESTAMPTZ-version of the function without the timezone argument, the timezone defaults to the session's timezone and so the function can't be used with continuous aggregates. Best practice is to use time_bucket_ng(interval, timestamptz, text) and specify the timezone.

Returns

The function returns the bucket's start time. The return value type is the same as ts.

===== PAGE: https://docs.tigerdata.com/api/days_in_month/ =====

days_in_month()

Given a timestamptz, returns how many days are in that month.

Samples

Calculate how many days in the month of January 1, 2022:

SELECT days_in_month('2021-01-01 00:00:00+03'::timestamptz)

The output looks like this:

days_in_month
----------------------
31

Required arguments

Name	Type	Description
date	TIMESTAMPTZ	Timestamp to use to calculate how many days in the month

===== PAGE: https://docs.tigerdata.com/api/month_normalize/ =====

month_normalize()

Translate a metric to a standard month. A standard month is calculated as the exact number of days in a year divided by the number of months in a year, so 365.25/12 = 30.4375. month_normalize() divides a metric by the number of days in the corresponding calendar month and multiplies it by 30.4375.

This enables you to compare metrics for different months and decide which one performed better, objectively. For example, in the following table that summarizes the number of sales for three months, January has the highest number of total sales:

Month	Sales
Jan	3000
Feb	2900
Mar	2900

When you normalize the sales metrics, you get the following result, showing that February in fact performed better:

Month	Normalized sales
Jan	2945.56
Feb	3152.46
Mar	2847.38

Samples

Get the normalized value for a metric of 1000, and a reference date of January 1, 2021:

SELECT month_normalize(1000,'2021-01-01 00:00:00+03'::timestamptz)

The output looks like this:

month_normalize
----------------------
981.8548387096774

Required arguments

Name	Type	Description
metric	float8
reference_date	TIMESTAMPTZ	Timestamp to normalize the metric with
days	float8	Optional, defaults to 365.25/12 if none provided

===== PAGE: https://docs.tigerdata.com/api/gauge_agg/ =====

gauge_agg()

Produces a GaugeSummary that can be used to accumulate gauge data for further calculations.

gauge_agg (
    ts TIMESTAMPTZ,
    value DOUBLE PRECISION
) RETURNS GaugeSummary

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

For more information about counter and gauge aggregation functions, see the hyperfunctions documentation.

Required arguments

Name	Type	Description
ts	TIMESTAMPTZ	The time at each point
value	DOUBLE PRECISION	The value at that timestamp

Only DOUBLE PRECISION values are accepted for the value parameter. For gauge data stored as other numeric types, cast it to DOUBLE PRECISION when using the function.

If there are NULL values in your data, the aggregate ignores them and aggregates only non-NULL values. If you only have NULL values, the aggregate returns NULL.

Optional arguments

Name	Type	Description
bounds	TSTZRANGE	The largest and smallest possible times that can be input to the aggregate. Calling with NULL, or leaving out the argument, results in an unbounded GaugeSummary

Bounds are required for extrapolation, but not for other accessor functions.

Returns

Column	Type	Description
gauge_agg	GaugeSummary	A GaugeSummary object that can be passed to accessor functions or other objects in the gauge aggregate API

The returned GaugeSummary can be used as an input the accessor functions delta, idelta_left, and idelta_right. When this feature is mature, it will support all the same accessor functions as CounterSummary, with the exception of num_resets.

Sample usage

Create a gauge summary from time-series data that has a timestamp, ts, and a gauge value, val. Get the instantaneous rate of change from the last 2 time intervals using the irate_right accessor:

WITH t as (
    SELECT
        time_bucket('1 day'::interval, ts) as dt,
        gauge_agg(ts, val) AS gs
    FROM foo
    WHERE id = 'bar'
    GROUP BY time_bucket('1 day'::interval, ts)
)
SELECT
    dt,
    irate_right(gs)
FROM t;

===== PAGE: https://docs.tigerdata.com/api/frequency-analysis/ =====

Frequency analysis

This section includes frequency aggregate APIs, which find the most common elements out of a set of vastly more varied values.

For these hyperfunctions, you need to install the TimescaleDB Toolkit Postgres extension.

===== PAGE: https://docs.tigerdata.com/api/informational-views/ =====

Information views

TimescaleDB makes complex database features like partitioning and data retention easy to use with our comprehensive APIs. TimescaleDB works hard to provide detailed information about the state of your data, hypertables, chunks, and any jobs or policies you have in place.

These views provide the data and statistics you need to keep track of your database.

===== PAGE: https://docs.tigerdata.com/api/configuration/ =====

Service configuration

Tiger Cloud service use the default Postgres server configuration settings. You can optimize your service configuration using the following TimescaleDB and Grand Unified Configuration (GUC) parameters.

• TimescaleDB configuration and tuning
• Grand Unified Configuration (GUC) parameters

===== PAGE: https://docs.tigerdata.com/api/administration/ =====

Administrative functions

These administrative APIs help you prepare a database before and after a restore event. They also help you keep track of your TimescaleDB setup data.

Dump TimescaleDB meta data

To help when asking for support and reporting bugs, TimescaleDB includes an SQL dump script. It outputs metadata from the internal TimescaleDB tables, along with version information.

This script is available in the source distribution in scripts/. To use it, run:

psql [your connect flags] -d your_timescale_db < dump_meta_data.sql > dumpfile.txt

Inspect dumpfile.txt before sending it together with a bug report or support question.

get_telemetry_report()

Returns the background telemetry string sent to Tiger Data.

If telemetry is turned off, it sends the string that would be sent if telemetry were enabled.

Sample usage

View the telemetry report:

SELECT get_telemetry_report();

timescaledb_post_restore()

Perform the required operations after you have finished restoring the database using pg_restore. Specifically, this resets the timescaledb.restoring GUC and restarts any background workers.

For more information, see Migrate using pg_dump and pg_restore.

Sample usage

Prepare the database for normal use after a restore:

SELECT timescaledb_post_restore();

timescaledb_pre_restore()

Perform the required operations so that you can restore the database using pg_restore. Specifically, this sets the timescaledb.restoring GUC to on and stops any background workers which could have been performing tasks.

The background workers are stopped until the timescaledb_post_restore() function is run, after the restore operation is complete.

For more information, see Migrate using pg_dump and pg_restore.

After using timescaledb_pre_restore(), you need to run timescaledb_post_restore() before you can use the database normally.

Sample usage

Prepare to restore the database:

SELECT timescaledb_pre_restore();

===== PAGE: https://docs.tigerdata.com/api/api-tag-overview/ =====

API reference tag overview

The TimescaleDB API Reference uses tags to categorize functions. The tags are Community, Experimental, Toolkit, and Experimental (Toolkit). This section explains each tag.

Community Community

This tag indicates that the function is available under TimescaleDB Community Edition, and are not available under the Apache 2 Edition. For more information, visit our TimescaleDB License comparison sheet.

Experimental (TimescaleDB Experimental Schema) Experimental

This tag indicates that the function is included in the TimescaleDB experimental schema. Do not use experimental functions in production. Experimental features could include bugs, and are likely to change in future versions. The experimental schema is used by TimescaleDB to develop new features more quickly. If experimental functions are successful, they can move out of the experimental schema and go into production use.

When you upgrade the timescaledb extension, the experimental schema is removed by default. To use experimental features after an upgrade, you need to add the experimental schema again.

For more information about the experimental schema, [read the Tiger Data blog post][experimental-blog].

Toolkit Toolkit

This tag indicates that the function is included in the TimescaleDB Toolkit extension. Toolkit functions are available under TimescaleDB Community Edition. For installation instructions, see the installation guide.

Experimental (TimescaleDB Toolkit) Experimental

This tag is used with the Toolkit tag. It indicates a Toolkit function that is under active development. Do not use experimental toolkit functions in production. Experimental toolkit functions could include bugs, and are likely to change in future versions.

These functions might not correctly handle unusual use cases or errors, and they could have poor performance. Updates to the TimescaleDB extension drop database objects that depend on experimental features like this function. If you use experimental toolkit functions on Timescale, this function is automatically dropped when the Toolkit extension is updated. For more information, see the TimescaleDB Toolkit docs.

===== PAGE: https://docs.tigerdata.com/api/api-reference/ =====

Tiger Cloud REST API reference

A comprehensive RESTful API for managing Tiger Cloud resources including VPCs, services, and read replicas.

Overview

API Version: 1.0.0 Base URL: https://console.cloud.timescale.com/public/api/v1

Authentication

The Tiger REST API uses HTTP Basic Authentication. Include your access key and secret key in the Authorization header.

Basic Authentication

Authorization: Basic <base64(access_key:secret_key)>

Example

curl -X GET "https://console.cloud.timescale.com/public/api/v1/projects/{project_id}/services" \
  -H "Authorization: Basic $(echo -n 'your_access_key:your_secret_key' | base64)"

Service Management

You use this endpoint to create a Tiger Cloud service with one of more of the following addons:

• time-series: a Tiger Cloud service optimized for real-time analytics. For time-stamped data like events, prices, metrics, sensor readings, or any information that changes over time.
• ai: a Tiger Cloud service instance with vector extensions.

To have multiple addons when you create a new service, set "addons": ["time-series", "ai"]. To create a vanilla Postgres instance, set addons to an empty list [].

List All Services

GET /projects/{project_id}/services

Retrieve all services within a project.

Response: 200 OK

[
  {
    "service_id": "p7zm9wqqii",
    "project_id": "jz22xtzemv",
    "name": "my-production-db",
    "region_code": "eu-central-1",
    "service_type": "TIMESCALEDB",
    "status": "READY",
    "created": "2024-01-15T10:30:00Z",
    "paused": false,
    "resources": [
      {
        "id": "resource-1",
        "spec": {
          "cpu_millis": 1000,
          "memory_gbs": 4,
          "volume_type": "gp2"
        }
      }
    ],
    "endpoint": {
      "host": "my-service.com",
      "port": 5432
    }
  }
]

Create a Service

POST /projects/{project_id}/services

Create a new Tiger Cloud service. This is an asynchronous operation.

Request Body:

{
  "name": "test-2",
  "addons": ["time-series"],
  "region_code": "eu-central-1",
  "cpu_millis": 1000,
  "memory_gbs": 4
}

Response: 202 Accepted

{
  "service_id": "p7zm9wqqii",
  "project_id": "jz22xtzemv",
  "name": "test-2",
  "region_code": "eu-central-1",
  "service_type": "TIMESCALEDB",
  "created": "2025-09-04T20:46:46.265680278Z",
  "paused": false,
  "status": "READY",
  "resources": [
      {
          "id": "100927",
          "spec": {
              "cpu_millis": 1000,
              "memory_gbs": 4,
              "volume_type": ""
          }
      }
  ],
  "metadata": {
      "environment": "PROD"
  },
  "endpoint": {
      "host": "p7zm8wqqii.jz4qxtzemv.tsdb.cloud.timescale.com",
      "port": 35482
  },
  "initial_password": "oamv8ch9t4ar2j8g"
}

Service Types:

• TIMESCALEDB: a Tiger Cloud service instance optimized for real-time analytics service For time-stamped data like events, prices, metrics, sensor readings, or any information that changes over time
• POSTGRES: a vanilla Postgres instance
• VECTOR: a Tiger Cloud service instance with vector extensions

Get a Service

GET /projects/{project_id}/services/{service_id}

Retrieve details of a specific service.

Response: 200 OK

{
  "service_id": "p7zm9wqqii",
  "project_id": "jz22xtzemv",
  "name": "test-2",
  "region_code": "eu-central-1",
  "service_type": "TIMESCALEDB",
  "created": "2025-09-04T20:46:46.26568Z",
  "paused": false,
  "status": "READY",
  "resources": [
      {
          "id": "100927",
          "spec": {
              "cpu_millis": 1000,
              "memory_gbs": 4,
              "volume_type": ""
          }
      }
  ],
  "metadata": {
      "environment": "DEV"
  },
  "endpoint": {
      "host": "p7zm8wqqii.jz4qxtzemv.tsdb.cloud.timescale.com",
      "port": 35482
  }
}

Service Status:

• QUEUED: Service creation is queued
• DELETING: Service is being deleted
• CONFIGURING: Service is being configured
• READY: Service is ready for use
• DELETED: Service has been deleted
• UNSTABLE: Service is in an unstable state
• PAUSING: Service is being paused
• PAUSED: Service is paused
• RESUMING: Service is being resumed
• UPGRADING: Service is being upgraded
• OPTIMIZING: Service is being optimized

Delete a Service

DELETE /projects/{project_id}/services/{service_id}

Delete a specific service. This is an asynchronous operation.

Response: 202 Accepted

Resize a Service

POST /projects/{project_id}/services/{service_id}/resize

Change CPU and memory allocation for a service.

Request Body:

{
  "cpu_millis": 2000,
  "memory_gbs": 8
}

Response: 202 Accepted

Update Service Password

POST /projects/{project_id}/services/{service_id}/updatePassword

Set a new master password for the service.

Request Body:

{
  "password": "a-very-secure-new-password"
}

Response: 204 No Content

Set Service Environment

POST /projects/{project_id}/services/{service_id}/setEnvironment

Set the environment type for the service.

Request Body:

{
  "environment": "PROD"
}

Environment Values:

• PROD: Production environment
• DEV: Development environment

Response: 200 OK

{
    "message": "Environment set successfully"
}

Configure High Availability

POST /projects/{project_id}/services/{service_id}/setHA

Change the HA configuration for a service. This is an asynchronous operation.

Request Body:

{
  "replica_count": 1
}

Response: 202 Accepted

Connection Pooler Management

Enable Connection Pooler

POST /projects/{project_id}/services/{service_id}/enablePooler

Activate the connection pooler for a service.

Response: 200 OK

{
  "message": "Connection pooler enabled successfully"
}

Disable Connection Pooler

POST /projects/{project_id}/services/{service_id}/disablePooler

Deactivate the connection pooler for a service.

Response: 200 OK

{
  "message": "Connection pooler disabled successfully"
}

Fork a Service

POST /projects/{project_id}/services/{service_id}/forkService

Create a new, independent service by taking a snapshot of an existing one.

Request Body:

{
    "name": "fork-test2",
    "region_code": "eu-central-1",
    "cpu_millis": 1000,
    "memory_gbs": 4
}

Response: 202 Accepted

{
    "service_id": "otewd3pem2",
    "project_id": "jz22xtzemv",
    "name": "fork-test2",
    "region_code": "eu-central-1",
    "service_type": "TIMESCALEDB",
    "created": "2025-09-04T20:54:09.53380732Z",
    "paused": false,
    "status": "READY",
    "resources": [
        {
            "id": "100929",
            "spec": {
                "cpu_millis": 1000,
                "memory_gbs": 4,
                "volume_type": ""
            }
        }
    ],
    "forked_from": {
        "project_id": "jz22xtzemv",
        "service_id": "p7zm9wqqii",
        "is_standby": false
    },
    "initial_password": "ph33bl5juuri5gem"
}

Read Replica Sets

Manage read replicas for improved read performance.

List Read Replica Sets

GET /projects/{project_id}/services/{service_id}/replicaSets

Retrieve all read replica sets associated with a primary service.

Response: 200 OK

[
  {
    "id": "l5alxb3s2g",
    "name": "replica-set-test2",
    "status": "active",
    "nodes": 1,
    "cpu_millis": 1000,
    "memory_gbs": 4,
    "endpoint": {
        "host": "l5alxb3s2g.jz4qxtzemv.tsdb.cloud.timescale.com",
        "port": 38448
    },
    "connection_pooler": {
        "endpoint": {
            "host": "l5alxb3s2g.jz4qxtzemv.tsdb.cloud.timescale.com",
            "port": 38543
        }
    },
    "metadata": {
        "environment": "DEV"
    }
  }
]

Replica Set Status:

• creating: Replica set is being created
• active: Replica set is active and ready
• resizing: Replica set is being resized
• deleting: Replica set is being deleted
• error: Replica set encountered an error

Create a Read Replica Set

POST /projects/{project_id}/services/{service_id}/replicaSets

Create a new read replica set. This is an asynchronous operation.

Request Body:

{
  "name": "replica-set-test2",
  "cpu_millis": 1000,
  "memory_gbs": 4,
  "nodes": 1
}

Response: 202 Accepted

{
  "id": "dsldm715t2",
  "name": "replica-set-test2",
  "status": "active",
  "nodes": 1,
  "cpu_millis": 1000,
  "memory_gbs": 4
}

Delete a Read Replica Set

DELETE /projects/{project_id}/services/{service_id}/replicaSets/{replica_set_id}

Delete a specific read replica set. This is an asynchronous operation.

Response: 202 Accepted

Resize a Read Replica Set

POST /projects/{project_id}/services/{service_id}/replicaSets/{replica_set_id}/resize

Change resource allocation for a read replica set. This operation is async.

Request Body:

{
  "cpu_millis": 500,
  "memory_gbs": 2,
  "nodes": 2
}

Response: 202 Accepted

{
    "message": "Replica set resize request accepted"
}

Read Replica Set Connection Pooler

Enable Replica Set Pooler

POST /projects/{project_id}/services/{service_id}/replicaSets/{replica_set_id}/enablePooler

Activate the connection pooler for a read replica set.

Response: 200 OK

{
  "message": "Connection pooler enabled successfully"
}

Disable Replica Set Pooler

POST /projects/{project_id}/services/{service_id}/replicaSets/{replica_set_id}/disablePooler

Deactivate the connection pooler for a read replica set.

Response: 200 OK

{
  "message": "Connection pooler disabled successfully"
}

Set Replica Set Environment

POST /projects/{project_id}/services/{service_id}/replicaSets/{replica_set_id}/setEnvironment

Set the environment type for a read replica set.

Request Body:

{
  "environment": "PROD"
}

Response: 200 OK

{
  "message": "Environment set successfully"
}

VPC Management

Virtual Private Clouds (VPCs) provide network isolation for your TigerData services.

List All VPCs

GET /projects/{project_id}/vpcs

List all Virtual Private Clouds in a project.

Response: 200 OK

[
  {
    "id": "1234567890",
    "name": "my-production-vpc",
    "cidr": "10.0.0.0/16",
    "region_code": "eu-central-1"
  }
]

Create a VPC

POST /projects/{project_id}/vpcs

Create a new VPC.

Request Body:

{
  "name": "my-production-vpc",
  "cidr": "10.0.0.0/16",
  "region_code": "eu-central-1"
}

Response: 201 Created

{
  "id": "1234567890",
  "name": "my-production-vpc",
  "cidr": "10.0.0.0/16",
  "region_code": "eu-central-1"
}

Get a VPC

GET /projects/{project_id}/vpcs/{vpc_id}

Retrieve details of a specific VPC.

Response: 200 OK

{
  "id": "1234567890",
  "name": "my-production-vpc",
  "cidr": "10.0.0.0/16",
  "region_code": "eu-central-1"
}

Rename a VPC

POST /projects/{project_id}/vpcs/{vpc_id}/rename

Update the name of a specific VPC.

Request Body:

{
  "name": "my-renamed-vpc"
}

Response: 200 OK

{
  "id": "1234567890",
  "name": "my-renamed-vpc",
  "cidr": "10.0.0.0/16",
  "region_code": "eu-central-1"
}

Delete a VPC

DELETE /projects/{project_id}/vpcs/{vpc_id}

Delete a specific VPC.

Response: 204 No Content

VPC Peering

Manage peering connections between VPCs across different accounts and regions.

List VPC Peerings

GET /projects/{project_id}/vpcs/{vpc_id}/peerings

Retrieve all VPC peering connections for a given VPC.

Response: 200 OK

[
  {
    "id": "1234567890",
    "peer_account_id": "acc-12345",
    "peer_region_code": "eu-central-1",
    "peer_vpc_id": "1234567890",
    "provisioned_id": "1234567890",
    "status": "active",
    "error_message": null
  }
]

Create VPC Peering

POST /projects/{project_id}/vpcs/{vpc_id}/peerings

Create a new VPC peering connection.

Request Body:

{
  "peer_account_id": "acc-12345",
  "peer_region_code": "eu-central-1",
  "peer_vpc_id": "1234567890"
}

Response: 201 Created

{
  "id": "1234567890",
  "peer_account_id": "acc-12345",
  "peer_region_code": "eu-central-1",
  "peer_vpc_id": "1234567890",
  "provisioned_id": "1234567890",
  "status": "pending"
}

Get VPC Peering

GET /projects/{project_id}/vpcs/{vpc_id}/peerings/{peering_id}

Retrieve details of a specific VPC peering connection.

Delete VPC Peering

DELETE /projects/{project_id}/vpcs/{vpc_id}/peerings/{peering_id}

Delete a specific VPC peering connection.

Response: 204 No Content

Service VPC Operations

Attach Service to VPC

POST /projects/{project_id}/services/{service_id}/attachToVPC

Associate a service with a VPC.

Request Body:

{
  "vpc_id": "1234567890"
}

Response: 202 Accepted

Detach Service from VPC

POST /projects/{project_id}/services/{service_id}/detachFromVPC

Disassociate a service from its VPC.

Request Body:

{
  "vpc_id": "1234567890"
}

Response: 202 Accepted

Data Models

VPC Object

{
  "id": "string",
  "name": "string",
  "cidr": "string",
  "region_code": "string"
}

Service Object

{
  "service_id": "string",
  "project_id": "string",
  "name": "string",
  "region_code": "string",
  "service_type": "TIMESCALEDB|POSTGRES|VECTOR",
  "created": "2024-01-15T10:30:00Z",
  "initial_password": "string",
  "paused": false,
  "status": "READY|CONFIGURING|QUEUED|...",
  "resources": [
    {
      "id": "string",
      "spec": {
        "cpu_millis": 1000,
        "memory_gbs": 4,
        "volume_type": "string"
      }
    }
  ],
  "metadata": {
    "environment": "PROD|DEV"
  },
  "endpoint": {
    "host": "string",
    "port": 5432
  },
  "connection_pooler": {
    "endpoint": {
      "host": "string",
      "port": 5432
    }
  }
}

Peering Object

{
  "id": "string",
  "peer_account_id": "string",
  "peer_region_code": "string",
  "peer_vpc_id": "string",
  "provisioned_id": "string",
  "status": "string",
  "error_message": "string"
}

Read Replica Set Object

{
  "id": "string",
  "name": "string",
  "status": "creating|active|resizing|deleting|error",
  "nodes": 2,
  "cpu_millis": 1000,
  "memory_gbs": 4,
  "metadata": {
    "environment": "PROD|DEV"
  },
  "endpoint": {
    "host": "string",
    "port": 5432
  },
  "connection_pooler": {
    "endpoint": {
      "host": "string",
      "port": 5432
    }
  }
}

Error Handling

Tiger Cloud REST API uses standard HTTP status codes and returns error details in JSON format.

Error Response Format

{
  "code": "ERROR_CODE",
  "message": "Human-readable error description"
}

Common Error Codes

• 400 Bad Request: Invalid request parameters or malformed JSON
• 401 Unauthorized: Missing or invalid authentication credentials
• 403 Forbidden: Insufficient permissions for the requested operation
• 404 Not Found: Requested resource does not exist
• 409 Conflict: Request conflicts with current resource state
• 500 Internal Server Error: Unexpected server error

Example Error Response

{
  "code": "INVALID_REQUEST",
  "message": "The service_type field is required"
}

===== PAGE: https://docs.tigerdata.com/api/glossary/ =====

Tiger Data glossary of terms

This glossary defines technical terms, concepts, and terminology used in Tiger Data documentation, database industry, and real-time analytics.

A

ACL (Access Control List): a table that tells a computer operating system which access rights each user has to a particular system object, such as a file directory or individual file.

ACID: a set of properties (atomicity, consistency, isolation, durability) that guarantee database transactions are processed reliably.

ACID compliance: a set of database properties—Atomicity, Consistency, Isolation, Durability—ensuring reliable and consistent transactions. Inherited from Postgres.

Adaptive query optimization: dynamic query plan adjustment based on actual execution statistics and data distribution patterns, improving performance over time.

Aggregate (Continuous Aggregate): a materialized, precomputed summary of query results over time-series data, providing faster access to analytics.

Alerting: the process of automatically notifying administrators when predefined conditions or thresholds are met in system monitoring.

Analytics database: a system optimized for large-scale analytical queries, supporting complex aggregations, time-based queries, and data exploration.

Anomaly detection: the identification of abnormal patterns or outliers within time-series datasets, common in observability, IoT, and finance.

Append-only storage: a storage pattern where data is only added, never modified in place. Ideal for time-series workloads and audit trails.

Archival: the process of moving old or infrequently accessed data to long-term, cost-effective storage solutions.

Auto-partitioning: automatic division of a hypertable into chunks based on partitioning dimensions to optimize scalability and performance.

Availability zone: an isolated location within a cloud region that provides redundant power, networking, and connectivity.

B

B-tree: a self-balancing tree data structure that maintains sorted data and allows searches, sequential access, insertions, and deletions in logarithmic time.

Background job: an automated task that runs in the background without user intervention, typically for maintenance operations like compression or data retention.

Background worker: a Postgres process that runs background tasks independently of client sessions.

Batch processing: handling data in grouped batches rather than as individual real-time events, often used for historical data processing.

Backfill: the process of filling in historical data that was missing or needs to be recalculated, often used during migrations or after schema changes.

Backup: a copy of data stored separately from the original data to protect against data loss, corruption, or system failure.

Bloom filter: a probabilistic data structure that tests set membership with possible false positives but no false negatives. TimescaleDB uses blocked bloom filters to speed up point lookups by eliminating chunks that don't contain queried values.

Buffer pool: memory area where frequently accessed data pages are cached to reduce disk I/O operations.

BRIN (Block Range Index): a Postgres index type that stores summaries about ranges of table blocks, useful for large tables with naturally ordered data.

Bytea: a Postgres data type for storing binary data as a sequence of bytes.

C

Cache hit ratio: the percentage of data requests served from memory cache rather than disk, indicating query performance efficiency.

Cardinality: the number of unique values in a dataset or database column.

Check constraint: a database constraint that limits the values that can be stored in a column by checking them against a specified condition.

Chunk: a horizontal partition of a hypertable that contains data for a specific time interval and space partition. See chunks.

Chunk interval: the time period covered by each chunk in a hypertable, which affects query performance and storage efficiency.

Chunk skipping: a query optimization technique that skips chunks not relevant to the query's time range, dramatically improving performance.

CIDR (Classless Inter-Domain Routing): a method for allocating IP addresses and routing IP packets.

Client credentials: authentication tokens used by applications to access services programmatically without user interaction.

Close: in financial data, the closing price of a security at the end of a trading period.

Cloud: computing services delivered over the internet, including servers, storage, databases, networking, software, analytics, and intelligence.

Cloud deployment: the use of public, private, or hybrid cloud infrastructure to host TimescaleDB, enabling elastic scalability and managed services.

Cloud-native: an approach to building applications that leverage cloud infrastructure, scalability, and services like Kubernetes.

Cold storage: a tier of data storage for infrequently accessed data that offers lower costs but higher access times.

Columnar: a data storage format that stores data column by column rather than row by row, optimizing for analytical queries.

Columnstore: TimescaleDB's columnar storage engine optimized for analytical workloads and compression.

Compression: the process of reducing data size by encoding information using fewer bits, improving storage efficiency and query performance. See compression.

Connection pooling: a technique for managing multiple database connections efficiently, reducing overhead for high-concurrency environments.

Consensus algorithm: protocols ensuring distributed systems agree on data state, critical for multi-node database deployments.

Compression policy: an automated rule that compresses hypertable chunks after they reach a specified age or size threshold.

Compression ratio: the ratio between the original data size and the compressed data size, indicating compression effectiveness.

Constraint: a rule enforced by the database to maintain data integrity and consistency.

Continuous aggregate: a materialized view that incrementally updates with new data, providing fast access to pre-computed aggregations. See continuous aggregates.

Counter aggregation: aggregating monotonic counter data, handling counter resets and extrapolation.

Cron: a time-based job scheduler in Unix-like computer operating systems.

Cross-region backup: a backup stored in a different geographical region from the primary data for disaster recovery.

D

Data lake: a centralized repository storing structured and unstructured data at scale, often integrated with time-series databases for analytics.

Data lineage: the tracking of data flow from source to destination, including transformations, essential for compliance and debugging.

Data pipeline: automated workflows for moving, transforming, and loading data between systems, often using tools like Apache Kafka or Apache Airflow.

Data migration: the process of moving data from one system, storage type, or format to another. See the migration guides.

Data retention: the practice of storing data for a specified period before deletion, often governed by compliance requirements or storage optimization. See data retention.

Data rollup: the process of summarizing detailed historical data into higher-level aggregates, balancing storage needs with query efficiency.

Data skew: uneven distribution of data across partitions or nodes, potentially causing performance bottlenecks.

Data tiering: a storage management strategy that places data on different storage tiers based on access patterns and performance requirements.

Data type: a classification that specifies which type of value a variable can hold, such as integer, string, or boolean.

Decompress: the process of restoring compressed data to its original, uncompressed state.

Delta: the difference between two values, commonly used in counter aggregations to calculate the change over time.

DHCP (Dynamic Host Configuration Protocol): a network management protocol used to automatically assign IP addresses and other network configuration parameters.

Dimension: a partitioning key in a hypertable that determines how data is distributed across chunks.

Disaster recovery: the process and procedures for recovering and protecting a business's IT infrastructure in the event of a disaster.

Double precision: a floating-point data type that provides more precision than the standard float type.

Downsample: the process of reducing the temporal resolution of time-series data by aggregating data points over longer time intervals.

Downtime: the period during which a system, service, or application is unavailable or not operational.

Dual-write and backfill: a migration approach where new data is written to both the source and target databases simultaneously, followed by backfilling historical data to ensure completeness.

Dual-write: a migration pattern where applications write data to both the source and target systems simultaneously.

E

Edge computing: processing data at or near the data source such as IoT devices, rather than solely in centralized servers, reducing latency.

Edge gateway: a device that aggregates data from sensors and performs preprocessing before sending data to cloud or centralized databases.

ELT (Extract, Load, Transform): a data pipeline pattern where raw data is loaded first, then transformed within the target system, leveraging database processing power.

Embedding: a vector representation of data such as text or images, that captures semantic meaning in a high-dimensional space.

Error rate: the percentage of requests or operations that result in errors over a given time period.

Euclidean distance: a measure of the straight-line distance between two points in multidimensional space.

Exactly-once: a message is delivered and processed precisely once. There is no loss and no duplicates.

Explain: a Postgres command that shows the execution plan for a query, useful for performance analysis.

Event sourcing: an architectural pattern storing all changes as a sequence of events, naturally fitting time-series database capabilities.

Event-driven architecture: a design pattern where components react to events such as sensor readings, requiring real-time data pipelines and storage.

Extension: a Postgres add-on that extends the database's functionality beyond the core features.

F

Fact table: the central table in a star schema containing quantitative measures, often time-series data with foreign keys to dimension tables.

Failover: the automatic switching to a backup system, server, or network upon the failure or abnormal termination of the primary system.

Financial time-series: high-volume, timestamped datasets like stock market feeds or trade logs, requiring low-latency, scalable databases like TimescaleDB.

Foreign key: a database constraint that establishes a link between data in two tables by referencing the primary key of another table.

Fork: a copy of a database service that shares the same data but can diverge independently through separate writes.

Free service: a free instance of Tiger Cloud with limited resources. You can create up to two free services under any pricing plan. When a free service reaches the resource limit, it converts to the read-only state. You can convert a free service to a standard one under paid pricing plans.

FTP (File Transfer Protocol): a standard network protocol used for transferring files between a client and server on a computer network.

G

Gap filling: a technique for handling missing data points in time-series by interpolation or other methods, often implemented with hyperfunctions.

GIN (Generalized Inverted Index): a Postgres index type designed for indexing composite values and supporting fast searches.

GiST (Generalized Search Tree): a Postgres index type that provides a framework for implementing custom index types.

GP-LTTB: an advanced downsampling algorithm that extends Largest-Triangle-Three-Buckets with Gaussian Process modeling.

GUC (Grand Unified Configuration): Postgres's configuration parameter system that controls various aspects of database behavior.

GUID (Globally Unique Identifier): a unique identifier used in software applications, typically represented as a 128-bit value.

H

Hash: an index type that provides constant-time lookups for equality comparisons but doesn't support range queries.

High-cardinality: refers to datasets with a large number of unique values, which can strain storage and indexing in time-series applications.

Histogram bucket: a predefined range of metrics organized for statistical analysis, commonly visualized in monitoring tools.

Hot standby: a replication configuration where the standby server can serve read-only queries while staying synchronized with the primary.

High availability: a system design that ensures an agreed level of operational performance, usually uptime, for a higher than normal period.

High: in financial data, the highest price of a security during a specific time period.

Histogram: a graphical representation of the distribution of numerical data, showing the frequency of data points in different ranges.

Historical data: previously recorded data that provides context and trends for analysis and decision-making.

HNSW (Hierarchical Navigable Small World): a graph-based algorithm for approximate nearest neighbor search in high-dimensional spaces.

Hot storage: a tier of data storage for frequently accessed data that provides the fastest access times but at higher cost.

Hypercore: TimescaleDB's hybrid storage engine that seamlessly combines row and column storage for optimal performance. See Hypercore.

Hyperfunction: an SQL function in TimescaleDB designed for time-series analysis, statistics, and specialized computations. See Hyperfunctions.

HyperLogLog: a probabilistic data structure used for estimating the cardinality of large datasets with minimal memory usage.

Hypershift: a migration tool and strategy for moving data to TimescaleDB with minimal downtime.

Hypertable: TimescaleDB's core abstraction that automatically partitions time-series data for scalability. See Hypertables.

I

Idempotency: the property where repeated operations produce the same result, crucial for reliable data ingestion and processing.

Ingest rate: the speed at which new data is written to the system, measured in rows per second. Critical for IoT and observability.

Inner product: a mathematical operation that combines two vectors to produce a scalar, used in similarity calculations.

Insert: an SQL operation that adds new rows of data to a database table.

Integer: a data type that represents whole numbers without decimal points.

Intercept: a statistical measure representing the y-intercept in linear regression analysis.

Internet gateway: an AWS VPC component that enables communication between instances in a VPC and the internet.

Interpolation: a method of estimating unknown values that fall between known data points.

IP allow list: a security feature that restricts access to specified IP addresses or ranges.

Isolation level: a database transaction property that defines the degree to which operations in one transaction are isolated from those in other concurrent transactions.

J

Job: an automated task scheduled to run at specific intervals or triggered by certain conditions.

Job execution: the process of running scheduled background tasks or automated procedures.

JIT (Just-In-Time) compilation: Postgres feature that compiles frequently executed query parts for improved performance, available in TimescaleDB.

Job history: a record of past job executions, including their status, duration, and any errors encountered.

JSON (JavaScript Object Notation): a lightweight data interchange format that is easy for humans to read and write.

JWT (JSON Web Token): a compact, URL-safe means of representing claims to be transferred between two parties.

L

Latency: the time delay between a request being made and the response being received.

Lifecycle policy: a set of rules that automatically manage data throughout its lifecycle, including retention and deletion.

Live migration: a data migration technique that moves data with minimal or zero downtime.

Load balancer: a service distributing traffic across servers or database nodes to optimize resource use and avoid single points of failure.

Log-Structured Merge (LSM) Tree: a data structure optimized for write-heavy workloads, though TimescaleDB primarily uses B-tree indexes for balanced read/write performance.

LlamaIndex: a framework for building applications with large language models, providing tools for data ingestion and querying.

LOCF (Last Observation Carried Forward): a method for handling missing data by using the most recent known value.

Logical backup: a backup method that exports data in a human-readable format, allowing for selective restoration.

Logical replication: a Postgres feature that replicates data changes at the logical level rather than the physical level.

Logging: the process of recording events, errors, and system activities for monitoring and troubleshooting purposes.

Low: in financial data, the lowest price of a security during a specific time period.

LTTB (Largest-Triangle-Three-Buckets): a downsampling algorithm that preserves the visual characteristics of time-series data.

M

Manhattan distance: a distance metric calculated as the sum of the absolute differences of their coordinates.

Manual compression: the process of compressing chunks manually rather than through automated policies.

Materialization: the process of computing and storing the results of a query or view for faster access.

Materialized view: a database object that stores the result of a query and can be refreshed periodically.

Memory-optimized query: a query pattern designed to minimize disk I/O by leveraging available RAM and efficient data structures.

Metric: a quantitative measurement used to assess system performance, business outcomes, or operational efficiency.

MFA (Multi-Factor Authentication): a security method that requires two or more verification factors to grant access.

Migration: the process of moving data, applications, or systems from one environment to another. See migration guides.

Monitoring: the continuous observation and measurement of system performance and health.

Multi-tenancy: an architecture pattern supporting multiple customers or applications within a single database instance, with proper isolation.

MQTT (Message Queuing Telemetry Transport): a lightweight messaging protocol designed for small sensors and mobile devices.

MST (Managed Service for TimescaleDB): a fully managed TimescaleDB service that handles infrastructure and maintenance tasks.

N

NAT Gateway: a network address translation service that enables instances in a private subnet to connect to the internet.

Node (database node): an individual server within a distributed system, contributing to storage, compute, or replication tasks.

Normalization: database design technique organizing data to reduce redundancy, though time-series data often benefits from denormalized structures.

Not null: a database constraint that ensures a column cannot contain empty values.

Numeric: a Postgres data type for storing exact numeric values with user-defined precision.

O

OAuth: an open standard for access delegation commonly used for token-based authentication and authorization.

Observability: the ability to measure the internal states of a system by examining its outputs.

OLAP (Online Analytical Processing): systems or workloads focused on large-scale, multidimensional, and complex analytical queries.

OLTP (Online Transaction Processing): high-speed transactional systems optimized for data inserts, updates, and short queries.

OHLC: an acronym for Open, High, Low, Close prices, commonly used in financial data analysis.

OHLCV: an extension of OHLC that includes Volume data for complete candlestick analysis.

Open: in financial data, the opening price of a security at the beginning of a trading period.

OpenTelemetry: open standard for collecting, processing, and exporting telemetry data, often stored in time-series databases.

Optimization: the process of making systems, queries, or operations more efficient and performant.

P

Parallel copy: a technique for copying large amounts of data using multiple concurrent processes to improve performance.

Parallel Query Execution: a Postgres feature that uses multiple CPU cores to execute single queries faster, inherited by TimescaleDB.

Partitioning: the practice of dividing large tables into smaller, more manageable pieces based on certain criteria.

Percentile: a statistical measure that indicates the value below which a certain percentage of observations fall.

Performance: a measure of how efficiently a system operates, often quantified by metrics like throughput, latency, and resource utilization.

pg_basebackup: a Postgres utility for taking base backups of a running Postgres cluster.

pg_dump: a Postgres utility for backing up database objects and data in various formats.

pg_restore: a Postgres utility for restoring databases from backup files created by pg_dump.

pgVector: a Postgres extension that adds vector similarity search capabilities for AI and machine learning applications. See pgvector.

pgai on Tiger Cloud: a cloud solution for building search, RAG, and AI agents with Postgres. Enables calling AI embedding and generation models directly from the database using SQL. See pgai.

pgvectorscale: a performance enhancement for pgvector featuring StreamingDiskANN indexing, binary quantization compression, and label-based filtering. See pgvectorscale.

pgvectorizer: a TimescaleDB tool for automatically vectorizing and indexing data for similarity search.

Physical backup: a backup method that copies the actual database files at the storage level.

PITR (Point-in-Time Recovery): the ability to restore a database to a specific moment in time.

Policy: an automated rule or procedure that performs maintenance tasks like compression, retention, or refresh operations.

Predictive maintenance: the use of time-series data to forecast equipment failure, common in IoT and industrial applications.

Postgres: an open-source object-relational database system known for its reliability, robustness, and performance.

PostGIS: a Postgres extension that adds support for geographic objects and spatial queries.

Primary key: a database constraint that uniquely identifies each row in a table.

psql: an interactive terminal-based front-end to Postgres that allows users to type queries interactively.

Q

QPS (Queries Per Second): a measure of database performance indicating how many queries a database can process per second.

Query: a request for data or information from a database, typically written in SQL.

Query performance: a measure of how efficiently database queries execute, including factors like execution time and resource usage.

Query planner/optimizer: a component determining the most efficient strategy for executing SQL queries based on database structure and indexes.

Query planning: the database process of determining the most efficient way to execute a query.

R

RBAC (Role-Based Access Control): a security model that assigns permissions to users based on their roles within an organization.

Read committed: an isolation level where transactions can read committed changes made by other transactions.

Read scaling: a technique for improving database performance by distributing read queries across multiple database replicas.

Read uncommitted: the lowest isolation level where transactions can read uncommitted changes from other transactions.

Read-only role: a database role with permissions limited to reading data without modification capabilities.

Read replica: a copy of the primary database that serves read-only queries, improving read scalability and geographic distribution.

Real-time analytics: the immediate analysis of incoming data streams, crucial for observability, trading platforms, and IoT monitoring.

Real: a Postgres data type for storing single-precision floating-point numbers.

Real-time aggregate: a continuous aggregate that includes both materialized historical data and real-time calculations on recent data.

Refresh policy: an automated rule that determines when and how continuous aggregates are updated with new data.

Region: a geographical area containing multiple data centers, used in cloud computing for data locality and compliance.

Repeatable read: an isolation level that ensures a transaction sees a consistent snapshot of data throughout its execution.

Replica: a copy of a database that can be used for read scaling, backup, or disaster recovery purposes.

Replication: the process of copying and maintaining data across multiple database instances to ensure availability and durability.

Response time: the time it takes for a system to respond to a request, measured from request initiation to response completion.

REST API: a web service architecture that uses HTTP methods to enable communication between applications.

Restore: the process of recovering data from backups to restore a database to a previous state.

Restore point: a snapshot of database state that can be used as a reference point for recovery operations.

Retention policy: an automated rule that determines how long data is kept before being deleted from the system.

Route table: a set of rules that determine where network traffic is directed within a cloud network.

RTO (Recovery Time Objective): the maximum acceptable time that systems can be down after a failure or disaster.

RPO (Recovery Point Objective): the maximum acceptable amount of data loss measured in time after a failure or disaster.

Rowstore: traditional row-oriented data storage where data is stored row by row, optimized for transactional workloads.

S

SAML (Security Assertion Markup Language): an XML-based standard for exchanging authentication and authorization data between security domains.

Scheduled job: an automated task that runs at predetermined times or intervals.

Schema evolution: the process of modifying database structure over time while maintaining compatibility with existing applications.

Schema: the structure of a database, including tables, columns, relationships, and constraints.

Security group: a virtual firewall that controls inbound and outbound traffic for cloud resources.

Service discovery: mechanisms allowing applications to dynamically locate services like database endpoints, often used in distributed environments.

Segmentwise recompression: a TimescaleDB compression technique that recompresses data segments to improve compression ratios.

Serializable: the highest isolation level that ensures transactions appear to run serially even when executed concurrently.

Service: see Tiger Cloud service.

Sharding: horizontal partitioning of data across multiple database instances, distributing load and enabling linear scalability.

SFTP (SSH File Transfer Protocol): a secure version of FTP that encrypts both commands and data during transmission.

SkipScan: query optimization for DISTINCT operations that incrementally jumps between ordered values without reading intermediate rows. Uses a Custom Scan node to efficiently traverse ordered indexes, dramatically improving performance over traditional DISTINCT queries.

Similarity search: a technique for finding items that are similar to a given query item, often used with vector embeddings.

SLA (Service Level Agreement): a contract that defines the expected level of service between a provider and customer.

SLI (Service Level Indicator): a quantitative measure of some aspect of service quality.

SLO (Service Level Objective): a target value or range for service quality measured by an SLI.

Slope: a statistical measure representing the rate of change in linear regression analysis.

SMTP (Simple Mail Transfer Protocol): an internet standard for email transmission across networks.

Snapshot: a point-in-time copy of data that can be used for backup and recovery purposes.

SP-GiST (Space-Partitioned Generalized Search Tree): a Postgres index type for data structures that naturally partition search spaces.

Storage optimization: techniques for reducing storage costs and improving performance through compression, tiering, and efficient data organization.

Streaming data: continuous flows of data generated by devices, logs, or sensors, requiring high-ingest, real-time storage solutions.

SQL (Structured Query Language): a programming language designed for managing and querying relational databases.

SSH (Secure Shell): a cryptographic network protocol for secure communication over an unsecured network.

SSL (Secure Sockets Layer): a security protocol that establishes encrypted links between networked computers.

Standard service: a regular Tiger Cloud service that includes the resources and features according to the pricing plan. You can create standard services under any of the paid plans.

Streaming replication: a Postgres replication method that continuously sends write-ahead log records to standby servers.

Synthetic monitoring: simulated transactions or probes used to test system health, generating time-series metrics for performance analysis.

T

Table: a database object that stores data in rows and columns, similar to a spreadsheet.

Tablespace: a Postgres storage structure that defines where database objects are physically stored on disk.

TCP (Transmission Control Protocol): a connection-oriented protocol that ensures reliable data transmission between applications.

TDigest: a probabilistic data structure for accurate estimation of percentiles in distributed systems.

Telemetry: the collection of real-time data from systems or devices for monitoring and analysis.

Text: a Postgres data type for storing variable-length character strings.

Throughput: a measure of system performance indicating the amount of work performed or data processed per unit of time.

Tiered storage: a storage strategy that automatically moves data between different storage classes based on access patterns and age.

Tiger Cloud: Tiger Data's managed cloud platform that provides TimescaleDB as a fully managed solution with additional features.

Tiger Lake: Tiger Data's service for integrating operational databases with data lake architectures.

Tiger Cloud service: an instance of optimized Postgres extended with database engine innovations such as TimescaleDB, in a cloud infrastructure that delivers speed without sacrifice. You can create free services and standard services.

Time series: data points indexed and ordered by time, typically representing how values change over time.

Time-weighted average: a statistical calculation that gives more weight to values based on the duration they were held.

Time bucketing: grouping timestamps into uniform intervals for analysis, commonly used with hyperfunctions.

Time-series forecasting: the application of statistical models to time-series data to predict future trends or events.

TimescaleDB: an open-source Postgres extension for real-time analytics that provides scalability and performance optimizations.

Timestamp: a data type that stores date and time information without timezone data.

Timestamptz: a Postgres data type that stores timestamp with timezone information.

TLS (Transport Layer Security): a cryptographic protocol that provides security for communication over networks.

Tombstone: marker indicating deleted data in append-only systems, requiring periodic cleanup processes.

Transaction isolation: the database property controlling the visibility of uncommitted changes between concurrent transactions.

TPS (Transactions Per Second): a measure of database performance indicating transaction processing capacity.

Transaction: a unit of work performed against a database that must be completed entirely or not at all.

Trigger: a database procedure that automatically executes in response to certain events on a table or view.

U

UDP (User Datagram Protocol): a connectionless communication protocol that provides fast but unreliable data transmission.

Unique: a database constraint that ensures all values in a column or combination of columns are distinct.

Uptime: the amount of time that a system has been operational and available for use.

Usage-based storage: a billing model where storage costs are based on actual data stored rather than provisioned capacity.

UUID (Universally Unique Identifier): a 128-bit identifier used to uniquely identify information without central coordination.

V

Vacuum: a Postgres maintenance operation that reclaims storage and updates database statistics.

Varchar: a variable-length character data type that can store strings up to a specified maximum length.

Vector operations: SIMD (Single Instruction, Multiple Data) optimizations for processing arrays of data, improving analytical query performance.

Vertical scaling (scale up): increasing system capacity by adding more power (CPU, RAM) to existing machines, as opposed to horizontal scaling.

Visualization tool: a platform or dashboard used to display time-series data in charts, graphs, and alerts for easier monitoring and analysis.

Vector: a mathematical object with magnitude and direction, used in machine learning for representing data as numerical arrays.

VPC (Virtual Private Cloud): a virtual network dedicated to your cloud account that provides network isolation.

VWAP (Volume Weighted Average Price): a financial indicator that shows the average price weighted by volume over a specific time period.

W

WAL (Write-Ahead Log): Postgres's method for ensuring data integrity by writing changes to a log before applying them to data files.

Warm storage: a storage tier that balances access speed and cost, suitable for data accessed occasionally.

Watermark: a timestamp that tracks the progress of continuous aggregate materialization.

WebSocket: a communication protocol that provides full-duplex communication channels over a single TCP connection.

Window function: an SQL function that performs calculations across related rows, particularly useful for time-series analytics and trend analysis.

Workload management: techniques for prioritizing and scheduling different types of database operations to optimize overall system performance.

X

XML (eXtensible Markup Language): a markup language that defines rules for encoding documents in a format that is both human-readable and machine-readable.

Y

YAML (YAML Ain't Markup Language): a human-readable data serialization standard commonly used for configuration files.

Z

Zero downtime: a system design goal where services remain available during maintenance, upgrades, or migrations without interruption.

Zero-downtime migration: migration strategies that maintain service availability throughout the transition process, often using techniques like dual-write and gradual cutover.

===== PAGE: https://docs.tigerdata.com/api/compression/ =====

Compression

Old API since TimescaleDB v2.18.0 Replaced by Hypercore.

Compression functionality is included in Hypercore.

Before you set up compression, you need to configure the hypertable for compression and then set up a compression policy.

Before you set up compression for the first time, read the compression blog post and documentation.

You can also compress chunks manually, instead of using an automated compression policy to compress chunks as they age.

Compressed chunks have the following limitations:

• ROW LEVEL SECURITY is not supported on compressed chunks.
• Creation of unique constraints on compressed chunks is not supported. You can add them by disabling compression on the hypertable and re-enabling after constraint creation.

Restrictions

In general, compressing a hypertable imposes some limitations on the types of data modifications that you can perform on data inside a compressed chunk.

This table shows changes to the compression feature, added in different versions of TimescaleDB:

TimescaleDB version	Supported data modifications on compressed chunks
1.5 - 2.0	Data and schema modifications are not supported.
2.1 - 2.2	Schema may be modified on compressed hypertables. Data modification not supported.
2.3	Schema modifications and basic insert of new data is allowed. Deleting, updating and some advanced insert statements are not supported.
2.11	Deleting, updating and advanced insert statements are supported.

In TimescaleDB 2.1 and later, you can modify the schema of hypertables that have compressed chunks. Specifically, you can add columns to and rename existing columns of compressed hypertables.

In TimescaleDB v2.3 and later, you can insert data into compressed chunks and to enable compression policies on distributed hypertables.

In TimescaleDB v2.11 and later, you can update and delete compressed data. You can also use advanced insert statements like ON CONFLICT and RETURNING.

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/ =====

Distributed hypertables ( Sunsetted v2.14.x )

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Distributed hypertables are an extension of regular hypertables, available when using a multi-node installation of TimescaleDB. Distributed hypertables provide the ability to store data chunks across multiple data nodes for better scale-out performance.

Most management APIs used with regular hypertable chunks also work with distributed hypertables as documented in this section. There are a number of APIs for specifically dealing with data nodes and a special API for executing SQL commands on data nodes.

===== PAGE: https://docs.tigerdata.com/self-hosted/install/ =====

Install self-hosted TimescaleDB

TimescaleDB is an open-source Postgres extension that powers Tiger Cloud. Designed for running real-time analytics on time-series data, it supercharges ingest, query, storage, and analytics performance.

You can install self-hosted TimescaleDB from source, with a pre-built Docker container, or on one of the supported platforms. This section provides instructions for installing the latest version of self-hosted TimescaleDB.

The following instructions are for development and testing installations. For a production environment, we strongly recommend that you implement the following, many of which you can achieve using Postgres tooling:

• Incremental backup and database snapshots, with efficient point-in-time recovery.
• High availability replication, ideally with nodes across multiple availability zones.
• Automatic failure detection with fast restarts, for both non-replicated and replicated deployments.
• Asynchronous replicas for scaling reads when needed.
• Connection poolers for scaling client connections.
• Zero-down-time minor version and extension upgrades.
• Forking workflows for major version upgrades and other feature testing.
• Monitoring and observability.

Deploying for production? With a Tiger Cloud service we tune your database for performance and handle scalability, high availability, backups, and management, so you can relax.

Installation

Refer to the installation documentation for detailed setup instructions.

For more details about the latest release, see the release notes section.

===== PAGE: https://docs.tigerdata.com/self-hosted/configuration/ =====

Configuration

By default, TimescaleDB uses the default Postgres server configuration settings. However, in some cases, these settings are not appropriate, especially if you have larger servers that use more hardware resources such as CPU, memory, and storage.

• Learn about configuration to understand how it works before you begin using it.
• Use the TimescaleDB tune tool.
• Manually edit the postgresql.conf configuration file.
• If you run TimescaleDB in a Docker container, configure within Docker.
• Find out more about the data that we collect.

===== PAGE: https://docs.tigerdata.com/self-hosted/backup-and-restore/ =====

Backup and restore

TimescaleDB takes advantage of the reliable backup and restore functionality provided by Postgres. There are a few different mechanisms you can use to back up your self-hosted TimescaleDB database:

• Logical backup with pg_dump and pg_restore.
• Physical backup with pg_basebackup or another tool.
• DEPRECATED Ongoing physical backups using write-ahead log (WAL) archiving.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

===== PAGE: https://docs.tigerdata.com/self-hosted/migration/ =====

Migrate your Postgres database to self-hosted TimescaleDB

You can migrate your existing Postgres database to self-hosted TimescaleDB.

There are several methods for migrating your data:

• If the database you want to migrate is smaller than 100 GB, migrate your entire database at once: This method directly transfers all data and schemas, including Timescale-specific features. Your hypertables, continuous aggregates, and policies are automatically available in the new self-hosted TimescaleDB instance.
• For databases larger than 100GB, migrate your schema and data separately: With this method, you migrate your tables one by one for easier failure recovery. If migration fails mid-way, you can restart from the failure point rather than from the beginning. However, Timescale-specific features won't be automatically migrated. Follow the instructions to restore your hypertables, continuous aggregates, and policies.
• If you need to move data from Postgres tables into hypertables within an existing self-hosted TimescaleDB instance, migrate within the same database: This method assumes that you have TimescaleDB set up in the same database instance as your existing table.
• If you have data in an InfluxDB database, migrate using Outflux: Outflux pipes exported data directly to your self-hosted TimescaleDB instance, and manages schema discovery, validation, and creation. Outflux works with earlier versions of InfluxDB. It does not work with InfluxDB version 2 and later.

Choose a migration method

Which method you choose depends on your database size, network upload and download speeds, existing continuous aggregates, and tolerance for failure recovery.

If you are migrating from an Amazon RDS service, Amazon charges for the amount of data transferred out of the service. You could be charged by Amazon for all data egressed, even if the migration fails.

If your database is smaller than 100 GB, choose to migrate your entire database at once. You can also migrate larger databases using this method, but the copying process must keep running, potentially over days or weeks. If the copy is interrupted, the process needs to be restarted. If you think an interruption in the copy is possible, choose to migrate your schema and data separately instead.

Migrating your schema and data separately does not retain continuous aggregates calculated using already-deleted data. For example, if you delete raw data after a month but retain downsampled data in a continuous aggregate for a year, the continuous aggregate loses any data older than a month upon migration. If you must keep continuous aggregates calculated using deleted data, migrate your entire database at once regardless of database size.

If you aren't sure which method to use, try copying the entire database at once to estimate the time required. If the time estimate is very long, stop the migration and switch to the other method.

Migrate an active database

If your database is actively ingesting data, take precautions to ensure that your self-hosted TimescaleDB instance contains the data that is ingested while the migration is happening. Begin by running ingest in parallel on the source and target databases. This ensures that the newest data is written to both databases. Then backfill your data with one of the two migration methods.

===== PAGE: https://docs.tigerdata.com/self-hosted/manage-storage/ =====

Manage storage using tablespaces

If you are running TimescaleDB on your own hardware, you can save storage by moving chunks between tablespaces. By moving older chunks to cheaper, slower storage, you can save on storage costs while still using faster, more expensive storage for frequently accessed data. Moving infrequently accessed chunks can also improve performance, because it isolates historical data from the continual read-and-write workload of more recent data.

Using tablespaces is one way to manage data storage costs with TimescaleDB. You can also use compression and data retention to reduce your storage requirements.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

Move data

To move chunks to a new tablespace, you first need to create the new tablespace and set the storage mount point. You can then use the move_chunk API call to move individual chunks from the default tablespace to the new tablespace. The move_chunk command also allows you to move indexes belonging to those chunks to an appropriate tablespace.

Additionally, move_chunk allows you reorder the chunk during the migration. This can be used to make your queries faster, and works in a similar way to the reorder_chunk command.

You must be logged in as a super user, such as the postgres user, to use the move_chunk() API call.

Moving data

1. Create a new tablespace. In this example, the tablespace is called history, it is owned by the postgres super user, and the mount point is /mnt/history:

   CREATE TABLESPACE history
   OWNER postgres
   LOCATION '/mnt/history';
   

2. List chunks that you want to move. In this example, chunks that contain data that is older than two days:

   SELECT show_chunks('conditions', older_than => INTERVAL '2 days');
   

3. Move a chunk and its index to the new tablespace. You can also reorder the data in this step. In this example, the chunk called _timescaledb_internal._hyper_1_4_chunk is moved to the history tablespace, and is reordered based on its time index:

   SELECT move_chunk(
     chunk => '_timescaledb_internal._hyper_1_4_chunk',
     destination_tablespace => 'history',
     index_destination_tablespace => 'history',
     reorder_index => '_timescaledb_internal._hyper_1_4_chunk_netdata_time_idx',
     verbose => TRUE
   );
   

4. You can verify that the chunk now resides in the correct tablespace by querying pg_tables to list all of the chunks on the tablespace:

   SELECT tablename from pg_tables
     WHERE tablespace = 'history' and tablename like '_hyper_%_%_chunk';
   

   You can also verify that the index is in the correct location:

   SELECT indexname FROM pg_indexes WHERE tablespace = 'history';
   

Move data in bulk

To move several chunks at once, select the chunks you want to move by using FROM show_chunks(...). For example, to move chunks containing data between 1 and 3 weeks old, in a hypertable named example:

SELECT move_chunk(
  chunk => i,
  destination_tablespace => '')
FROM show_chunks('example', now() - INTERVAL '1 week', now() - INTERVAL '3 weeks') i;

Examples

After moving a chunk to a slower tablespace, you can move it back to the default, faster tablespace:

SELECT move_chunk(
  chunk => '_timescaledb_internal._hyper_1_4_chunk',
  destination_tablespace => 'pg_default',
  index_destination_tablespace => 'pg_default',
  reorder_index => '_timescaledb_internal._hyper_1_4_chunk_netdata_time_idx'
);

You can move a data chunk to the slower tablespace, but keep the chunk's indexes on the default, faster tablespace:

SELECT move_chunk(
  chunk => '_timescaledb_internal._hyper_1_4_chunk',
  destination_tablespace => 'history',
  index_destination_tablespace => 'pg_default',
  reorder_index => '_timescaledb_internal._hyper_1_4_chunk_netdata_time_idx'
);

You can also keep the data in pg_default but move the index to history. Alternatively, you can set up a third tablespace called history_indexes, and move the data to history and the indexes to history_indexes.

In TimescaleDB v2.0 and later, you can use move_chunk with the job scheduler framework. For more information, see the jobs section.

===== PAGE: https://docs.tigerdata.com/self-hosted/replication-and-ha/ =====

High availability

Postgres relies on replication for high availability, failover, and balancing read loads across multiple nodes. Replication ensures that data written to the primary Postgres database is mirrored on one or more nodes. By virtue of having multiple nodes with an exact copy of the primary database available, the primary database can be replaced with a replica node in the event of a failure or outage on the primary server. Replica nodes can also be used as read only databases, also called read replicas, allowing reads to be horizontally scaled by spreading the read query volume across multiple nodes.

• Learn about high availability to understand how it works before you begin using it.
• Configure replication.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

===== PAGE: https://docs.tigerdata.com/self-hosted/tooling/ =====

Additional tooling

Get the most from TimescaleDB with open source tools that help you perform common tasks.

• Automatically configure your TimescaleDB instance with timescaledb-tune
• Install TimescaleDB Toolkit to access more hyperfunctions and function pipelines

===== PAGE: https://docs.tigerdata.com/self-hosted/upgrades/ =====

Upgrade TimescaleDB

A major upgrade is when you update from TimescaleDB X.<minor version> to Y.<minor version>. A minor upgrade is when you update from TimescaleDB <major version>.x, to TimescaleDB <major version>.y. You upgrade your self-hosted TimescaleDB installation in-place.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

This section shows you how to:

• Upgrade self-hosted TimescaleDB to a new minor version.
• Upgrade self-hosted TimescaleDB to a new major version.
• Upgrade self-hosted TimescaleDB running in a Docker container to a new minor version.
• Upgrade Postgres to a new version.
• Downgrade self-hosted TimescaleDB to the previous minor version.

===== PAGE: https://docs.tigerdata.com/self-hosted/uninstall/ =====

Uninstall TimescaleDB

If you want to uninstall TimescaleDB because it does not meet your requirements, you can uninstall it without having to uninstall Postgres.

• Learn how to uninstall TimescaleDB in macOS

===== PAGE: https://docs.tigerdata.com/self-hosted/multinode-timescaledb/ =====

Multi-node

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

If you have a larger workload, you might need more than one TimescaleDB instance. TimescaleDB multi-node allows you to run and manage multiple instances, giving you faster data ingest, and more responsive and efficient queries.

• Learn about multi-node to understand how it works before you begin using it.
• Set up multi-node in a self-hosted environment.
• Set up authentication for your cluster
• Configure your cluster
• Administer your cluster
• Grow or shrink your cluster
• Set up high availability (HA) for your cluster
• Maintain your multi-node environment

===== PAGE: https://docs.tigerdata.com/self-hosted/distributed-hypertables/ =====

Distributed hypertables

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Distributed hypertables are hypertables that span multiple nodes. With distributed hypertables, you can scale your data storage across multiple machines and benefit from parallelized processing for some queries.

Many features of distributed hypertables work the same way as standard hypertables. To learn how hypertables work in general, see the hypertables section.

In this section:

• Learn about distributed hypertables for multi-node databases
• Create a distributed hypertable
• Insert data into distributed hypertables
• Query data in distributed hypertables
• Alter and drop distributed hypertables
• Create foreign keys on distributed hypertables
• Set triggers on distributed hypertables

===== PAGE: https://docs.tigerdata.com/mst/about-mst/ =====

About Managed Service for TimescaleDB

Managed Service for TimescaleDB (MST) is TimescaleDB hosted on Azure and GCP. MST is offered in partnership with Aiven.

Tiger Cloud is a high-performance developer focused cloud that provides Postgres services enhanced with our blazing fast vector search. You can securely integrate Tiger Cloud with your AWS, GCS or Azure infrastructure. Create a Tiger Cloud service and try for free.

If you need to run TimescaleDB on GCP or Azure, you're in the right place — keep reading.

Your Managed Service for TimescaleDB account has three main components: projects, services, and databases.

Projects

When you sign up for Managed Service for TimescaleDB, an empty project is created for you automatically. Projects are the highest organization level, and they contain all your services and databases. You can use projects to organize groups of services. Each project can also have its own billing settings.

To create a new project: In MST Console, click Projects > Create project.

services

Each project contains one or more services. You can have multiple services under each project, and each service corresponds to a cloud service provider tier. You can access all your services from the Services tab within your projects.

For more information about getting your first service up and running, see the Managed Service for TimescaleDB installation section.

When you have created, and named, a new Managed Service for TimescaleDB service, you cannot rename it. If you need to have your service running under a different name, you need to create a new service, and manually migrate the data. For more information about migrating data, see migrating your data.

For information about billing on Managed Service for TimescaleDB, see the billing section.

Databases

Each service can contain one or more databases. To view existing databases, or to create a new database, select a service in the services list, click Databases, then click Create database.

Service level agreement

Managed Service for TimescaleDB is provided through a partnership with Aiven. This provides you with a service commitment to deliver 99.99% availability. For more information, see the Aiven Service Level Agreement policy.

Service configuration plans

When you create a new service, you need to select a configuration plan. The plan determines the number of VMs the service runs in, the high availability configuration, the number of CPU cores, and size of RAM and storage volumes.

The plans are:

• Basic Plans: include 2 days of backups and automatic backup and restore if your instance fails.
• Dev Plans: include 1 day of backups and automatic backup and restore if your instance fails.
• Pro Plans: include 3 days of backups and automatic failover to a hot standby if your instance fails.

The Basic and Dev plans are serviced by a single virtual machine (VM) node. This means that if the node fails, the service is unavailable until a new VM is built. This can result in data loss, if some of the latest changes to the data weren't backed up before the failure. Sometimes, it can also take a long time to return the service back to normal operation, because a new VM needs to be created and restored from backups before the service can resume. The time to recover depends on the amount of data you have to restore.

The Pro plans are much more resilient to failures. A single node failure causes no data loss, and the possible downtime is minimal. If an acting TimescaleDB master node fails, an up-to-date replica node is automatically promoted to become the new master. This means there is only a small outage while applications reconnect to the database and access the new master.

You can upgrade your plan while the service is running. The service is reconfigured to run on larger VMs in the background and when the reconfiguration is complete, the DNS names are pointed to the new hosts. This can cause a short disruption to your service while DNS changes are propagated.

Within each configuration plan option, there are several plan types available:

• IO-Optimized and Compute-Optimized These configurations are optimized for input/output (I/O) performance, using SSD storage media.
• Storage-Optimized: These configurations usually have larger amounts of overall storage, using HDD storage media.
• Dev-Only: These configurations are typically smaller footprints, and lower cost, designed for development and testing scenarios.

High availability

Most minor failures are handled automatically without making any changes to your service deployment. This includes failures such as service process crashes, or a temporary loss of network access. The service automatically restores normal operation when the crashed process restarts automatically or when the network access is restored.

However, more severe failure modes, such as losing a single node entirely, require more drastic recovery measures. Losing an entire node or a virtual machine could happen for example due to hardware failure or a severe software failure.

A failing node is automatically detected by the MST monitoring infrastructure. Either the node starts reporting that its own self-diagnostics is reporting problems or the node stops communicating entirely. The monitoring infrastructure automatically schedules a new replacement node to be created when this happens.

In case of database failover, the service URL of your service remains the same. Only the IP address changes to point at the new master node.

Managed Service for TimescaleDB availability features differ based on the service plan:

• Basic and Dev plans: These are single-node plans. Basic plans include a two-day backup history, and Dev plans include a one-day backup history.
• Pro plans: These are two-node plans with a master and a standby for higher availability, and three-day backup histories.

Single node

In the Basic and Dev plans, if you lose the only node from the service, it immediately starts the automatic process of creating a new replacement node. The new node starts up, restores its state from the latest available backup, and resumes the service. Because there was just a single node providing the service, the service is unavailable for the duration of the restore operation. Also, any writes made since the backup of the latest write-ahead log (WAL) file is lost. Typically this time window is limited to either five minutes, or one WAL file.

Highly available nodes

In Pro plans, if a Postgres standby fails, the master node keeps running normally and provides normal service level to the client applications. When the new replacement standby node is ready and synchronized with the master, it starts replicating the master in real time and normal operation resumes.

If the Postgres master fails, the combined information from the MST monitoring infrastructure and the standby node is used to make a failover decision. On the nodes, the open source monitoring daemon PGLookout, in combination with the information from the MST system infrastructure, reports the failover. If the master node is down completely, the standby node promotes itself as the new master node and immediately starts serving clients. A new replacement node is automatically scheduled and becomes the new standby node.

If both master and standby nodes fail at the same time, two new nodes are automatically scheduled for creation and become the new master and standby nodes respectively. The master node restores itself from the latest available backup, which means that there can be some degree of data loss involved. For example, any writes made since the backup of the latest write-ahead log (WAL) file can be lost.

The amount of time it takes to replace a failed node depends mainly on the cloud region and the amount of data that needs to be restored. However, in the case of services with two-node Pro plans, the surviving node keeps serving clients even during the recreation of the other node. This process is entirely automatic and requires no manual intervention.

For backups and restoration, Managed Service for TimescaleDB uses the open source backup daemon PGHoard that MST maintains. It makes real-time copies of write-ahead log (WAL) files to an object store in a compressed and encrypted format.

Connection limits

Managed Service for TimescaleDB limits the maximum number of connections to each service. The maximum number of allowed connections depends on your service plan. To see the current connection limit for your service, navigate to the service Overview tab and locate the Connection Limit section.

If you have a lot of clients or client threads connecting to your database, use connection pooling to limit the number of connections. For more information about connection pooling, see the connection pooling section.

If you have a high number of connections to your database, your service might run more slowly, and could run out of memory. Remain aware of how many open connections your have to your database at any given time.

Service termination protection

You can protect your services from accidentally being terminated, by enabling service termination protection. When termination protection is enabled, you cannot power down the service from the web console, the REST API, or with a command-line client. To power down a protected service, you need to turn off termination protection first. Termination protection does not interrupt service migrations or upgrades.

To enable service termination protection, navigate to the service Overview tab. Locate the Termination protection section, and toggle to enable protection.

If you run out of free sign-up credit, and have not entered a valid credit card for payment, your service is powered down, even if you have enabled termination protection.

Idle connections

Managed Service for TimescaleDB uses the default keep alive settings for TCP connections. The default settings are:

• tcp_keepalives_idle: 7200
• tcp_keepalive_count: 9
• tcp_keepalives_interval: 75

If you have long idle database connection sessions, you might need to adjust these settings to ensure that your TCP connection remains stable. If you experience a broken TCP connection, when you reconnect make sure that your client resolves the DNS address correctly, as the underlying address changes during automatic failover.

For more information about adjusting keep alive settings, see the Postgres documentation.

Long running queries

Managed Service for TimescaleDB does not cancel database queries. If you have created a query that is taking a very long time, or that has hung, it could lock resources on your service, and could prevent database administration tasks from being performed.

You can find out if you have any long-running queries by navigating to the service Current Queries tab. You can also cancel long running queries from this tab.

Alternatively, you can use your connection client to view running queries with this command:

SELECT * FROM pg_stat_activity
    WHERE state <> 'idle';

Cancel long-running queries using this command, with the PID of the query you want to cancel:

SELECT pg_terminate_backend(<PID>);

If you want to automatically cancel any query that runs over a specified length of time, you can use this command:

SET statement_timeout = <milliseconds>

===== PAGE: https://docs.tigerdata.com/mst/installation-mst/ =====

Get started with Managed Service for TimescaleDB

Managed Service for TimescaleDB (MST) is TimescaleDB hosted on Azure and GCP. MST is offered in partnership with Aiven.

Tiger Cloud is a high-performance developer focused cloud that provides Postgres services enhanced with our blazing fast vector search. You can securely integrate Tiger Cloud with your AWS, GCS or Azure infrastructure. Create a Tiger Cloud service and try for free.

If you need to run TimescaleDB on GCP or Azure, you're in the right place — keep reading.

Create your first service

A service in Managed Service for TimescaleDB is a cloud instance on your chosen cloud provider, which you can install your database on.

Creating your first service

1. Sign in to your MST Console.

2. Click Create service and choose TimescaleDB, and update your preferences:

   

   • In the Select Your Cloud Service Provider field, click your preferred provider.
   • In the Select Your Cloud Service Region field, click your preferred server location. This is often the server that's physically closest to you.
   • In the Select Your Service Plan field, click your preferred plan, based on the hardware configuration you require. If you are in your trial period, and just want to try the service out, or develop a proof of concept, we recommend the Dev plan, because it is the most cost-effective during your trial period.

3. In the information bar on the right of the screen, review the settings you have selected for your service, and click Create Service. The service takes a few minutes to provision.

Connect to your service from the command prompt

When you have a service up and running, you can connect to it from your local system using the psql command-line utility. This is the same tool you might have used to connect to Postgres before, but if you haven't installed it yet, check out the installing psql section.

Connecting to your service from the command prompt

1. Sign in to your MST Console.

2. In the Services tab, find the service you want to connect to, and check it is marked as Running.

3. Click the name of the service you want to connect to see the connection information. Take a note of the host, port, and password.

4. On your local system, at the command prompt, connect to the service, using your own service details:

   psql -x "postgres://tsdbadmin:<PASSWORD>@<HOSTNAME>:<PORT>/defaultdb?sslmode=require"
   

   If your connection is successful, you'll see a message like this, followed by the psql prompt:

   psql (13.3, server 13.4)
   SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
   Type "help" for help.
   defaultdb=>
   

Check that you have the TimescaleDB extension

TimescaleDB is provided as an extension to your Postgres database, and it is enabled by default when you create a new service on Managed Service for TimescaleDB You can check that the TimescaleDB extension is installed by using the \dx command at the psql prompt. It looks like this:

defaultdb=> \dx

List of installed extensions
-[ RECORD 1 ]------------------------------------------------------------------
Name        | plpgsql
Version     | 1.0
Schema      | pg_catalog
Description | PL/pgSQL procedural language
-[ RECORD 2 ]------------------------------------------------------------------
Name        | timescaledb
Version     | 2.5.1
Schema      | public
Description | Enables scalable inserts and complex queries for time-series data

defaultdb=>

Install and update TimescaleDB Toolkit

Run this command on each database you want to use the Toolkit with:

CREATE EXTENSION timescaledb_toolkit;

Update an installed version of the Toolkit using this command:

ALTER EXTENSION timescaledb_toolkit UPDATE;

Where to next

Now that you have your first service up and running, you can check out the Managed Service for TimescaleDB section in the documentation, and find out what you can do with it.

If you want to work through some tutorials to help you get up and running with TimescaleDB and time-series data, check out the tutorials section.

You can always contact us if you need help working something out, or if you want to have a chat.

===== PAGE: https://docs.tigerdata.com/mst/ingest-data/ =====

Ingest data

There are several different ways of ingesting your data into Managed Service for TimescaleDB. This section contains instructions to:

• Bulk upload from a .csv file
• Insert data directly using a client driver, such as JDBC, ODBC, or Node.js
• Insert data directly using a message queue, such as Kafka

Before you begin, make sure you have created your service, and can connect to it using psql.

Preparing your new database

1. Use psql to connect to your service.

   psql -h <HOSTNAME> -p <PORT> -U <USERNAME> -W -d <DATABASE_NAME>
   

   You retrieve the service URL, port, and login credentials from the service overview in the MST dashboard.

2. Create a new database for your data. In this example, the new database is called new_db:

   CREATE DATABASE new_db;
   \c new_db;
   

3. Create a new SQL table in your database. The columns you create for the table must match the columns in your source data. In this example, the table is storing weather condition data, and has columns for the timestamp, location, and temperature:

   CREATE TABLE conditions (
     time        TIMESTAMPTZ         NOT NULL,
     location    text                NOT NULL,
     temperature DOUBLE PRECISION    NULL
   );
   

4. Load the timescaledb Postgres extension:

   CREATE EXTENSION timescaledb;
   \dx
   

5. Convert the SQL table into a hypertable:

   SELECT create_hypertable('conditions', by_range('time'));
   

   The by_range dimension builder is an addition to TimescaleDB 2.13.

When you have successfully set up your new database, you can ingest data using one of these methods.

Bulk upload from CSV files

If you have a dataset stored in a .csv file, you can import it into an empty hypertable. You need to begin by creating the new table, before you import the data.

Before you begin, make sure you have prepared your new database.

Bulk uploading from a CSV file

1. Insert data into the new hypertable using the timescaledb-parallel-copy tool. You should already have the tool installed, but you can install it manually from our GitHub repository if you need to. In this example, we are inserting the data using four workers:

   timescaledb-parallel-copy \
   --connection '<service_url>' \
   --table conditions \
   --file ~/Downloads/example.csv \
   --workers 4 \
   --copy-options "CSV" \
   --skip-header
   

   We recommend that you set the number of workers lower than the number of available CPU cores on your client machine or server, to prevent the workers having to compete for resources. This helps your ingest go faster.

2. OPTIONAL: If you don't want to use the timescaledb-parallel-copy tool, or if you have a very small dataset, you can use the Postgres COPY command instead:

   psql '<service_url>/new_db?sslmode=require' -c "\copy conditions FROM <example.csv> WITH (FORMAT CSV, HEADER)"
   

Insert data directly using a client driver

You can use a client driver such as JDBC, Python, or Node.js, to insert data directly into your new database.

See the Postgres instructions for using the ODBC driver.

See the Code Quick Starts for using various languages, including Python and node.js.

Insert data directly using a message queue

If you have data stored in a message queue, you can import it into your service. This section provides instructions on using the Kafka Connect Postgres connector.

This connector deploys Postgres change events from Kafka Connect to a runtime service. It monitors one or more schemas in a service, and writes all change events to Kafka topics, which can then be independently consumed by one or more clients. Kafka Connect can be distributed to provide fault tolerance, which ensures the connectors are running and continually keeping up with changes in the database.

You can also use the Postgres connector as a library without Kafka or Kafka Connect. This allows applications and services to directly connect to MST and obtain the ordered change events. In this environment, the application must record the progress of the connector so that when it is restarted, the connect can continue where it left off. This approach can be useful for less critical use cases. However, for production use cases, we recommend that you use the connector with Kafka and Kafka Connect.

See these instructions for using the Kafka connector.

===== PAGE: https://docs.tigerdata.com/mst/user-management/ =====

User management

You can add new users, and manage existing users, in MST Console. New users can be added to an entire project, or a single service.

Project members

You can invite new users to join your project as project members. There are several roles available for project members:

Role	Invite more users	Modify billing information	Manage existing services	Start and stop services	View service information
Admin	✅	✅	✅	✅	✅
Operator	❌	❌	✅	✅	✅
Developer	✅	❌	✅	❌	✅
Read-only	❌	❌	❌	❌	✅

Users who can manage existing services can create databases and connect to them, on a service that already exists. To create a new service, users need the start and stop services permission.

Adding project members

1. Sign in to your MST Console.
2. Check that you are in the project that you want to change the members for, and click Members.
3. In the Project members page, type the email address of the member you want to add, and select a role for the member.
4. Click Send invitation.
5. The new user is sent an email inviting them to the project, and the invite shows in the Pending invitations list. You can click Withdraw invitation to remove an invitation before it has been accepted.
6. When they accept the invitation, the user details show in the Members list. You can edit a member role by selecting a new role in the list. You can delete a member by clicking the delete icon in the list.

Service users

By default, when you create a new service, a new tsdbadmin user is created. This is the user that you use to connect to your new service.

The tsdbadmin user is the owner of the database, but is not a superuser. To access features requiring a superuser, log in as the postgres user instead.

The tsdbadmin user for Managed Service for TimescaleDBs can:

• Create a database
• Create a role
• Perform replication
• Bypass row level security (RLS)

This allows you to use the tsdbadmin user to create another user with any other roles. For a complete list of roles available, see the Postgres role attributes documentation.

Your service must be running before you can manage users.

Adding service users

1. Sign in to MST Console. By default, you start in the Services view, showing any services you currently have in your project.

2. Click the name of the service that you want to add users to.

3. Select Users, then click Add service user:

   

4. In the Username field, type a name for your user. If you want to allow the user to be replicated, toggle Allow replication. Click Add service user to save the user.

5. The new user shows in the Username list.

   To view the password, click the eye icon. Use the options in the list to change the replication setting and password, or delete the user.

Multi-factor user authentication

You can use multi-factor authentication (MFA) to log in to MST Console. This requires an authentication code, provided by the Google Authenticator app on your mobile device.

You can see which authentication method is in use by each member of your Managed Service for TimescaleDB project. From the dashboard, navigate to the Members section. Each member is listed in the table with an authentication method of either Password or Two-Factor.

Before you begin, install the Google Authenticator app on your mobile device. For more information, and installation instructions, see the Google Authenticator documentation.

Configuring multi-factor authentication

1. Sign in to MST Console.
2. Click the User information icon in the top-right of the dashboard to go to the User profile section.
3. In the Authentication tab, toggle Two-factor authentication to Enabled, and enter your password.
4. On your mobile device, open the Google Authenticator app, tap + and select Scan a QR code.
5. On your mobile device, scan the QR code provided by Managed Service for TimescaleDB.
6. In your MST dashboard, enter the confirmation code provided by the Google Authenticator app, and click Enable Two-Factor Auth.

If you lose access to the mobile device you use for multi-factor authentication, you cannot sign in to your Managed Service for TimescaleDB account. To regain access to your account, on the login screen, click Forgot password? and follow the step to reset your password. When you have regained access to your account, reconfigure multi-factor authentication.

User authentication tokens

Every time a registered user logs in, Managed Service for TimescaleDB creates a new authentication token. This occurs for login events using the portal, and using the API. By default, authentication tokens expire after 30 days, but the expiry date is adjusted every time the token is used. This means that tokens can be used indefinitely, if the user logs in at least every 30 days.

You can see the list of all current authentication tokens in the Managed Service for TimescaleDB dashboard. Sign in to your account, and click the User information icon in the top-right of the dashboard to go to the User profile section. In the Authentication tab, the table lists all current authentication tokens.

When you make authentication changes, such as enabling two factor authentication or resetting a password, all existing tokens are revoked. In some cases, a new token is immediately created so that the web console session remains valid. You can also manually revoke authentication tokens from the User profile page individually, or click Revoke all tokens to revoke all current tokens.

Additionally, you can click Generate token to create a new token. When you generate a token on this page, you can provide a description, maximum age, and an extension policy. Generating authentication tokens in this way allows you to use them with monitoring applications that make automatic API calls to Managed Service for TimescaleDB.

There is a limit to how many valid authentication tokens are allowed per user. This limit is different for tokens that are created as a result of a sign in operation, and for tokens created explicitly. For automatically created tokens, the system automatically deletes the oldest tokens as new ones are created. For explicitly created tokens, older tokens are not deleted unless they expire or are manually revoked. This can result in explicitly created tokens that stop working, even though they haven't expired or been revoked. To avoid this, make sure you sign out at the end of every user session, instead of just discarding your authentication token. This is especially important for automation tools that automatically sign in.

===== PAGE: https://docs.tigerdata.com/mst/billing/ =====

Billing on Managed Service for TimescaleDB

By default, all new services require a credit card, which is charged at the end of the month for all charges accrued over that month. Each project is charged separately. Your credit card statement records the transaction as coming from Aiven, as Aiven provides billing services for Managed Service for TimescaleDB.

Managed Service for TimescaleDB uses hourly billing. This charge is automatically calculated, based on the services you are running in your project. The price charged for your project includes:

• Virtual machine
• Networking
• Backups
• Setting up

Managed Service for TimescaleDB does not charge you for network traffic used by your service. However, your application cloud service provider might charge you for the network traffic going to or from your service.

Terminating or powering a service down stops the accumulation of new charges immediately. However, the minimum hourly charge unit is one hour. For example, if you launch a service and shut it down after 40 minutes, you are charged for one full hour.

Migrating to different service plan levels does not incur extra charges for the migration itself. Note, though, that some service plan levels are more costly per hour, and your new service is charged at the new rate.

Migrating a service to another cloud region or different cloud provider does not incur extra charges.

All prices listed for Managed Service for TimescaleDB are inclusive of credit card and processing fees. However, in some cases, your credit card provider might charge additional fees, such as an international transaction fee. These fees are not charged by Tiger Data or Aiven.

Billing groups

Create billing groups to set up common billing profiles for projects within an organization. Billing groups make it easier to manage your costs since you receive a consolidated invoice for all projects assigned to a billing group and can pay with one saved payment method.

Billing groups can only be used in one organization. Credits are assigned per billing group and are automatically used to cover charges of any project assigned to that group.

You can track spending by exporting cost information to business intelligence tools using the invoice API.

To access billing groups in MST Console, you must be a super admin or account owner.

Create a billing group

To create a billing group, take the following steps:

1. In MST Console, click Billing > Billing groups > Create billing group.

2. Enter a name for the billing group and click Continue.

3. Enter the billing details.

   You can copy these details from another billing group by selecting it from the list. Click Continue.

4. Select the projects to add to this billing group and click Continue

   You can skip this step and add projects later.

5. Check the information in the Summary step. To make changes to any section, click Edit.

6. When you have confirmed everything is correct, click Create & Assign.

Manage billing groups

To view and update your billing groups, take the following steps:

• Rename billing groups:

  1. In MST Console, go to Billing > Billing groups and find the billing group to rename.
  2. Click Actions > Rename.
  3. Enter the new name and click Rename.

• Update your billing information:

  1. In MST Console, go to Billing > Billing groups and click on the name of the group to update.
  2. Open the Billing information tab and click Edit to update the details for each section.

• Delete billing groups

  1. In MST Console, open Billing > Billing groups and select the group to delete.
  2. On the Projects tab, confirm that the billing group has no projects. If there are projects listed, move them to a different billing group.
  3. Go back to the list of billing groups and click Actions > Delete next to the group to be deleted.

Assign and unassign projects

To manage projects in billing groups, take the following steps.

• Assign projects to a billing group:

  1. In MST Console, go to Billing > Billing groups.
  2. Select the billing group to assign the project to.
  3. On the Projects tab, click Assign projects.
  4. Select the projects and click Assign projects.
  5. Click Cancel to close the dialog box.

  Assigning a project that is already assigned to another billing group will unassign it from that billing group.

• Move a project to another billing group

  1. In MST Console, go to Billing > Billing groups.
  2. Click on the name of the billing group that the project is currently assigned to.
  3. On the Projects tab, find the project to move.
  4. Click the three dots for that project and select the billing group to move it to.

Taxation

Aiven provides billing services for Managed Service for TimescaleDB. These services are provided by Aiven Ltd, a private limited company incorporated in Finland.

If you are within the European Union, Finnish law requires that you are charged a value-added tax (VAT). The VAT percentage depends on where you are domiciled. For business customers in EU countries other than Finland, you can use the reverse charge mechanism of 2006/112/EC article 196, by entering a valid VAT ID into the billing information of your project.

If you are within the United States, no tax is withheld from your payments. In most cases, you do not require a W-8 form to confirm this, however, if you require a W-8BEN-E form describing this status, you can request one.

If you are elsewhere in the world, no taxes are applied to your account, according to the Value-Added Tax Act of Finland, section 69 h.

Corporate billing

If you prefer to pay by invoice, or if you are unable to provide a credit card for billing, you can switch your project to corporate billing instead. Under this model, invoices are generated at the end of the month based on actual usage, and are sent in .pdf format by email to the billing email addresses you configured in your dashboard.

Payment terms for corporate invoices are 14 days net, by bank transfer, to the bank details provided on the invoice. By default, services are charged in US Dollars (USD), but you can request your invoices be sent in either Euros (EUR) or Pounds Sterling (GBP) at the invoice date's currency exchange rates.

To switch from credit card to corporate billing, make sure your billing profile and email address is correct in your project's billing settings, and send a message to the Tiger Data support team asking to be changed to corporate billing.

===== PAGE: https://docs.tigerdata.com/mst/connection-pools/ =====

Connection pools

When you connect to your database, you consume server resources. If you have a lot of connections to your database, you can consume a lot of server resources. One way to mitigate this is to use connection pooling, which allows you to have high numbers of connections, but keep your server resource use low. The more client connections you have to your database, the more useful connection pooling becomes.

By default, Postgres creates a separate backend process for each connection to the server. Connection pooling uses a tool called PGBouncer to pool multiple connections to a single backend process. PGBouncer automatically interleaves the client queries to use a limited number of backend connections more efficiently, leading to lower resource use on the server and better total performance.

Without connection pooling, the database connections are handled directly by Postgres backend processes, one process per connection:

When you add connection pooling, fewer backend connections are required. This frees up server resources for other tasks, such as disk caching:

Connection pooling allows you to handle up to 5000 database client connections simultaneously. You can calculate how many connections you can handle by the number of CPU cores you have available. You should have at least one connection per core, but make sure you are not overloading each core. A good number of connections to aim for is three to five times the available CPU cores, depending on your workload.

Connection pooling modes

There are several different pool modes:

• Transaction (default)
• Session
• Statement

Transaction pooling mode

This is the default pooling mode. It allows each client connection to take turns using a backend connection during a single transaction. When the transaction is committed, the backend connection is returned back into the pool and the next waiting client connection reuses the same connection immediately. This provides quick response times for queries as long as the most transactions are performed quickly. This is the most commonly used mode.

Session pooling mode

This mode holds a client connection until the client disconnects. When the client disconnects, the server connection is returned back into the connection pool free connection list, to wait for the next client connection. Client connections are accepted at TCP level, but their queries only proceed when another client disconnects and frees up the backend connection back into the pool. This mode is useful when you require a wait queue for incoming connections, while keeping the server memory usage low. However, it is not useful in most common scenarios because the backend connections are recycled very slowly.

Statement pooling mode

This mode is similar to the transaction pool mode, except that instead of allowing a full transaction to be run, it cycles the server side connections after each and every database statement (SELECT, INSERT, UPDATE, DELETE, for example). Transactions containing multiple SQL statements are not allowed in this mode. This mode is best suited to specialized workloads that use sharding front-end proxies.

Set up a connection pool

You can set up a connection pool from the MST Console. Make sure you have already created a service that you want to add connection pooling to.

Setting up a connection pool

1. In MST Console, navigate to the Services list, and click the name of the service you want to add connection pooling to.
2. In the Service overview page, navigate to the Pools tab. When you have created some pools, they are shown here.
3. Click Add Pool to create a new pool.
4. In the Create New Connection Pool dialog, use these settings:
   • In the Pool name field, type a name for your new pool. This name becomes the database dbname connection parameter for your pooled client connectons.
   • In the Database field, select a database to connect to. Each pool can only connect to one database.
   • In the Pool Mode field, select which pool mode to use.
   • In the Pool Size field, select the maximum number of server connections this pool can use at any one time.
   • In the Username field, select which database username to connect to the database with.
5. Click Create to create the pool, and see the details of the new pool in the list. You can click Info next to the pool details to see more information, including the URI and port details.

Pooled servers use a different port number than regular servers. This allows you to use both pooled and un-pooled connections at the same time.

===== PAGE: https://docs.tigerdata.com/mst/viewing-service-logs/ =====

Viewing service logs

Occasionally there is a need to inspect logs from Managed Service for TimescaleDB. For example, to debug query performance or inspecting errors caused by a specific workload.

There are different built-in ways to inspect service logs at Managed Service for TimescaleDB:

• When you select a specific service, navigate to the Logs tab to see recent events. Logs can be browsed back in time.

• Download logs using the command-line client by running:

  avn service logs -S desc -f --project <PROJECT_NAME> <SERVICE_NAME>
  

• REST API endpoint is available for fetching the same information two above methods output, in case programmatic access is needed.

Service logs included on the normal service price are stored only for a few days. Unless you are using logs integration to another service, older logs are not accessible.

===== PAGE: https://docs.tigerdata.com/mst/vpc-peering/ =====

VPC peering

Virtual Private Cloud (VPC) peering is a method of connecting separate Cloud private networks to each other. It makes it possible for the virtual machines in the different VPCs to talk to each other directly without going through the public internet. VPC peering is limited to VPCs that share the same Cloud provider.

VPC peering setup is a per project and per region setting. This means that all services created and running utilize the same VPC peering connection. If needed, you can have multiple projects that peer with different connections.

services are only accessible using your VPC's internal network. They are not accessible from the public internet. TLS certificates for VPC peered services are signed by the MST project CA and cannot be validated against a public CA (Let's Encrypt). You can choose whether you want to run on a VPC peered network or on the public internet for every service.

You can set up VPC peering on:

• [Amazon Web Services (AWS)] vpc-aws
• [Google Cloud Platform (GCP)] vpc-gcp
• [Microsoft Azure] vpc-azure

===== PAGE: https://docs.tigerdata.com/mst/integrations/ =====

Integrations for Managed Service for TimescaleDB

Managed Service for TimescaleDB integrates with the other tools you are already using. You can combine your services with third-party tools and build a complete cloud data platform.

You can integrate Managed Service for TimescaleDB with:

• Grafana
• Loggly
• Datadog
• Prometheus
• Syslog
• External Elasticsearch
• External OpenSearch

===== PAGE: https://docs.tigerdata.com/mst/extensions/ =====

Supported Postgres extensions in Managed Service for TimescaleDB

Managed Service for TimescaleDB supports many Postgres extensions. See available extensions for a full list.

Add an extension

You can add a supported extension to your database from the command line.

Some extensions have dependencies. When adding these, make sure to create them in the proper order.

Some extensions require disconnecting and reconnecting the client connection before they are fully available.

Adding an extension

1. Connect to your database as the tsdbadmin user.
2. Run CREATE EXTENSION IF NOT EXISTS <extension_name>.

Available extensions

These extensions are available on Managed Service for TimescaleDB:

• address_standardizer
• address_standardizer_data_us
• aiven_extras
• amcheck
• anon
• autoinc
• bloom
• bool_plperl
• btree_gin
• btree_gist
• citext
• cube
• dblink
• dict_int
• dict_xsyn
• earthdistance
• file_fdw
• fuzzystrmatch
• h3
• h3_postgis
• hll
• hstore
• hstore_plperl
• insert_username
• intagg
• intarray
• isn
• jsonb_plperl
• lo
• ltree
• moddatetime
• pageinspect
• pg_buffercache
• pg_cron
• pg_freespacemap
• pg_prewarm
• pg_repack
• pg_similarity
• pg_stat_monitor
• pg_stat_statements
• pg_surgery
• pg_trgm
• pg_visibility
• pg_walinspect
• pgaudit
• pgcrypto
• pgrouting
• pgrowlocks
• pgstattuple
• plperl
• plpgsql
• postgis
• postgis_raster
• postgis_sfcgal
• postgis_tiger_geocoder
• postgis_topology
• postgres_fdw
• refint
• rum
• seg
• sslinfo
• tablefunc
• tcn
• timescaledb
• tsm_system_rows
• tsm_system_time
• unaccent
• unit
• uuid-ossp
• vector
• vectorscale
• xml2
• timescaledb_toolkit

The postgis_legacy extension is not packaged or supported as an extension by the PostGIS project. Tiger Data provides the extension package for Managed Service for TimescaleDB.

Request an extension

You can request an extension not on the list by contacting Support. In your request, specify the database service and user database where you want to use the extension.

Untrusted language extensions are not supported. This restriction preserves our ability to offer the highest possible service level. An example of an untrusted language extension is plpythonu.

You can contact Support directly from Managed Service for TimescaleDB. Click the life-preserver icon in the upper-right corner of your dashboard.

===== PAGE: https://docs.tigerdata.com/mst/dblink-extension/ =====

Using the dblink extension in Managed Service for TimescaleDB

The dblink Postgres extension allows you to connect to other Postgres databases and to run arbitrary queries.

You can use foreign data wrappers (FDWs) to define a remote foreign server to access its data. The database connection details such as hostnames are kept in a single place, and you only need to create a user mapping to store remote connections credentials.

Prerequisites

Before you begin, sign in to your service, navigate to the Overview tab, and take a note of these parameters for the Postgres remote server. Alternatively, you can use the avn service get command in the Aiven client:

• HOSTNAME: The remote database hostname
• PORT: The remote database port
• USER: The remote database user to connect. The default user is tsdbadmin.
• PASSWORD: The remote database password for the USER
• DATABASE_NAME: The remote database name. The default database name is defaultdb.

Enable the dblink extension

To enable the dblink extension on an MST Postgres service:

1. Connect to the database as the tsdbadmin user:

   psql -x "postgres://tsdbadmin:<PASSWORD>@<HOSTNAME>:<PORT>/defaultdb?sslmode=require"
   

2. Create the dblink extension

   CREATE EXTENSION dblink;
   

3. Create a table named inventory:

 CREATE TABLE inventory (id int);

1. Insert data into the inventory table:

 INSERT INTO inventory (id) VALUES (100), (200), (300);

Create a foreign data wrapper using dblink_fdw

1. Create a user user1 who can access the dblink

 CREATE USER user1 PASSWORD 'secret1'

1. Create a remote server definition named mst_remote, using dblink_fdw and the connection details of the service.

   
   CREATE SERVER mst_remote
       FOREIGN DATA WRAPPER dblink_fdw
       OPTIONS (
                host 'HOST',
                dbname 'DATABASE_NAME',
                port 'PORT'
                );
   

2. Create a user mapping for the user1 to automatically authenticate as the tsdbadmin when using the dblink:

   
       CREATE USER MAPPING FOR user1
          SERVER mst_remote
          OPTIONS (
           user 'tsdbadmin',
           password 'PASSWORD'
           );
   

3. Enable user1 to use the remote Postgres connection mst_remote:

 GRANT USAGE ON FOREIGN SERVER mst_remote TO user1;

Query data using a foreign data wrapper

In this example in the user1 user queries the remote table inventory defined in the target Postgres database from the mst_remote server definition:

Quering data using a foreign data wrapper

To query a foreign data wrapper, you must be a database user with the necessary permissions on the remote server.

1. Connect to the service as user1 with necessary grants to the remote server.

2. Establish the dblink connection to the remote target server:

 SELECT dblink_connect('my_new_conn', 'mst_remote');

1. Query using the foreign server definition as parameter:

 SELECT * FROM dblink('my_new_conn','SELECT * FROM inventory') AS t(a int);

Output is similar to:

    a
  -----
   100
   200
   300
 (3 rows)

===== PAGE: https://docs.tigerdata.com/mst/security/ =====

Security overview

This section covers how Managed Service for TimescaleDB handles security of your data while it is stored.

Cloud provider accounts

services are hosted by cloud provider accounts controlled by Tiger Data. These accounts are managed only by Tiger Data and Aiven operations personnel. Members of the public cannot directly access the cloud provider account resources.

Virtual machines

Your services are located on one or more virtual machines. Each virtual machine is dedicated to a single customer, and is never multi-tenanted. Customer data never leaves the virtual machine, except when uploaded to an offsite backup location.

When you create a new service, you need to select a cloud region. When the virtual machine is launched, it does so in the cloud region you have chosen. Your data never leaves the chosen cloud region.

If a cloud region has multiple Availability Zones, or a similar high-availability mechanism, the virtual machines are distributed evenly across the zones. This provides the best possible service if an Availability Zone becomes unavailable.

Access to the virtual machine providing your service is restricted. Software that is accessing your database needs to run on a different virtual machine. To reduce latency, it is best for it to be using a virtual machine provided by the same cloud provider, and in the same region, if possible.

Virtual machines are not reused. They are terminated and wiped when you upgrade or delete your service.

Project security

Every Managed Service for TimescaleDB project has its own certificate authority. This certificate authority is used to sign certificates used internally by your services to communicate between different cluster nodes and to management systems.

You can download your project certificate authority in MST Console. In the Services tab, click the service you want to find the certificate for. In the service Overview tab, under Connection information, locate the CA Certificate section, and click Show to see the certificate. It is recommended that you set up your browser or client to trust that certificate.

All server certificates are signed by the project certificate authority OF MST Console.

Data encryption

Managed Service for TimescaleDB at-rest data encryption covers both active service instances as well as service backups in cloud object storage.

Service instances and the underlying virtual machines use full volume encryption. The encryption method uses LUKS, with a randomly generated ephemeral key per each instance, and per volume. The keys are never re-used, and are disposed of when the instance is destroyed. This means that a natural key rotation occurs with roll-forward upgrades. By default, the LUKS mode is aes-xts-plain64:sha256, with a 512-bit key.

Backups are encrypted with a randomly generated key per file. These keys are in turn encrypted with an RSA key-encryption key-pair, and stored in the header section of each backup segment. The file encryption is performed with AES-256 in CTR mode, with HMAC-SHA256 for integrity protection. The RSA key-pair is randomly generated for each service. The key lengths are 256-bit for block encryption, 512-bit for the integrity protection, and 3072-bits for the RSA key.

Encrypted backup files are stored in the object storage in the same region that the virtual machines are located for the service.

Networking security

Access to provided services is only provided over TLS encrypted connections. TLS ensures that third-parties can't eavesdrop or modify the data while it's in transit between your service and the clients accessing your service. You cannot use unencrypted plain text connections.

Communication between virtual machines within Managed Service for TimescaleDB is secured with either TLS or IPsec. You cannot use unencrypted plaintext connections.

Virtual machines network interfaces are protected by a dynamically configured firewall based on iptables, which only allows connections from specific addresses. This is used for network traffic from the internal network to other VMs in the same service, and for external public network, to client connections.

By default, new services accept incoming traffic from all sources, which is used to simplify initial set up of your service. It is highly recommended that you restrict the IP addresses that are allowed to establish connections to your services.

Configure allowed incoming IP addresses for your service

1. In MST Console, select the service to update.

2. In Overview check the Port number.

   This is the port that you are managing inbound access for.

3. In Network, check IP filters. The default value is `Open for all.

4. Click the ellipsis (...) to the right of Network, then select Set public IP filters.

5. Set the Allowed inbound IP addresses:

   

Networking with VPC peering

When you set up VPC peering, you cannot access your services using public internet-based access. Service addresses are published in the public DNS record, but they can only be connected to from your peered VPC network using private network addresses.

The virtual machines providing your service are hosted by cloud provider accounts controlled by Tiger Data.

Customer data privacy

Customer data privacy is of utmost importance at Tiger Data. Tiger Data works with Aiven to provide Managed Service for TimescaleDB.

In most cases, all the resources required for providing your services are automatically created, maintained, and terminated by the Managed Service for TimescaleDB infrastructure, with no manual operator intervention required.

The Tiger Data Operations Team are able to securely log in to your service Virtual Machines, for the purposes of troubleshooting, as required. Tiger Data operators never access customer data unless you explicitly request them to do so, to troubleshoot a technical issue. This access is logged and audited.

There is no ability for any customer or member of the public to access any virtual machines used in Managed Service for TimescaleDB.

Managed Service for TimescaleDB services are periodically assessed and penetration tested for any security issues by an independent professional cyber-security vendor.

Aiven is fully GDPR-compliant, and has executed data processing agreements (DPAs) with relevant cloud infrastructure providers. If you require a DPA, or if you want more information about information security policies, contact Tiger Data.

===== PAGE: https://docs.tigerdata.com/mst/postgresql-read-replica/ =====

Create a read-only replica of Postgres

Postgres read-only replicas allow you to perform read-only queries against the replica and reduce the load on the primary server. You can optimize query response times across different geographical locations because the replica can be created in different regions or on different cloud providers. For information about creating a read-only replica using the Aiven client, see the documentation on creating a read replica using the CLI.

If you are running a Managed Service for TimescaleDB Pro plan, you have standby nodes available in a high availability setup. The standby nodes support read-only queries to reduce the effect of slow queries on the primary node.

Creating a replica of Postgres

1. In MST Console, click the service you want to create a remote replica for.

2. In Overview, click Create a read replica.

3. In Create a PostgreSQL read replica, type a name for the remote replica, select the cloud provider, location, plan that you want to use, and click Create.

When the read-only replica is created it is listed as a service in your project. The Overview tab of the replica also lists the name of the primary service for the replica. To promote a read-only replica as a master database, click the Promote to master button.

Using read-only replica for the service on MST

1. In the Overview page of the read-only replica for the service on MST, copy the Service URI.

2. At the psql prompt, connect to the read-only service:

   psql <SERVICE_URI>
   

3. To check whether you are connected to a primary or replica node:

   SELECT * FROM pg_is_in_recovery();
   

   If the output is TRUE you are connected to the replica, and if the output is FALSE you are connected to the primary server.

Managed Service for TimescaleDB uses asynchronous replication, so some lag is expected. When you run an INSERT operation on the primary node, a small delay of less than a second is expected for the change to propagate to the replica.

===== PAGE: https://docs.tigerdata.com/mst/maintenance/ =====

Maintenance

On Managed Service for TimescaleDB, software updates are handled automatically, and you do not need to perform any actions to keep up to date.

Non-critical software updates are applied during a maintenance window that you can define to suit your workload. If a security vulnerability is found that affects you, maintenance might be performed outside of your scheduled maintenance window.

After maintenance updates have been applied, if a new version of the TimescaleDB binary has been installed, you need to update the extension to use the new version. To do this, use this command:

ALTER EXTENSION timescaledb UPDATE;

After a maintenance update, the DNS name remains the same, but the IP address it points to changes.

Non-critical maintenance updates

Non-critical upgrades are made available before the upgrade is performed automatically. During this time you can click Apply upgrades to start the upgrade at any time. However, after the time expires, usually around a week, the upgrade is triggered automatically in the next available maintenance window for your service. You can configure the maintenance window so that these upgrades are started only at a particular time, on a set day of the week. If there are no pending upgrades available during a regular maintenance window, no changes are performed.

When you are considering your maintenance window schedule, you might prefer to choose a day and time that usually has very low activity, such as during the early hours of the morning, or over the weekend. This can help minimize the impact of a short service interruption. Alternatively, you might prefer to have your maintenance window occur during office hours, so that you can monitor your system during the upgrade.

Adjusting your maintenance window

1. In MST Console, click the service that you want to manage the maintenance window for.
2. Click the ellipses (...) to the right of Maintenance, then click Change maintenence window.
3. In the Service Maintenance Window dialog, select the day of the week and the time (in Universal Coordinated Time) you want the maintenance window to start. Maintenance windows can run for up to four hours.
4. Click Save Changes.

Critical updates

Critical upgrades and security fixes are installed outside normal maintenance windows when necessary, and sometimes require a short outage.

Upgrades are performed as rolling upgrades where completely new server instances are built alongside the old ones. When the new instances are up and running they are synchronized with the old servers, and a controlled automatic failover is performed to switch the service to the new upgraded servers. The old servers are retired automatically after the new servers have taken over. The controlled failover is a very quick and safe operation and it takes less than a minute to get clients connected again. In most cases, there is five to ten second outage during this process.

===== PAGE: https://docs.tigerdata.com/mst/failover/ =====

Failover

One standby read-only replica server is configured, for each service on a Pro plan. You can query a read-only replica server, but cannot write to a read-only replica server. When a master server fails, the standby replica server is automatically promoted as master. If you manually created a read-only replica service, then if a master server fails, the read-only replica services are not promoted as master servers.

The two distinct cases during which failovers occur are:

• When the master or replica fails unexpectedly, for example because the hardware hosting the virtual machine fails.
• When controlled failover happens because of upgrades.

Uncontrolled master or replica fail

When a replica server fails unexpectedly, there is no way to know whether the server really failed, or whether there is a temporary network glitch with the cloud provider's network.

There is a 300 second timeout before Managed Service for TimescaleDB automatically decides the server is gone and spins up a new replica server. During these 300 seconds, replica.servicename.timescaledb.io points to a server that may not serve queries anymore. The DNS record pointing to the master server servicename.timescaledb.io continues to serve the queries. If the replica server does not come back up within 300 seconds, replica.servicename.timescaledb.io points to the master server, until a new replica server is built.

When the master server fails, a replica server waits for 60 seconds before promoting itself as master. During this 60-second timeout, the master server servicename.timescaledb.io remains unavailable and does not respond. However, replica.servicename.timescaledb.io works in read-only mode. After the replica server promotes itself as master, servicename.timescaledb.io points to the new master server, and replica.servicename.timescaledb.io continues to point to the new master server. A new replica server is built automatically, and after it is in sync, replica.servicename.timescaledb.io points to the new replica server.

Controlled failover during upgrades

When applying upgrades or plan changes on business or premium plans, the standby server is replaced:

A new server is started, the backup is restored, and the new server starts following the old master server. After the new server is up and running, replica.servicename.timescaledb.io is updated, and the old replica server is deleted.

For premium plans, this step is executed for both replica servers before the master server is replaced. Two new servers are started, a backup is restored, and one new server is synced up to the old master server. When it is time to switch the master to a new server, the old master is terminated and one of the new replica servers is immediately promoted as a master. At this point, servicename.timescaledb.io is updated to point at the new master server. Similarly, the new master is removed from the replica.servicename.timescaledb.io record.

===== PAGE: https://docs.tigerdata.com/mst/manage-backups/ =====

Back up and restore your Managed Service for TimescaleDB

services are automatically backed up, with full backups daily, and write-ahead log (WAL) continuously recorded. All backups are encrypted.

Managed Service for TimescaleDB uses pghoard, a Postgres backup daemon and restore tool, to store backup data in cloud object stores. The number of backups stored and the retention time of the backup depend on the service plan.

The size of logical backups can be different from the size of the Managed Service for TimescaleDB backup that appears on the web console. In some cases, the difference is significant. Backup sizes that appear in the MST Console are for daily backups, before encryption and compression. To view the size of each database, including space consumed by indexes, you can use the \l+ command at the psql prompt.

Logical and binary backups

The two types of backups are binary backups and logical backups. Full backups are version-specific binary backups which, when combined with WAL, allow consistent recovery to a point in time (PITR). You can create a logical backup with the pg_dump command.

This table lists the differences between binary and logical backups when backing up indexes, transactions, and data:

Type	Binary	Logical
index	contains all data from indexes	does not contain index data, it contains only queries used to recreate indexes from other data
transactions	contains uncommitted transactions	does not contain uncommitted transactions
data	contains deleted and updated rows which have not been cleaned up by Postgres VACUUM process, and all databases, including templates	does not contain any data already deleted, and depending on the options given, the output might be compressed

Restore a service

Managed Service for TimescaleDB provides a point-in-time recovery (PITR). To restore your service from a backup, click the Restore button in the Backups tab for your service. The backups are taken automatically by Managed Service for TimescaleDB and retained for a few days depending on your plan type.

Plan type	Backup retention period
Dev	1 day
Basic	2 days
Pro	3 days

Manually creating a backup

You can use pg_dump to create a backup manually. The pg_dump command allows you to create backups that can be directly restored elsewhere if required.

Typical parameters for the command pg_dump include:

pg_dump '<SERVICE_URL_FROM_PORTAL>' -f '<TARGET_FILE/DIR>' -j '<NUMBER_OF_JOBS>' -F '<BACKUP_FORMAT>'

The pg_dump command can also be run against one of the standby nodes. For example, use this command to create a backup in directory format using two concurrent jobs. The results are stored to a directory named backup:

pg_dump 'postgres://tsdbadmin:password@mypg-myproject.a.timescaledb.io:26882/defaultdb?sslmode=require' -f backup -j 2 -F directory

You can put all backup files to single tar file and upload to Amazon S3. For example:

export BACKUP_NAME=backup-date -I.tartar -cf $BACKUP_NAME backup/s3cmd put $BACKUP_NAME s3://pg-backups/$BACKUP_NAME

===== PAGE: https://docs.tigerdata.com/mst/aiven-client/ =====

Aiven Client for Managed Service for TimescaleDB

You can use Aiven Client to manage your services in Managed Service for TimescaleDB.

You can use the Aiven Client tool to:

• Connect to Managed Service for TimescaleDB
• Create a service
• Create a fork
• Add authentication plugins to your attached Grafana service

Instructions:

• Install and configure the Aiven client
• Fork services with Aiven client
• Configure Grafana authentication plugins
• Send Grafana emails
• Create a read-only replica with the Aiven client

Install and configure the Aiven client

Aiven Client is a command line tool for fully managed services. To use Aiven Client, you first need to create an authentication token. Then, you configure the client to connect to your Managed Service for TimescaleDB using the command line.

Create an authentication token in Managed Service for TimescaleDB

To connect to Managed Service for TimescaleDB using Aiven Client, create an authentication token.

1. In Managed Service for TimescaleDB, click User Information in the top right corner.
2. In the User Profile page, navigate to the Authenticationtab.
3. Click Generate Token.
4. In the Generate access token dialog, type a descriptive name for the token. Leave the rest of the fields blank.
5. Copy the generated authentication token and save it.

Install the Aiven Client

The Aiven Client is provided as a Python package. If you've already installed Python, you can install the client on Linux, MacOS, or Windows systems using pip:

pip install aiven-client

For more information about installing the Aiven Client, see the Aiven documentation.

Configure Aiven Client to connect to Managed Service for TimescaleDB

To access Managed Service for TimescaleDB with the Aiven Client, you need an authentication token. Aiven Client uses this to access your services on Managed Service for TimescaleDB.

Configuring Aiven Client to connect to Managed Service for TimescaleDB

1. Change to the install directory that contains the configuration files:

   cd ~/.config/aiven/
   

2. Open the aiven-credentials.json using any editor and update these lines with your Managed Service for TimescaleDB User email, and the authentication token that you generated:

   {
     "auth_token": "ABC1+123...TOKEN==",
     "user_email": "your.email@timescale.com"
   }
   

3. Save the aiven-credentials.json file.

4. To verify that you can access your services on Managed Service for TimescaleDB, type:

   avn project list
   

   This command shows a list of all your projects:

    PROJECT_NAME       DEFAULT_CLOUD            CREDIT_CARD
    =============     =======================   ===================
    project-xxxx      timescale-aws-us-east-1   xxxx-xxxx-xxxx-xxxx
    project-yyyy      timescale-aws-us-east-1   xxxx-xxxx-xxxx-xxxx
    project-zzzz      timescale-aws-us-east-1   xxxx-xxxx-xxxx-xxxx
   

Fork services with Aiven client

When you a fork a service, you create an exact copy of the service, including the underlying database. You can use a fork of your service to:

• Create a development copy of your production environment.
• Set up a snapshot to analyze an issue or test an upgrade.
• Create an instance in a different cloud, geographical location, or under a different plan.

For more information about projects, plans, and other details about services, see About Managed Service for TimescaleDB.

Creating a fork of your service

1. In the Aiven client, connect to your service.

2. Switch to the project that contains the service you want to fork:

    avn project switch <PROJECT>
   

3. List the services in the project, and make a note of the service that you want to fork, listed under SERVICE_NAME column in the output.

    avn service list
   

4. Get the details of the service that you want to fork:

   avn service get <SERVICE_NAME>
   

5. Create the fork:

   avn service create <NAME_OF_FORK> --project <PROJECT_ID>\
   -t <SERVICE_TYPE> --plan <PLAN> --cloud <CLOUD_NAME>\
   -c service_to_fork_from=<NAME_OF_SERVICE_TO_FORK>
   

Example

To create a fork named grafana-fork for a service named grafana with these parameters:

• PROJECT_ID: project-fork
• CLOUD_NAME: timescale-aws-us-east-1
• PLAN_TYPE: dashboard-1

   avn service create grafana-fork --project project-fork -t grafana --plan dashboard-1 --cloud timescale-aws-us-east-1  -c service_to_fork_from=grafana

You can switch to project-fork and view the newly created grafana-fork using:

   avn service list

Configure Grafana authentication plugins

Grafana supports multiple authentication plugins, in addition to built-in username and password authentication.

On Managed Service for TimescaleDB, Grafana supports Google, GitHub, and GitLab authentication. You can configure authentication integration using the Aiven command-line client.

Integrating the Google authentication plugin

To integrate Google authentication with Grafana service on Managed Service for TimescaleDB, you need to create your Google OAuth keys. Copy your client ID and client secret to a secure location.

How to integrate the Google authentication plugin

1. In the Aiven Client, connect to your service.

2. Switch to the project that contains the Grafana service you want to integrate:

    avn switch <PROJECT>
   

3. List the services in the project. Make a note of the Grafana service that you want to integrate, listed under SERVICE_NAME column in the output.

    avn service list
   

4. Get the details of the service that you want to integrate:

   avn service get <SERVICE_NAME>
   

5. Integrate the plugin with your services using the <CLIENT_ID> and <CLIENT_SECRET> from your Google developer console:

   avn service update -c auth_google.allowed_domains=<G-SUITE_DOMAIN>\
   -c auth_google.client_id=<CLIENT_ID>\
   -c auth_google.client_secret=<CLIENT_SECRET><SERVICE_NAME>
   

6. Log in to Grafana with your service credentials.

7. Navigate to Configuration → Plugins and verify that the Google OAuth application is listed as a plugin.

When you allow sign-ups using the -c auth_google.allow_sign_up=true option, by default each new user is created with viewer permissions and added to their own newly created organizations. To specify different permissions, use -c user_auto_assign_org_role=ROLE_NAME. To add all new users to the main organization, use the -c user_auto_assign_org=true option.

Integrating the GitHub authentication plugin

To integrate GitHub authentication with Grafana service on Managed Service for TimescaleDB, you need to create your GitHub OAuth application. Store your client ID and client secret in a secure location.

How to integrate the GitHub authentication plugin

1. In the Aiven Client, connect to your service.

2. Switch to the project that contains the Grafana service you want to integrate:

    avn switch <PROJECT>
   

3. List the services in the project, and make a note of the Grafana service that you want to integrate, listed under SERVICE_NAME column in the output.

    avn service list
   

4. Get the details of the service that you want to integrate:

   avn service get <SERVICE_NAME>
   

5. Integrate the plugin with your service using the <CLIENT_ID>, and <CLIENT_SECRET> from your GitHub OAuth application:

   avn service update -c auth_github.client_id=<CLIENT_ID>\
   -c auth_github.client_secret=<CLIENT_SECRET> <SERVICE_NAME>
   
   

6. Log in to Grafana with your service credentials.

7. Navigate to Configuration → Plugins. The Plugins page lists GitHub OAuth application for the Grafana instance.

When you allow sign-ups using the -c auth_github.allow_sign_up=true option, by default each new user is created with viewerpermission and added to their own newly created organizations. To specify different permissions, use -c user_auto_assign_org_role=ROLE_NAME. To add all new users to the main organization, use the -c user_auto_assign_org=true option.

Integrating the GitLab authentication plugin

To integrate the GitLab authentication with Grafana service on Managed Service for TimescaleDB, you need to create your GitLab OAuth application. Copy your client ID, client secret, and GitLab groups name to a secure location.

If you use your own instance of GitLab instead of gitlab.com, then you need to set the following:

• auth_gitlab.api_url
• auth_github.auth_url
• auth_github.token_url

How to integrate the GitLab authentication plugin

1. In the Aiven Client, connect to your MST_SERVICE_LONG.

2. Switch to the project that contains the Grafana service you want to integrate:

    avn project switch <PROJECT>
   

3. List the services in the project. Note the Grafana service that you want to integrate, listed under SERVICE_NAME column in the output.

    avn service list
   

4. Get the details of the service that you want to integrate:

   avn service get <SERVICE_NAME>
   

5. Integrate the plugin with your service using the <CLIENT_ID>, <CLIENT_SECRET>, and <GITLAB_GROUPS> from your GitLab OAuth application:

   avn service update -c auth_gitlab.client_id=<CLIENT_ID>\
   -c auth_gitlab.client_secret=<CLIENT_SECRET>\
   -c auth_gitlab.allowed_groups=<GITLAB_GROUPS> <SERVICE_NAME>
   
   

6. Log in to Grafana with your service credentials.

7. Navigate to Configuration → Plugins. The Plugins page lists GitLab OAuth application for the Grafana instance.

When you allow sign-ups using the -c auth_gitlab.allow_sign_up=true option, by default each new user is created with viewerpermission and added to their own newly created organizations. To specify different permissions, use -c user_auto_assign_org_role=ROLE_NAME. To add all new users to the main organization, use the -c user_auto_assign_org=true option.

Send Grafana emails

Use the Aiven client to configure the Simple Mail Transfer Protocol (SMTP) server settings and send emails from Managed Service for TimescaleDB for Grafana. This includes invite emails, reset password emails, and alert messages.

Prerequisites

Before you begin, make sure you have:

• (Optional): Made a note of these values in the SMTP server: IP or hostname, SMTP server port, Username, Password, Sender email address, and Sender name.

Configuring the SMTP server for Grafana service

1. In the Aiven client, connect to your service.

2. Switch to the project that contains the Grafana service you want to integrate:

    avn project switch <PROJECT>
   

3. List the services in the project. Note the Grafana service that you want to configure, listed under SERVICE_NAME column in the output.

    avn service list
   

4. Get the details of the service that you want to integrate:

   avn service get <SERVICE_NAME>
   

5. Configure the Grafana service using the SMTP values:

      avn service update --project <PROJECT> <SERVICE_NAME>\
      -c smtp_server.host=smtp.example.com \
      -c smtp_server.port=465 \
      -c smtp_server.username=emailsenderuser \
      -c smtp_server.password=emailsenderpass \
      -c smtp_server.from_address="grafana@yourcompany.com"
   

6. Review all available custom options, and configure:

      avn service types -v
   

You can now send emails for your Grafana service on MST.

Create a read-only replica with Aiven client

Read-only replicas enable you to perform read-only queries against the replica and reduce the load on the primary server. They are also a good way to optimize query response times across different geographical locations. You can achieve this by placing the replicas in different regions or even different cloud providers.

Creating a read-only replica of your service

1. In the Aiven client, connect to your service.

2. Switch to the project that contains the service you want to create a read-only replica for:

   avn project switch <PROJECT>
   

3. List the services in the project. Note the service for which you will create a read-only replica. You can find it listed under the SERVICE_NAME column in the output:

   avn service list
   

4. Get the details of the service that you want to fork:

   avn service get <SERVICE_NAME>
   

5. Create a read-only replica:

   avn service create <NAME_OF_REPLICA> --project <PROJECT_ID>\
   -t pg --plan <PLAN_TYPE> --cloud timescale-aws-us-east-1\
   -c pg_read_replica=true\
   -c service_to_fork_from=<NAME_OF_SERVICE_TO_FORK>\
   -c pg_version=11 -c variant=timescale
   

Example

To create a fork named replica-fork for a service named timescaledb with these parameters:

• PROJECT_ID: fork-project
• CLOUD_NAME: timescale-aws-us-east-1
• PLAN_TYPE: timescale-basic-100-compute-optimized

avn service create replica-fork --project fork-project\
-t pg --plan timescale-basic-100-compute-optimized\
--cloud timescale-aws-us-east-1 -c pg_read_replica=true\
-c service_to_fork_from=timescaledb -c\
pg_version=11 -c variant=timescale

You can switch to project-fork and view the newly created replica-fork using:

avn service list

===== PAGE: https://docs.tigerdata.com/mst/migrate-to-mst/ =====

Migrate from self-hosted TimescaleDB to Managed Service for TimescaleDB

You can migrate your data from self-hosted TimescaleDB to Managed Service for TimescaleDB and automate most of the common operational tasks.

Each service has a database named defaultdb, and a default user account named tsdbadmin. You use MST Console to create additional users and databases using the Users and Databases tabs.

You can switch between different plan sizes in Managed Service for TimescaleDB. However, during the migration process, choose a plan size that has the same storage size or slightly larger than the currently allocated plan. This allows you to limit the downtime during the migration process and have sufficient compute and storage resources.

Depending on your database size and network speed, migration can take a very long time. During this time, any new writes that happen during the migration process are not included. To prevent data loss, turn off all the writes to the source self-hosted TimescaleDB database before you start migration.

Before migrating for production, do a cold run without turning off writes to the source self-hosted TimescaleDB database. This gives you an estimate of the time the migration process takes, and helps you to practice migrating without causing downtime to your customers.

If you prefer the features of Tiger Cloud, you can easily migrate your data from an service to a Tiger Cloud service.

Prerequisites

Before you migrate your data, do the following:

• Set up the migration machine:

  You run the migration commands on the migration machine. It must have enough disk space to hold the dump file.

  • Install the Postgres pg_dump and pg_restore utilities on a migration machine.

  • Install a client to connect to self-hosted TimescaleDB and Managed Service for TimescaleDB.

    These instructions use psql, but any client works.

• Create a target service:

  For more information, see the Install Managed Service for TimescaleDB. Provision your target service with enough space for all your data.

• On the source self-hosted TimescaleDB and the target service, ensure that you are running:

  • The same major version of Postgres.

    For information, see upgrade Postgres.

  • The same major version of TimescaleDB

    For more information, see Upgrade TimescaleDB to a major version.

Migrate your data to a service

To move your data from self-hosted TimescaleDB instance to a service, run the following commands from your migration machine:

1. Take offline the applications that connect to the source self-hosted TimescaleDB instance

   The duration of migration is proportional to the amount of data stored in your database. By disconnecting your app from your database, you avoid possible data loss.

2. Set your connection strings

   These variables hold the connection information for the source self-hosted TimescaleDB instance and the target service:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   export TARGET="postgres://tsdbadmin:<password>@<host>:<port>/defaultdb?sslmode=require"
   

3. Dump the data from your source Tiger Cloud service

   pg_dump -d "source" --no-owner -Fc -v -f dump.bak
   

4. Put your target service in the right state for restoring

   psql -d "target" -c "SELECT timescaledb_pre_restore();"
   

5. Upload your data to the target service

   pg_restore -d "target" --jobs 4 -Fc dump.bak
   

   The --jobs option specifies the number of CPUs to use to dump and restore the database concurrently.

6. Return your target service to normal operations

   psql -d "target" -c "SELECT timescaledb_post_restore();"
   

7. Connect to your new database and update your table statistics by running [ANALYZE] analyze on your entire dataset:

   psql -d "target" defaultdb=> ANALYZE;
   

To migrate from multiple databases, you repeat this migration procedure one database after another.

Troubleshooting

If you see the following errors during migration, you can safely ignore them. The migration still runs successfully.

• For pg_dump:

  pg_dump: warning: there are circular foreign-key constraints on this table:
  pg_dump: hypertable
  pg_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
  pg_dump: Consider using a full dump instead of a --data-only dump to avoid this problem.
  pg_dump: NOTICE:  hypertable data are in the chunks, no data will be copied
  DETAIL:  Data for hypertables are stored in the chunks of a hypertable so COPY TO of a hypertable will not copy any data.
  HINT:  Use "COPY (SELECT * FROM <hypertable>) TO ..." to copy all data in hypertable, or copy each chunk individually.
  

• For pg_restore:

  pg_restore: while PROCESSING TOC:
  pg_restore: from TOC entry 4142; 0 0 COMMENT EXTENSION timescaledb
  pg_restore: error: could not execute query: ERROR:  must be owner of extension timescaledb
  Command was: COMMENT ON EXTENSION timescaledb IS 'Enables scalable inserts and complex queries for time-series data';
  
  

===== PAGE: https://docs.tigerdata.com/mst/restapi/ =====

# Using REST API in Managed Service for TimescaleDB

Managed Service for TimescaleDB has an API for integration and automation tasks.
For information about using the endpoints, see the [API Documentation][aiven-api].
MST offers an HTTP API with token authentication and JSON-formatted data. You
can use the API for all the tasks that can be performed using the MST Console.
To get started you need to first create an authentication token, and then use
the token in the header to use the API endpoints.

1.  In [Managed Service for TimescaleDB][mst-login], click `User Information` in the top right corner.
1.  In the `User Profile` page, navigate to the `Authentication`tab.
1.  Click `Generate Token`.
1.  In the `Generate access token` dialog, type a descriptive name for the
   token and leave the rest of the fields blank.
1.  Copy the generated authentication token and save it.

### Using cURL to get your details

1.  Set the environment variable `MST_API_TOKEN` with the access token that you generate:

   ```bash
   export MST_API_TOKEN="access token"
   ```

1.  To get the details about the current user session using the `/me` endpoint:

   ```bash
   curl -s -H "Authorization: aivenv1 $MST_API_TOKEN" https://api.aiven.io/v1/me|json_pp
   ```

   The output looks similar to this:

   ```bash
   {
       "user": {
           "auth": [],
           "create_time": "string",
           "features": { },
           "intercom": {},
           "invitations": [],
           "project_membership": {},
           "project_memberships": {},
           "projects": [],
           "real_name": "string",
           "state": "string",
           "token_validity_begin": "string",
           "user": "string",
           "user_id": "string"
       }
   }
   ```


===== PAGE: https://docs.tigerdata.com/mst/identify-index-issues/ =====

# Identify and resolve issues with indexes in Managed Service for TimescaleDB

Postgres indexes can be corrupted for a variety of reasons, including
software bugs, hardware failures, or unexpected duplicated data. `REINDEX` allows
you to rebuild the index in such situations.

## Rebuild non-unique indexes

You can rebuild corrupted indexes that do not have `UNIQUE` in their definition.
You can run the `REINDEX` command for all indexes of a table (`REINDEX TABLE`),
and for all indexes in the entire database (`REINDEX DATABASE`).
For more information on the `REINDEX` command, see the [Postgres documentation][postgres-docs].

This command creates a new index that replaces the old one:

```sql
REINDEX INDEX <index-name>;

When you use REINDEX, the tables are locked and you may not be able to use the database, until the operation is complete.

In some cases, you might need to manually build a second index concurrently with the old index, and then remove the old index:

CREATE INDEX CONCURRENTLY test_index_new ON table_a (...);
DROP INDEX CONCURRENTLY test_index_old;
ALTER INDEX test_index_new RENAME TO test_index;

Rebuild unique indexes

A UNIQUE index works on one or more columns where the combination is unique in the table. When the index is corrupted or disabled, duplicated physical rows appear in the table, breaking the uniqueness constraint of the index. When you try to rebuild an index that is not unique, the REINDEX command fails. To resolve this issue, first remove the duplicate rows from the table and then rebuild the index.

Identify conflicting duplicated rows

To identify conflicting duplicate rows, you need to run a query that counts the number of rows for each combination of columns included in the index definition.

For example, this route table has a unique_route_index index defining unique rows based on the combination of the source and destination columns:

CREATE TABLE route(
    source TEXT,
    destination TEXT,
    description TEXT
    );

CREATE UNIQUE INDEX unique_route_index
    ON route (source, destination);

If the unique_route_index is corrupt, you can find duplicated rows in the route table using this query:

SELECT
    source,
    destination,
    count
FROM
    (SELECT
        source,
        destination,
        COUNT(*) AS count
    FROM route
    GROUP BY
        source,
        destination) AS foo
WHERE count > 1;

The query groups the data by the same source and destination fields defined in the index, and filters any entries with more than one occurrence.

Resolve the problematic entries in the rows by manually deleting or merging the entries until no duplicates exist. After all duplicate entries are removed, you can use the REINDEX command to rebuild the index.

===== PAGE: https://docs.tigerdata.com/about/whitepaper/ =====

Tiger Data architecture for real-time analytics

Tiger Data has created a powerful application database for real-time analytics on time-series data. It integrates seamlessly with the Postgres ecosystem and enhances it with automatic time-based partitioning, hybrid row-columnar storage, and vectorized execution—enabling high-ingest performance, sub-second queries, and full SQL support at scale.

Tiger Cloud offers managed database services that provide a stable and reliable environment for your applications. Each service is based on a Postgres database instance and the TimescaleDB extension.

By making use of incrementally updated materialized views and advanced analytical functions, TimescaleDB reduces compute overhead and improves query efficiency. Developers can continue using familiar SQL workflows and tools, while benefiting from a database purpose-built for fast, scalable analytics.

This document outlines the architectural choices and optimizations that power TimescaleDB and Tiger Cloud’s performance and scalability while preserving Postgres’s reliability and transactional guarantees.

Want to read this whitepaper from the comfort of your own computer?

[Tiger Data architecture for real-time analytics (PDF)](https://assets.timescale.com/docs/downloads/tigerdata-whitepaper.pdf)

Introduction

What is real-time analytics?

Real-time analytics enables applications to process and query data as it is generated and as it accumulates, delivering immediate and ongoing insights for decision-making. Unlike traditional analytics, which relies on batch processing and delayed reporting, real-time analytics supports both instant queries on fresh data and fast exploration of historical trends—powering applications with sub-second query performance across vast, continuously growing datasets.

Many modern applications depend on real-time analytics to drive critical functionality:

• IoT monitoring systems track sensor data over time, identifying long-term performance patterns while still surfacing anomalies as they arise. This allows businesses to optimize maintenance schedules, reduce costs, and improve reliability.
• Financial and business intelligence platforms analyze both current and historical data to detect trends, assess risk, and uncover opportunities—from tracking stock performance over a day, week, or year to identifying spending patterns across millions of transactions.
• Interactive customer dashboards empower users to explore live and historical data in a seamless experience—whether it's a SaaS product providing real-time analytics on business operations, a media platform analyzing content engagement, or an e-commerce site surfacing personalized recommendations based on recent and past behavior.

Real-time analytics isn't just about reacting to the latest data, although that is critically important. It's also about delivering fast, interactive, and scalable insights across all your data, enabling better decision-making and richer user experiences. Unlike traditional ad-hoc analytics used by analysts, real-time analytics powers applications—driving dynamic dashboards, automated decisions, and user-facing insights at scale.

To achieve this, real-time analytics systems must meet several key requirements:

• Low-latency queries ensure sub-second response times even under high load, enabling fast insights for dashboards, monitoring, and alerting.
• Low-latency ingest minimizes the lag between when data is created and when it becomes available for analysis, ensuring fresh and accurate insights.
• Data mutability allows for efficient updates, corrections, and backfills, ensuring analytics reflect the most accurate state of the data.
• Concurrency and scalability enable systems to handle high query volumes and growing workloads without degradation in performance.
• Seamless access to both recent and historical data ensures fast queries across time, whether analyzing live, streaming data, or running deep historical queries on days or months of information.
• Query flexibility provides full SQL support, allowing for complex queries with joins, filters, aggregations, and analytical functions.

Tiger Cloud: real-time analytics from Postgres

Tiger Cloud is a high-performance database that brings real-time analytics to applications. It combines fast queries, high ingest performance, and full SQL support—all while ensuring scalability and reliability. Tiger Cloud extends Postgres with the TimescaleDB extension. It enables sub-second queries on vast amounts of incoming data while providing optimizations designed for continuously updating datasets.

Tiger Cloud achieves this through the following optimizations:

• Efficient data partitioning: automatically and transparently partitioning data into chunks, ensuring fast queries, minimal indexing overhead, and seamless scalability
• Row-columnar storage: providing the flexibility of a row store for transactions and the performance of a column store for analytics
• **Optimized query execution: **using techniques like chunk and batch exclusion, columnar storage, and vectorized execution to minimize latency
• Continuous aggregates: precomputing analytical results for fast insights without expensive reprocessing
• **Cloud-native operation: **compute/compute separation, elastic usage-based storage, horizontal scale out, data tiering to object storage
• **Operational simplicity: **offering high availability, connection pooling, and automated backups for reliable and scalable real-time applications

With Tiger Cloud, developers can build low-latency, high-concurrency applications that seamlessly handle streaming data, historical queries, and real-time analytics while leveraging the familiarity and power of Postgres.

Data model

Today's applications demand a database that can handle real-time analytics and transactional queries without sacrificing speed, flexibility, or SQL compatibility (including joins between tables). TimescaleDB achieves this with hypertables, which provide an automatic partitioning engine, and hypercore, a hybrid row-columnar storage engine designed to deliver high-performance queries and efficient compression (up to 95%) within Postgres.

Efficient data partitioning

TimescaleDB provides hypertables, a table abstraction that automatically partitions data into chunks in real time (using time stamps or incrementing IDs) to ensure fast queries and predictable performance as datasets grow. Unlike traditional relational databases that require manual partitioning, hypertables automate all aspects of partition management, keeping locking minimal even under high ingest load.

At ingest time, hypertables ensure that Postgres can deal with a constant stream of data without suffering from table bloat and index degradation by automatically partitioning data across time. Because each chunk is ordered by time and has its own indexes and storage, writes are usually isolated to small, recent chunks—keeping index sizes small, improving cache locality, and reducing the overhead of vacuum and background maintenance operations. This localized write pattern minimizes write amplification and ensures consistently high ingest performance, even as total data volume grows.

At query time, hypertables efficiently exclude irrelevant chunks from the execution plan when the partitioning column is used in a WHERE clause. This architecture ensures fast query execution, avoiding the gradual slowdowns that affect non-partitioned tables as they accumulate millions of rows. Chunk-local indexes keep indexing overhead minimal, ensuring index operations scans remain efficient regardless of dataset size.

Hypertables are the foundation for all of TimescaleDB’s real-time analytics capabilities. They enable seamless data ingestion, high-throughput writes, optimized query execution, and chunk-based lifecycle management—including automated data retention (drop a chunk) and data tiering (move a chunk to object storage).

Row-columnar storage

Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Columnar storage layout

TimescaleDB’s columnar storage layout optimizes analytical query performance by structuring data efficiently on disk, reducing scan times, and maximizing compression rates. Unlike traditional row-based storage, where data is stored sequentially by row, columnar storage organizes and compresses data by column, allowing queries to retrieve only the necessary fields in batches rather than scanning entire rows. But unlike many column store implementations, TimescaleDB’s columnstore supports full mutability—inserts, upserts, updates, and deletes, even at the individual record level—with transactional guarantees. Data is also immediately visible to queries as soon as it is written.

Columnar batches

TimescaleDB uses columnar collocation and columnar compression within row-based storage to optimize analytical query performance while maintaining full Postgres compatibility. This approach ensures efficient storage, high compression ratios, and rapid query execution.

A rowstore chunk is converted to a columnstore chunk by successfully grouping together sets of rows (typically up to 1000) into a single batch, then converting the batch into columnar form.

Each compressed batch does the following:

• Encapsulates columnar data in compressed arrays of up to 1,000 values per column, stored as a single entry in the underlying compressed table
• Uses a column-major format within the batch, enabling efficient scans by co-locating values of the same column and allowing the selection of individual columns without reading the entire batch
• Applies advanced compression techniques at the column level, including run-length encoding, delta encoding, and Gorilla compression, to significantly reduce storage footprint (by up to 95%) and improve I/O performance.

While the chunk interval of rowstore and columnstore batches usually remains the same, TimescaleDB can also combine columnstore batches so they use a different chunk interval.

This architecture provides the benefits of columnar storage—optimized scans, reduced disk I/O, and improved analytical performance—while seamlessly integrating with Postgres’s row-based execution model.

Segmenting and ordering data

To optimize query performance, TimescaleDB allows explicit control over how data is physically organized within columnar storage. By structuring data effectively, queries can minimize disk reads and execute more efficiently, using vectorized execution for parallel batch processing where possible.

• Group related data together to improve scan efficiency: organizing rows into logical segments ensures that queries filtering by a specific value only scan relevant data sections. For example, in the above, querying for a specific ID is particularly fast. (Implemented with SEGMENTBY.)
• Sort data within segments to accelerate range queries: defining a consistent order reduces the need for post-query sorting, making time-based queries and range scans more efficient. (Implemented with ORDERBY.)
• Reduce disk reads and maximize vectorized execution: a well-structured storage layout enables efficient batch processing (Single Instruction, Multiple Data, or SIMD vectorization) and parallel execution, optimizing query performance.

By combining segmentation and ordering, TimescaleDB ensures that columnar queries are not only fast but also resource-efficient, enabling high-performance real-time analytics.

Data mutability

Traditional databases force a trade-off between fast updates and efficient analytics. Fully immutable storage is impractical in real-world applications, where data needs to change. Asynchronous mutability—where updates only become visible after batch processing—introduces delays that break real-time workflows. In-place mutability, while theoretically ideal, is prohibitively slow in columnar storage, requiring costly decompression, segmentation, ordering, and recompression cycles.

Hypercore navigates these trade-offs with a hybrid approach that enables immediate updates without modifying compressed columnstore data in place. By staging changes in an interim rowstore chunk, hypercore allows updates and deletes to happen efficiently while preserving the analytical performance of columnar storage.

Real-time writes without delays

All new data which is destined for a columnstore chunk is first written to an interim rowstore chunk, ensuring high-speed ingestion and immediate queryability. Unlike fully columnar systems that require ingestion to go through compression pipelines, hypercore allows fresh data to remain in a fast row-based structure before being later compressed into columnar format in ordered batches as normal.

Queries transparently access both the rowstore and columnstore chunks, meaning applications always see the latest data instantly, regardless of its storage format.

Efficient updates and deletes without performance penalties

When modifying or deleting existing data, hypercore avoids the inefficiencies of both asynchronous updates and in-place modifications. Instead of modifying compressed storage directly, affected batches are decompressed and staged in the interim rowstore chunk, where changes are applied immediately.

These modified batches remain in row storage until they are recompressed and reintegrated into the columnstore (which happens automatically via a background process). This approach ensures updates are immediately visible, but without the expensive overhead of decompressing and rewriting entire chunks. This approach avoids:

• The rigidity of immutable storage, which requires workarounds like versioning or copy-on-write strategies
• The delays of asynchronous updates, where modified data is only visible after batch processing
• The performance hit of in-place mutability, which makes compressed storage prohibitively slow for frequent updates
• The restrictions some databases have on not altering the segmentation or ordering keys

Query optimizations

Real-time analytics isn’t just about raw speed—it’s about executing queries efficiently, reducing unnecessary work, and maximizing performance. TimescaleDB optimizes every step of the query lifecycle to ensure that queries scan only what’s necessary, make use of data locality, and execute in parallel for sub-second response times over large datasets.

Skip unnecessary data

TimescaleDB minimizes the amount of data a query touches, reducing I/O and improving execution speed:

Primary partition exclusion (row and columnar)

Queries automatically skip irrelevant partitions (chunks) based on the primary partitioning key (usually a timestamp), ensuring they only scan relevant data.

Secondary partition exclusion (columnar)

Min/max metadata allows queries filtering on correlated dimensions (e.g., order_id or secondary timestamps) to exclude chunks that don’t contain relevant data.

Postgres indexes (row and columnar)

Unlike many databases, TimescaleDB supports sparse indexes on columnstore data, allowing queries to efficiently locate specific values within both row-based and compressed columnar storage. These indexes enable fast lookups, range queries, and filtering operations that further reduce unnecessary data scans.

Batch-level filtering (columnar)

Within each chunk, compressed columnar batches are organized using SEGMENTBY keys and ordered by ORDERBY columns. Indexes and min/max metadata can be used to quickly exclude batches that don’t match the query criteria.

Maximize locality

Organizing data for efficient access ensures queries are read in the most optimal order, reducing unnecessary random reads and reducing scans of unneeded data.

• Segmentation: Columnar batches are grouped using SEGMENTBY to keep related data together, improving scan efficiency.
• Ordering: Data within each batch is physically sorted using ORDERBY, increasing scan efficiency (and reducing I/O operations), enabling efficient range queries, and minimizing post-query sorting.
• Column selection: Queries read only the necessary columns, reducing disk I/O, decompression overhead, and memory usage.

Parallelize execution

Once a query is scanning only the required columnar data in the optimal order, TimescaleDB is able to maximize performance through parallel execution. As well as using multiple workers, TimescaleDB accelerates columnstore query execution by using Single Instruction, Multiple Data (SIMD) vectorization, allowing modern CPUs to process multiple data points in parallel.

The TimescaleDB implementation of SIMD vectorization currently allows:

• Vectorized decompression, which efficiently restores compressed data into a usable form for analysis.
• Vectorized filtering, which rapidly applies filter conditions across data sets.
• Vectorized aggregation, which performs aggregate calculations, such as sum or average, across multiple data points concurrently.

Accelerating queries with continuous aggregates

Aggregating large datasets in real time can be expensive, requiring repeated scans and calculations that strain CPU and I/O. While some databases attempt to brute-force these queries at runtime, compute and I/O are always finite resources—leading to high latency, unpredictable performance, and growing infrastructure costs as data volume increases.

Continuous aggregates, the TimescaleDB implementation of incrementally updated materialized views, solve this by shifting computation from every query run to a single, asynchronous step after data is ingested. Only the time buckets that receive new or modified data are updated, and queries read precomputed results instead of scanning raw data—dramatically improving performance and efficiency.

When you know the types of queries you'll need ahead of time, continuous aggregates allow you to pre-aggregate data along meaningful time intervals—such as per-minute, hourly, or daily summaries—delivering instant results without on-the-fly computation.

Continuous aggregates also avoid the time-consuming and error-prone process of maintaining manual rollups, while continuing to offer data mutability to support efficient updates, corrections, and backfills. Whenever new data is inserted or modified in chunks which have been materialized, TimescaleDB stores invalidation records reflecting that these results are stale and need to be recomputed. Then, an asynchronous process re-computes regions that include invalidated data, and updates the materialized results. TimescaleDB tracks the lineage and dependencies between continuous aggregates and their underlying data, to ensure the continuous aggregates are regularly kept up-to-date. This happens in a resource-efficient manner, and where multiple invalidations can be coalesced into a single refresh (as opposed to refreshing any dependencies at write time, such as via a trigger-based approach).

Continuous aggregates themselves are stored in hypertables, and they can be converted to columnar storage for compression, and raw data can be dropped, reducing storage footprint and processing cost. Continuous aggregates also support hierarchical rollups (e.g., hourly to daily to monthly) and real-time mode, which merges precomputed results with the latest ingested data to ensure accurate, up-to-date analytics.

This architecture enables scalable, low-latency analytics while keeping resource usage predictable—ideal for dashboards, monitoring systems, and any workload with known query patterns.

Hyperfunctions for real-time analytics

Real-time analytics requires more than basic SQL functions—efficient computation is essential as datasets grow in size and complexity. Hyperfunctions, available through the timescaledb_toolkit extension, provide high-performance, SQL-native functions tailored for time-series analysis. These include advanced tools for gap-filling, percentile estimation, time-weighted averages, counter correction, and state tracking, among others.

A key innovation of hyperfunctions is their support for partial aggregation, which allows TimescaleDB to store intermediate computational states rather than just final results. These partials can later be merged to compute rollups efficiently, avoiding expensive reprocessing of raw data and reducing compute overhead. This is especially effective when combined with continuous aggregates.

Consider a real-world example: monitoring request latencies across thousands of application instances. You might want to compute p95 latency per minute, then roll that up into hourly and daily percentiles for dashboards or alerts. With traditional SQL, calculating percentiles requires a full scan and sort of all underlying data—making multi-level rollups computationally expensive.

With TimescaleDB, you can use the percentile_agg hyperfunction in a continuous aggregate to compute and store a partial aggregation state for each minute. This state efficiently summarizes the distribution of latencies for that time bucket, without storing or sorting all individual values. Later, to produce an hourly or daily percentile, you simply combine the stored partials—no need to reprocess the raw latency values.

This approach provides a scalable, efficient solution for percentile-based analytics. By combining hyperfunctions with continuous aggregates, TimescaleDB enables real-time systems to deliver fast, resource-efficient insights across high-ingest, high-resolution datasets—without sacrificing accuracy or flexibility.

Cloud-native architecture

Real-time analytics requires a scalable, high-performance, and cost-efficient database that can handle high-ingest rates and low-latency queries without overprovisioning. Tiger Cloud is designed for elasticity, enabling independent scaling of storage and compute, workload isolation, and intelligent data tiering.

Independent storage and compute scaling

Real-time applications generate continuous data streams while requiring instant querying of both fresh and historical data. Traditional databases force users to pre-provision fixed storage, leading to unnecessary costs or unexpected limits. Tiger Cloud eliminates this constraint by dynamically scaling storage based on actual usage:

• Storage expands and contracts automatically as data is added or deleted, avoiding manual intervention.
• Usage-based billing ensures costs align with actual storage consumption, eliminating large upfront allocations.
• Compute can be scaled independently to optimize query execution, ensuring fast analytics across both recent and historical data.

With this architecture, databases grow alongside data streams, enabling seamless access to real-time and historical insights while efficiently managing storage costs.

Workload isolation for real-time performance

Balancing high-ingest rates and low-latency analytical queries on the same system can create contention, slowing down performance. Tiger Cloud mitigates this by allowing read and write workloads to scale independently:

• The primary database efficiently handles both ingestion and real-time rollups without disruption.
• Read replicas scale query performance separately, ensuring fast analytics even under heavy workloads.

This separation ensures that frequent queries on fresh data don’t interfere with ingestion, making it easier to support live monitoring, anomaly detection, interactive dashboards, and alerting systems.

Intelligent data tiering for cost-efficient real-time analytics

Not all real-time data is equally valuable—recent data is queried constantly, while older data is accessed less frequently. Tiger Cloud can be configured to automatically tier data to cheaper bottomless object storage, ensuring that hot data remains instantly accessible, while historical data is still available.

• Recent, high-velocity data stays in high-performance storage for ultra-fast queries.
• Older, less frequently accessed data is automatically moved to cost-efficient object storage but remains queryable and available for building continuous aggregates.

While many systems support this concept of data cooling, TimescaleDB ensures that the data can still be queried from the same hypertable regardless of its current location. For real-time analytics, this means applications can analyze live data streams without worrying about storage constraints, while still maintaining access to long-term trends when needed.

Cloud-native database observability

Real-time analytics doesn’t just require fast queries—it requires the ability to understand why queries are fast or slow, where resources are being used, and how performance changes over time. That’s why Tiger Cloud is built with deep observability features, giving developers and operators full visibility into their database workloads.

At the core of this observability is Insights, Tiger Cloud’s built-in query monitoring tool. Insights captures per-query statistics from our whole fleet in real time, showing you exactly how your database is behaving under load. It tracks key metrics like execution time, planning time, number of rows read and returned, I/O usage, and buffer cache hit rates—not just for the database as a whole, but for each individual query.

Insights lets you do the following:

• Identify slow or resource-intensive queries instantly
• Spot long-term performance regressions or trends
• Understand query patterns and how they evolve over time
• See the impact of schema changes, indexes, or continuous aggregates on workload performance
• Monitor and compare different versions of the same query to optimize execution

All this is surfaced through an intuitive interface, available directly in Tiger Cloud, with no instrumentation or external monitoring infrastructure required.

Beyond query-level visibility, Tiger Cloud also exposes metrics around service resource consumption, compression, continuous aggregates, and data tiering, allowing you to track how data moves through the system—and how those background processes impact storage and query performance.

Together, these observability features give you the insight and control needed to operate a real-time analytics database at scale, with confidence, clarity, and performance you can trust**.**

Ensuring reliability and scalability

Maintaining high availability, efficient resource utilization, and data durability is essential for real-time applications. Tiger Cloud provides robust operational features to ensure seamless performance under varying workloads.

• High-availability (HA) replicas: deploy multi-AZ HA replicas to provide fault tolerance and ensure minimal downtime. In the event of a primary node failure, replicas are automatically promoted to maintain service continuity.
• Connection pooling: optimize database connections by efficiently managing and reusing them, reducing overhead and improving performance for high-concurrency applications.
• Backup and recovery: leverage continuous backups, Point-in-Time Recovery (PITR), and automated snapshotting to protect against data loss. Restore data efficiently to minimize downtime in case of failures or accidental deletions.

These operational capabilities ensure Tiger Cloud remains reliable, scalable, and resilient, even under demanding real-time workloads.

Conclusion

Real-time analytics is critical for modern applications, but traditional databases struggle to balance high-ingest performance, low-latency queries, and flexible data mutability. Tiger Cloud extends Postgres to solve this challenge, combining automatic partitioning, hybrid row-columnar storage, and intelligent compression to optimize both transactional and analytical workloads.

With continuous aggregates, hyperfunctions, and advanced query optimizations, Tiger Cloud ensures sub-second queries even on massive datasets that combine current and historic data. Its cloud-native architecture further enhances scalability with independent compute and storage scaling, workload isolation, and cost-efficient data tiering—allowing applications to handle real-time and historical queries seamlessly.

For developers, this means building high-performance, real-time analytics applications without sacrificing SQL compatibility, transactional guarantees, or operational simplicity.

Tiger Cloud delivers the best of Postgres, optimized for real-time analytics.

===== PAGE: https://docs.tigerdata.com/about/pricing-and-account-management/ =====

Billing and account management

As we enhance our offerings and align them with your evolving needs, pricing plans provide more value, flexibility, and efficiency for your business. Whether you're a growing startup or a well-established enterprise, our plans are structured to support your journey towards greater success.

This page explains pricing plans for Tiger Cloud, and how to easily manage your Tiger Data account.

Pricing plans give you:

• Enhanced performance: with increased CPU and storage capacities, your apps run smoother and more efficiently, even under heavy loads.
• Improved scalability: as your business grows, so do your demands. Pricing plans scale with you, they provide the resources and support you need at each stage of your growth. Scale up or down based on your current needs, ensuring that you only pay for what you use.
• Better support: access to enhanced support options, including production support and dedicated account management, ensures you have the help you need when you need it.
• Greater flexibility: we know that one size doesn't fit all. Pricing plans give you the flexibility to choose the features and support levels that best match your business and engineering requirements. The ability to add features like I/O boost and customize your pricing plan means you can tailor Tiger Cloud services to fit your specific needs.
• Cost efficiency: by aligning our pricing with the value delivered, we ensure that you get the most out of every dollar spent. Our goal is to help you achieve more with less.

It’s that simple! You don't pay for automated backups or networking costs, such as data ingest or egress. There are no per-query fees, nor additional costs to read or write data. It's all completely transparent, easily understood, and up to you.

Using self-hosted TimescaleDB and our open-source products is still free.

If you create a Tiger Data account from AWS Marketplace, the pricing options are pay-as-you-go and annual commit. See AWS pricing for details.

Disaggregated, consumption-based compute and storage

With Tiger Cloud, you are not limited to pre-set compute and storage. Get as much as you need when provisioning your services or later, as your needs grow.

• Compute: pay only for the compute resources you run. Compute is metered on an hourly basis, and you can scale it up to 64,000 IOPS at any time. You can also scale out using replicas as your application grows. We also provide services to help you lower your compute needs while improving query performance. Tiger Cloud is very efficient and generally needs less compute than other databases to deliver the same performance. The best way to size your needs is to sign up for a free trial and test with a realistic workload.

• Storage: pay only for the storage you consume. You have high-performance storage for more-accessed data, and low-cost bottomless storage in S3 for other data. The high-performance storage offers you up to 64 TB of compressed (typically 80-100 TB uncompressed) data and is metered on your average GB consumption per hour. We can help you compress your data by up to 98% so you pay even less. For low-cost storage, Tiger Data charges only for the size of your data in S3 in the Apache Parquet format, regardless of whether it was compressed in Tiger Cloud before tiering. There are no additional expenses, such as data transfer or compute. For easy upgrades, each service stores the TimescaleDB binaries. This contributes up to 900 MB to overall storage, which amounts to less than $.80/month in additional storage costs.

Use Tiger Cloud for free

Are you just starting out with Tiger Cloud? On our Free pricing plan, you can create up to 2 zero-cost services with limited resources. When a free service reaches the resource limit, it converts to a read-only state.

The Free pricing plan and services are currently in beta.

Ready to try a more feature-rich paid plan? Activate a 30-day free trial of our Performance (no credit card required) or Scale plan. After your trial ends, we may remove your data unless you’ve added a payment method.

After you have completed your 30-day trial period, choose the pricing plan that suits your business and engineering needs. And even when you upgrade from the Free pricing plan, you can still have up to 2 zero-cost services—or convert the ones you already have into standard ones, to have more resources.

If you want to try out features in a higher pricing plan before upgrading, contact us.

Upgrade or downgrade your pricing plans at any time

You can upgrade or downgrade between the Free, Performance, and Scale plans whenever you want using Tiger Cloud Console. To downgrade to the Free plan, you must only have free services running in your project.

If you switch your pricing plan mid-month, your prices are prorated to when you switch. Your services are not interrupted when you switch, so you can keep working without any hassle. To move to Enterprise, get in touch with Tiger Data.

Monitor usage and costs

You keep track of your monthly usage in Tiger Cloud Console. Console shows your resource usage and dashboards with performance insights. This allows you to closely monitor your services’ performance, and any need to scale your services or upgrade your pricing plan.

Console also shows your month-to-date accrued charges, as well as a forecast of your expected month-end bill. Your previous invoices are also available as PDFs for download.

You are charged for all active services in your account, even if you are not actively using them. To reduce costs, pause or delete your unused services.

Tiger Data support

Tiger Data runs a global support organization with Customer Satisfaction (CSAT) scores above 99%. Support covers all timezones, and is fully staffed at weekend hours.

All paid pricing plans have free Developer Support through email with a target response time of 1 business day; we are often faster. If you need 24x7 responsiveness, talk to us about Production Support.

Charging for HA and read replicas

HA and read replicas are both charged at the same rate as your primary services, based on the compute and primary storage consumed by your replicas. Data tiered to our bottomless storage tier is shared by all database replicas; replicas accessing tiered storage do not add to your bill.

Charging over regions

Storage is priced the same across all regions. However, compute prices vary depending on the region. This is because our cloud provider (AWS) prices infrastructure differently based on region.

Features included in each pricing plan

The available pricing plans are:

• Free: for small non-production projects.
• Performance: for cost-focused, smaller projects. No credit card required to start.
• Scale: for developers handling critical and demanding apps.
• Enterprise: for enterprises with mission-critical apps.

The Free pricing plan and services are currently in beta.

The features included in each pricing plan are:

Feature	Free	Performance	Scale	Enterprise
Compute and storage
Number of services	Up to 2 free services	Up to 2 free and 4 standard services	Up to 2 free and and unlimited standard services	Up to 2 free and and unlimited standard services
CPU limit per service	Shared	Up to 8 CPU	Up to 32 CPU	Up to 64 CPU
Memory limit per service	Shared	Up to 32 GB	Up to 128 GB	Up to 256 GB
Storage limit per service	750 MB	Up to 16 TB	Up to 16 TB	Up to 64 TB
Bottomless storage on S3			Unlimited	Unlimited
Independently scale compute and storage		Standard services only	Standard services only	Standard services only
Data services and workloads
Relational	✓	✓	✓	✓
Time-series	✓	✓	✓	✓
Vector search	✓	✓	✓	✓
AI workflows (coming soon)	✓	✓	✓	✓
Cloud SQL editor	3 seats	3 seats	10 seats	20 seats
Charts	✓	✓	✓	✓
Dashboards		2	Unlimited	Unlimited
Storage and performance
IOPS	Shared	3,000 - 5,000	5,000 - 8,000	5,000 - 8,000
Bandwidth (autoscales)	Shared	125 - 250 Mbps	250 - 500 Mbps	Up to 500 mbps
I/O boost			Add-on: Up to 16K IOPS, 1000 Mbps BW	Add-on: Up to 32K IOPS, 4000 Mbps BW
Availability and monitoring
High-availability replicas (Automated multi-AZ failover)		✓	✓	✓
Read replicas			✓	✓
Cross-region backup				✓
Backup reports			14 days	14 days
Point-in-time recovery and forking	1 day	3 days	14 days	14 days
Performance insights	Limited	✓	✓	✓
Metrics and log exporters			✓	✓
Security and compliance
Role-based access	✓	✓	✓	✓
End-to-end encryption	✓	✓	✓	✓
Private Networking (VPC)		1 multi-attach VPC	Unlimited multi-attach VPCs	Unlimited multi-attach VPCs
AWS Transit Gateway			✓	✓
HIPAA compliance				✓
IP address allow list	1 list with up to 10 IP addresses	1 list with up to 10 IP addresses	Up to 10 lists with up to 10 IP addresses each	Up to 10 lists with up to 100 IP addresses each
Multi-factor authentication	✓	✓	✓	✓
Federated authentication (SAML)				✓
SOC 2 Type 2 report			✓	✓
Penetration testing report				✓
Security questionnaire and review				✓
Pay by invoice		Available at minimum spend	Available at minimum spend	✓
Uptime SLAs		Standard	Standard	Enterprise
Support and technical services
Community support	✓	✓	✓	✓
Email support		✓	✓	✓
Production support		Add-on	Add-on	✓
Named account manager				✓
JOIN services (Jumpstart Onboarding and INtegration)			Available at minimum spend	✓

For a personalized quote, get in touch with Tiger Data.

Example billing calculation

You are billed at the end of each month in arrears, based on your actual usage that month. Your monthly invoice includes an itemized cost accounting for each Tiger Cloud service and any additional charges.

Tiger Cloud charges are based on consumption:

• Compute: metered on an hourly basis. You can scale compute up and down at any time.
• Storage: metered based on your average GB consumption per hour. Storage grows and shrinks automatically with your data.

Your monthly price for compute and storage is computed similarly. For example, over the last month your Tiger Cloud service has been running compute for 500 hours total:

• 375 hours with 2 CPU
• 125 hours 4 CPU

Compute cost = (375 x hourly price for 2 CPU) + (125 x hourly price for 4 CPU)

Some add-ons such as tiered storage, HA replicas, and connection pooling may incur additional charges. These charges are clearly marked in your billing snapshot in Tiger Cloud Console.

Manage your Tiger Cloud pricing plan

You handle all details about your Tiger Cloud project including updates to your pricing plan, payment methods, and add-ons in the billing section in Tiger Cloud Console:

• Details: an overview of your pricing plan, usage, and payment details. You can add up to three credit cards to your Wallet. If you prefer to pay by invoice, contact Tiger Data and ask to change to corporate billing.

• History: the list of your downloadable Tiger Cloud invoices.

• Emails: the addresses Tiger Data uses to communicate with you. Payment confirmations and alerts are sent to the email address you signed up with. Add another address to send details to other departments in your organization.

• Pricing plan: choose the pricing plan supplying the features that suit your business and engineering needs.

• Add-ons: add Production support and improved database performance for mission-critical workloads.

AWS Marketplace pricing

When you get Tiger Cloud at AWS Marketplace, the following pricing options are available:

• Pay-as-you-go: your consumption is calculated at the end of the month and included in your AWS invoice. No upfront costs, standard Tiger Cloud rates apply.
• Annual commit: your consumption is calculated at the end of the month ensuring predictable pricing and seamless billing through your AWS account. We confirm the contract terms with you before finalizing the commitment.

===== PAGE: https://docs.tigerdata.com/about/changelog/ =====

Changelog

All the latest features and updates to Tiger Cloud.

TimescaleDB 2.22.1 – configurable indexing, enhanced partitioning, and faster queries

October 10, 2025

TimescaleDB 2.22.1 introduces major performance and flexibility improvements across indexing, compression, and query execution. TimescaleDB 2.22.1 was released on September 30th and is now available to all users of Tiger.

Highlighted features

• Configurable sparse indexes: manually configure sparse indexes (min-max or bloom) on one or more columns of compressed hypertables, optimizing query performance for specific workloads and reducing I/O. In previous versions, these were automatically created based on heuristics and could not be modified.

• UUIDv7 support: native support for UUIDv7 for both compression and partitioning. UUIDv7 embeds a time component, improving insert locality and enabling efficient time-based range queries while maintaining global uniqueness.

  • Vectorized UUID compression: new vectorized compression for UUIDv7 columns doubles query performance and improves storage efficiency by up to 30%.

  • UUIDv7 partitioning: hypertables can now be partitioned on UUIDv7 columns, combining time-based chunking with globally unique IDs—ideal for large-scale event and log data.

• Multi-column SkipScan: expands SkipScan to support multiple distinct keys, delivering millisecond-fast deduplication and DISTINCT ON queries across billions of rows. Learn more in our blog post and documentation.

• Compression improvements: default segmentby and orderby settings are now applied at compression time for each chunk, automatically adapting to evolving data patterns for better performance. This was previously set at the hypertable level and fixed across all chunks.

Deprecations

The experimental Hypercore Table Access Method (TAM) has been removed in this release following advancements in the columnstore architecture.

For a comprehensive list of changes, refer to the TimescaleDB 2.22 & 2.22.1 release notes.

Kafka Source Connector (beta)

September 19, 2025

The new Kafka Source Connector enables you to connect your existing Kafka clusters directly to Tiger Cloud and ingest data from Kafka topics into hypertables. Developers often build proxies or run JDBC Sink Connectors to bridge Kafka and Tiger Cloud, which is error-prone and time-consuming. With the Kafka Source Connector, you can seamlessly start ingesting your Kafka data natively without additional middleware.

• Supported formats: AVRO
• Supported platforms: Confluent Cloud and Amazon Managed Streaming for Apache Kafka

Phased update rollouts, pg_cron, larger compute options, and backup reports

September 12, 2025

🛡️ Phased rollouts for TimescaleDB minor releases

Starting with TimescaleDB 2.22.0, minor releases will now roll out in phases. Services tagged #dev will get upgraded first, followed by #prod after 21 days. This gives you time to validate upgrades in #dev before they reach #prod services. Subscribe to get an email notification before your #prod service is upgraded. See Maintenance and upgrades for details.

⏰ pg_cron extension

pg_cron is now available on Tiger Cloud! With pg_cron, you can:

• Schedule SQL commands to run automatically—like generating weekly sales reports or cleaning up old log entries every night at 2 AM.
• Automate routine maintenance tasks such as refreshing materialized views hourly to keep dashboards current.
• Eliminate external cron jobs and task schedulers, keeping all your automation logic within PostgreSQL.

To enable pg_cron on your service, contact our support team. We're working on making this self-service in future updates.

⚡️ Larger compute options: 48 and 64 CPU

For the most demanding workloads, you can now create services with 48 and 64 CPUs. These options are only available on our Enterprise plan, and they're dedicated instances that are not shared with other customers.

📋 Backup report for compliance

Scale and Enterprise customers can now see a list of their backups in Tiger Cloud Console. For customers with SOC 2 or other compliance needs, this serves as auditable proof of backups.

🗺️ New router for Tiger Cloud Console

The UI just got snappier and easier to navigate with improved interlinking. For example, click an object in the Jobs page to see what hypertable the job is associated with.

New data import wizard

September 5, 2025

To make navigation easier, we’ve introduced a cleaner, more intuitive UI for data import. It highlights the most common and recommended option, PostgreSQL Dump & Restore, while organizing all import options into clear categories, to make navigation easier.

The new categories include:

• PostgreSQL Dump & Restore

• Upload Files: CSV, Parquet, TXT

• Real-time Data Replication: source connectors

• Migrations & Other Options

  

A new data import component has been added to the overview dashboard, providing a clear view of your imports. This includes quick start, in-progress status, and completed imports:

🚁 Enhancements to the Postgres source connector

August 28, 2025

• Easy table selection: You can now sync the complete source schema in one go. Select multiple tables from the drop-down menu and start the connector.

• Sync metadata: Connectors now display the following detailed metadata:

  • Initial data copy: The number of rows copied at any given point in time.
  • Change data capture: The replication lag represented in time and data size.

• Improved UX design: In-progress syncs with separate sections showing the tables and metadata for initial data copy and change data capture, plus a dedicated tab where you can add more tables to the connector.

  

🦋 Developer role GA and hypertable transformation in Console

August 21, 2025

Developer role (GA)

The Developer role in Tiger Cloud is now generally available. It’s a project‑scoped permission set that lets technical users build and operate services, create or modify resources, run queries, and use observability—without admin or billing access. This enforces least‑privilege by default, reducing risk and audit noise, while keeping governance with Admins/Owners and billing with Finance. This means faster delivery (fewer access escalations), protected sensitive settings, and clear boundaries, so the right users can ship changes safely, while compliance and cost control remain intact.

Transform a table to a hypertable from the Explorer

In Console, you can now easily create hypertables from your regular Postgres tables directly from the Explorer. Clicking on any Postgres table shows an option to open up the hypertable action. Follow the simple steps to set up your partition key and transform the table to a hypertable.

Cross-region backups, Postgres options, and onboarding

August 14, 2025

Cross-region backups

You can now store backups in a different region than your service, which improves resilience and helps meet enterprise compliance requirements. Cross‑region backups are available on our Enterprise plan for free at launch; usage‑based billing may be introduced later. For full details, please see the docs.

Standard Postgres instructions for onboarding

We have added basic instructions for INSERT, UPDATE, DELETE commands to the Tiger Cloud console. It's now shown as an option in the Import Data page.

Postgres-only service type

In Tiger Cloud, you now have an option to choose Postgres-only in the service creation flow. Just click Looking for plan PostgreSQL? on the Service Type screen.

Viewer role GA, EXPLAIN plans, and chunk index sizes in Explorer

July 31, 2025

GA release of the viewer role in role-based access

The viewer role is now generally available for all projects and organizations. It provides read-only access to services, metrics, and logs without modify permissions. Viewers cannot create, update, or delete resources, nor manage users or billing. It's ideal for auditors, analysts, and cross-functional collaborators who need visibility but not control.

EXPLAIN plans in Insights

You can now find automatically generated EXPLAIN plans on queries that take longer than 10 seconds within Insights. EXPLAIN plans can be very useful to determine how you may be able to increase the performance of your queries.

Chunk index size in Explorer

Find the index size of hypertable chunks in the Explorer. This information can be very valuable to determine if a hypertable's chunk size is properly configured.

TimescaleDB v2.21 and catalog objects in the Console Explorer

July 25, 2025

🏎️ TimescaleDB v2.21—ingest millions of rows/second and faster columnstore UPSERTs and DELETEs

TimescaleDB v2.21 was released on July 8 and is now available to all developers on Tiger Cloud.

Highlighted features in TimescaleDB v2.21 include:

• High-scale ingestion performance (tech preview): introducing a new approach that compresses data directly into the columnstore during ingestion, demonstrating over 1.2M rows/second in tests with bursts over 50M rows/second. We are actively seeking design partners for this feature.
• Faster data updates (UPSERTs): columnstore UPSERTs are now 2.5x faster for heavily constrained tables, building on the 10x improvement from v2.20.
• Faster data deletion: DELETE operations on non-segmentby columns are 42x faster, reducing I/O and bloat.
• Reduced bloat after recompression: optimized recompression processes lead to less bloat and more efficient storage.
• Enhanced continuous aggregates:
  • Concurrent refresh policies enable multiple continuous aggregates to update concurrently.
  • Batched refreshes are now enabled by default for more efficient processing.
• Complete chunk management: full support for splitting columnstore chunks, complementing the existing merge capabilities.

For a comprehensive list of changes, refer to the TimescaleDB v2.21 release notes.

🔬 Catalog objects available in the Console Explorer

You can now view catalog objects in the Console Explorer. Check out the internal schemas for PostgreSQL and TimescaleDB to better understand the inner workings of your database. To turn on/off visibility, select your service in Tiger Cloud Console, then click Explorer and toggle Show catalog objects.

Iceberg Destination Connector (Tiger Lake)

July 18, 2025

We have released a beta Iceberg destination connector that enables Scale and Enterprise users to integrate Tiger Cloud services with Amazon S3 tables. This enables you to connect Tiger Cloud to data lakes seamlessly. We are actively developing several improvements that will make the overall data lake integration process even smoother.

To use this feature, select your service in Tiger Cloud Console, then navigate to Connectors and select the Amazon S3 Tables destination connector. Integrate the connector to your S3 table bucket by providing the ARN roles, then simply select the tables that you want to sync into S3 tables. See the documentation for details.

🔆Console just got better

July 11, 2025

✏️ Editable jobs in Console

You can now edit jobs directly in Console! We've added the handy pencil icon in the top right corner of any job view. Click a job, hit the edit button, then make your changes. This works for all jobs, even user-defined ones. Tiger Cloud jobs come with custom wizards to guide you through the right inputs. This means you can spot and fix issues without leaving the UI - a small change that makes a big difference!

📊 Connection history

Now you can see your historical connection counts right in the Connections tab! This helps spot those pesky connection management bugs before they impact your app. We're logging max connections every hour (sampled every 5 mins) and might adjust based on your feedback. Just another way we're making the Console more powerful for troubleshooting.

🔐 New in Public Beta: Read-Only Access through RBAC

We’ve just launched Read/Viewer-only access for Tiger Cloud projects into public beta!

You can now invite users with view-only permissions — perfect for folks who need to see dashboards, metrics, and query results, without the ability to make changes.

This has been one of our most requested RBAC features, and it's a big step forward in making Tiger Cloud more secure and collaborative.

No write access. No config changes. Just visibility.

In Console, Go to Project Settings > Users & Roles to try it out, and let us know what you think!

👀 Super useful doc updates

July 4, 2025

Updates to instructions for livesync

In the Console UI, we have clarified the step-by-step procedure for setting up your livesync from self-hosted installations by:

• Adding definitions for some flags when running your Docker container.
• Including more detailed examples of the output from the table synchronization list.

New optional argument for add_continuous_aggregate_policy API

Added the new refresh_newest_first optional argument that controls the order of incremental refreshes.

🚀 Multi-command queries in SQL editor, improved job page experience, multiple AWS Transit Gateways, and a new service creation flow

June 20, 2025

Run multiple statements in SQL editor

Execute complex queries with multiple commands in a single run—perfect for data transformations, table setup, and batch operations.

Branch conversations in SQL assistant

Start new discussion threads from any point in your SQL assistant chat to explore different approaches to your data questions more easily.

Smarter results table

• Expand JSON data instantly: turn complex JSON objects into readable columns with one click—no more digging through nested data structures.
• Filter with precision: use a new smart filter to pick exactly what you want from a dropdown of all available values.

Jobs page improvements

Individual job pages now display their corresponding configuration for TimescaleDB job types—for example, columnstore, retention, CAgg refreshes, tiering, and others.

Multiple AWS Transit Gateways

You can now connect multiple AWS Transit Gateways, when those gateways use overlapping CIDRs. Ideal for teams with zero-trust policies, this lets you keep each network path isolated.

How it works: when you create a new peering connection, Tiger Cloud reuses the existing Transit Gateway if you supply the same ID—otherwise it automatically creates a new, isolated Transit Gateway.

Updated service creation flow

The new service creation flow makes the choice of service type clearer. You can now create distinct types with Postgres extensions for real-time analytics (TimescaleDB), AI (pgvectorscale, pgai), and RTA/AI hybrid applications.

⚙️ Improved Terraform support and TimescaleDB v2.20.3

June 13, 2025

Terraform support for Exporters and AWS Transit Gateway

The latest version of the Timescale Terraform provider (2.3.0) adds support for:

• Creating and attaching observability exporters to your services.
• Securing the connections to your Timescale Cloud services with AWS Transit Gateway.
• Configuring CIDRs for VPC and AWS Transit Gateway connections.

Check the Timescale Terraform provider documentation for more details.

TimescaleDB v2.20.3

This patch release for TimescaleDB v2.20 includes several bug fixes and minor improvements. Notable bug fixes include:

• Adjustments to SkipScan costing for queries that require a full scan of indexed data.
• A fix for issues encountered during dump and restore operations when chunk skipping is enabled.
• Resolution of a bug related to dropped "quals" (qualifications/conditions) in SkipScan.

For a comprehensive list of changes, refer to the TimescaleDB 2.20.3 release notes.

🧘 Read replica sets, faster tables, new anthropic models, and VPC support in data mode

June 6, 2025

Horizontal read scaling with read replica sets

Read replica sets are an improved version of read replicas. They let you scale reads horizontally by creating up to 10 replica nodes behind a single read endpoint. Just point your read queries to the endpoint and configure the number of replicas you need without changing your application logic. You can increase or decrease the number of replicas in the set dynamically, with no impact on the endpoint.

Read replica sets are used to:

• Scale reads for read-heavy workloads and dashboards.
• Isolate internal analytics and reporting from customer-facing applications.
• Provide high availability and fault tolerance for read traffic.

All existing read replicas have been automatically upgraded to a replica set with one node—no action required. Billing remains the same.

Read replica sets are available for all Scale and Enterprise customers.

Faster, smarter results tables in data mode

We've completely rebuilt how query results are displayed in the data mode to give you a faster, more powerful way to work with your data. The new results table can handle millions of rows with smooth scrolling and instant responses when you sort, filter, or format your data. You'll find it today in notebooks and presentation pages, with more areas coming soon.

What's new:

• Your settings stick around: when you customize how your table looks—applying filters, sorting columns, or formatting data—those settings are automatically saved. Switch to another tab and come back, and everything stays exactly how you left it.
• Better ways to find what you need: filter your results by any column value, with search terms highlighted so you can quickly spot what you're looking for. The search box is now available everywhere you work with data.
• Export exactly what you want: download your entire table or just select the specific rows and columns you need. Both CSV and Excel formats are supported.
• See patterns in your data: highlight cells based on their values to quickly spot trends, outliers, or important thresholds in your results.
• Smoother navigation: click any row number to see the full details in an expanded view. Columns automatically resize to show your data clearly, and web links in your results are now clickable.

As a result, working with large datasets is now faster and more intuitive. Whether you're exploring millions of rows or sharing results with your team, the new table keeps up with how you actually work with data.

Latest anthropic models added to SQL assistant

Data mode's SQL assistant now supports Anthropic's latest models:

• Sonnet 4
• Sonnet 4 (extended thinking)
• Opus 4
• Opus 4 (extended thinking)

VPC support for passwordless data mode connections

We previously made it much easier to connect newly created services to Timescale’s data mode. We have now expanded this functionality to services using a VPC.

🕵🏻️ Enhanced service monitoring, TimescaleDB v2.20, and livesync for Postgres

May 30, 2025

Updated top-level navigation - Monitoring tab

In Timescale Console, we have consolidated multiple top-level service information tabs into the single Monitoring tab. This tab houses information previously displayed in the Recommendations, Jobs, Connections, Metrics, Logs, and Insights tabs.

Monitor active connections

In the Connections section under Monitoring, you can now see information like the query being run, the application name, and duration for all current connections to a service.

The information in Connections enables you to debug misconfigured applications, or cancel problematic queries to free up other connections to your database.

TimescaleDB v2.20 - query performance and faster data updates

All new services created on Timescale Cloud are created using TimescaleDB v2.20. Existing services will be automatically upgraded during their maintenance window.

Highlighted features in TimescaleDB v2.20 include:

• Efficiently handle data updates and upserts (including backfills, that are now up to 10x faster).
• Up to 6x faster point queries on high-cardinality columns using new bloom filters.
• Up to 2500x faster DISTINCT operations with SkipScan, perfect for quickly getting a unique list or the latest reading from any device, event, or transaction.
• 8x more efficient Boolean column storage with vectorized processing, resulting in 30-45% faster queries.
• Enhanced developer flexibility with continuous aggregates now supporting window and mutable functions, plus customizable refresh orders.

Postgres 13 and 14 deprecated on Tiger Cloud

TimescaleDB version 2.20 is not compatible with Postgres versions v14 and below. TimescaleDB 2.19.3 is the last bug-fix release for Postgres 14. Future fixes are for Postgres 15+ only. To continue receiving critical fixes and security patches, and to take advantage of the latest TimescaleDB features, you must upgrade to Postgres 15 or newer. This deprecation affects all Tiger Cloud services currently running Postgres 13 or Postgres 14.

The timeline for the Postgres 13 and 14 deprecation is as follows:

• Deprecation notice period begins: starting in early June 2025, you will receive email communication.
• Customer self-service upgrade window: June 2025 through September 14, 2025. We strongly encourage you to manually upgrade Postgres during this period.
• Automatic upgrade deadline: your service will be automatically upgraded from September 15, 2025.

Enhancements to livesync for Postgres

You now can:

• Edit a running livesync to add and drop tables from an existing configuration:

  • For existing tables, Timescale Console stops the livesync while keeping the target table intact.
  • Newly added tables sync their existing data and transition into the Change Data Capture (CDC) state.

• Create multiple livesync instances for Postgres per service. This is an upgrade from our initial launch which limited users to one LiveSync per service.

  This enables you to sync data from multiple Postgres source databases into a single Timescale Cloud service.

• No more hassle looking up schema and table names for livesync configuration from the source. Starting today, all schema and table names are available in a dropdown menu for seamless source table selection.

➕ More storage types and IOPS

May 22, 2025

🚀 Enhanced storage: scale to 64 TB and 32,000 IOPS

We're excited to introduce enhanced storage, a new storage type in Timescale Cloud that significantly boosts both capacity and performance. Designed for customers with mission-critical workloads.

With enhanced storage, Timescale Cloud now supports:

• Up to 64 TB of storage per Timescale Cloud service (4x increase from the previous limit)
• Up to 32,000 IOPS, enabling high-throughput ingest and low-latency queries

Powered by AWS io2 volumes, enhanced storage gives your workloads the headroom they need—whether you're building financial data pipelines, developing IoT platforms, or processing billions of rows of telemetry. No more worrying about storage ceilings or IOPS bottlenecks. Enable enhanced storage in Timescale Console under Operations → Compute & Storage. Enhanced storage is currently available on the Enterprise pricing plan only. Learn more here.

↔️ New export and import options

May 15, 2025

🔥 Ship TimescaleDB metrics to Prometheus

We’re excited to release the Prometheus Exporter for Timescale Cloud, making it easy to ship TimescaleDB metrics to your Prometheus instance. With the Prometheus Exporter, you can:

• Export TimescaleDB metrics like CPU, memory, and storage
• Visualize usage trends with your own Grafana dashboards
• Set alerts for high CPU load, low memory, or storage nearing capacity

To get started, create a Prometheus Exporter in the Timescale Console, attach it to your service, and configure Prometheus to scrape from the exposed URL. Metrics are secured with basic auth. Available on Scale and Enterprise plans. Learn more here.

📥 Import text files into Postgres tables

Our import options in Timescale Console have expanded to include local text files. You can add the content of multiple text files (one file per row) into a Postgres table for use with Vectorizers while creating embeddings for evaluation and development. This new option is located in Service > Actions > Import Data.

🤖 Automatic document embeddings from S3 and a sample dataset for AI testing

May 09, 2025

Automatic document embeddings from S3

pgai vectorizer now supports automatic document vectorization. This makes it dramatically easier to build RAG and semantic search applications on top of unstructured data stored in Amazon S3. With just a SQL command, developers can create, update, and synchronize vector embeddings from a wide range of document formats—including PDFs, DOCX, XLSX, HTML, and more—without building or maintaining complex ETL pipelines.

Instead of juggling multiple systems and syncing metadata, vectorizer handles the entire process: downloading documents from S3, parsing them, chunking text, and generating vector embeddings stored right in Postgres using pgvector. As documents change, embeddings stay up-to-date automatically—keeping your Postgres database the single source of truth for both structured and semantic data.

Sample dataset for AI testing

You can now import a dataset directly from Hugging Face using Timescale Console. This dataset is ideal for testing vectorizers, you find it in the Import Data page under the Service > Actions tab.

🔁 Livesync for S3 and passwordless connections for data mode

April 25, 2025

Livesync for S3 (beta)

Livesync for S3 is our second livesync offering in Timescale Console, following livesync for Postgres. This feature helps users sync data in their S3 buckets to a Timescale Cloud service, and simplifies data importing. Livesync handles both existing and new data in real time, automatically syncing everything into a Timescale Cloud service. Users can integrate Timescale Cloud alongside S3, where S3 stores data in raw form as the source for multiple destinations.

With livesync, users can connect Timescale Cloud with S3 in minutes, rather than spending days setting up and maintaining an ingestion layer.

UX improvements to livesync for Postgres

In livesync for Postgres, getting started requires setting the WAL_LEVEL to logical, and granting specific permissions to start a publication on the source database. To simplify this setup process, we have added a detailed two-step checklist with comprehensive configuration instructions to Timescale Console.

Passwordless data mode connections

We’ve made connecting to your Timescale Cloud services from data mode in Timescale Console even easier! All new services created in Timescale Cloud are now automatically accessible from data mode without requiring you to enter your service credentials. Just open data mode, select your service, and start querying.

We will be expanding this functionality to existing services in the coming weeks (including services using VPC peering), so stay tuned.

☑️ Embeddings spot checks, TimescaleDB v2.19.3, and new models in SQL Assistant

April 18, 2025

Embeddings spot checks

In Timescale Cloud, you can now quickly check the quality of the embeddings from the vectorizers' outputs. Construct a similarity search query with additional filters on source metadata using a simple UI. Run the query right away, or copy it to the SQL editor or data mode and further customize it to your needs. Run the check in Timescale Console > Services > AI:

TimescaleDB v2.19.3

New services created in Timescale Cloud now use TimescaleDB v2.19.3. Existing services are in the process of being automatically upgraded to this version.

This release adds a number of bug fixes including:

• Fix segfault when running a query against columnstore chunks that group by multiple columns, including UUID segmentby columns.
• Fix hypercore table access method segfault on DELETE operations using a segmentby column.

New OpenAI, Llama, and Gemini models in SQL Assistant

The data mode's SQL Assistant now includes support for the latest models from OpenAI and Llama: GPT-4.1 (including mini and nano) and Llama 4 (Scout and Maverick). Additionally, we've added support for Gemini models, in particular Gemini 2.0 Nano and 2.5 Pro (experimental and preview). With the new additions, SQL Assistant supports more than 20 language models so you can select the one best suited to your needs.

🪵 TimescaleDB v2.19, new service overview page, and log improvements

April 11, 2025

TimescaleDB v2.19—query performance and concurrency improvements

Starting this week, all new services created on Timescale Cloud use TimescaleDB v2.19. Existing services will be upgraded gradually during their maintenance window.

Highlighted features in TimescaleDB v2.19 include:

• Improved concurrency of INSERT, UPDATE, and DELETE operations on the columnstore by no longer blocking DML statements during the recompression of a chunk.
• Improved system performance during continuous aggregate refreshes by breaking them into smaller batches. This reduces systems pressure and minimizes the risk of spilling to disk.
• Faster and more up-to-date results for queries against continuous aggregates by materializing the most recent data first, as opposed to old data first in prior versions.
• Faster analytical queries with SIMD vectorization of aggregations over text columns and GROUP BY over multiple columns.
• Enable chunk size optimization for better query performance in the columnstore by merging them with merge_chunk.

New service overview page

The service overview page in Timescale Console has been overhauled to make it simpler and easier to use. Navigate to the Overview tab for any of your services and you will find an architecture diagram and general information pertaining to it. You may also see recommendations at the top, for how to optimize your service.

To leave the product team your feedback, open Help & Support on the left and select Send feedback to the product team.

Find logs faster

Finding logs just got easier! We've added a date, time, and timezone picker, so you can jump straight to the exact moment you're interested in—no more endless scrolling.

📒Faster vector search and improved job information

April 4, 2025

pgvectorscale 0.7.0: faster filtered filtered vector search with filtered indexes

This pgvectorscale release adds label-based filtered vector search to the StreamingDiskANN index. This enables you to return more precise and efficient results by combining vector similarity search with label filtering while still uitilizing the ANN index. This is a common need for large-scale RAG and Agentic applications that rely on vector searches with metadata filters to return relevant results. Filtered indexes add even more capabilities for filtered search at scale, complementing the high accuracy streaming filtering already present in pgvectorscale. The implementation is inspired by Microsoft's Filtered DiskANN research. For more information, see the pgvectorscale release notes and a usage example.

Job errors and individual job pages

Each job now has an individual page in Timescale Console, and displays additional details about job errors. You use this information to debug failing jobs.

To see the job information page, in Timescale Console, select the service to check, then click Jobs > job ID to investigate.

• Successful jobs:

  

• Unsuccessful jobs with errors:

  

🤩 In-Console Livesync for Postgres

March 21, 2025

You can now set up an active data ingestion pipeline with livesync for Postgres in Timescale Console. This tool enables you to replicate your source database tables into Timescale's hypertables indefinitely. Yes, you heard that right—keep livesync running for as long as you need, ensuring that your existing source Postgres tables stay in sync with Timescale Cloud. Read more about setting up and using Livesync for Postgres.

💾 16K dimensions on pgvectorscale plus new pgai Vectorizer support

March 14, 2025

pgvectorscale 0.6 — store up to 16K dimension embeddings

pgvectorscale 0.6.0 now supports storing vectors with up to 16,000 dimensions, removing the previous limitation of 2,000 from pgvector. This lets you use larger embedding models like OpenAI's text-embedding-3-large (3072 dim) with Postgres as your vector database. This release also includes key performance and capability enhancements, including NEON support for SIMD distance calculations on aarch64 processors, improved inner product distance metric implementation, and improved index statistics. See the release details here.

pgai Vectorizer supports models from AWS Bedrock, Azure AI, Google Vertex via LiteLLM

Access embedding models from popular cloud model hubs like AWS Bedrock, Azure AI Foundry, Google Vertex, as well as HuggingFace and Cohere as part of the LiteLLM integration with pgai Vectorizer. To use these models with pgai Vectorizer on Timescale Cloud, select Other when adding the API key in the credentials section of Timescale Console.

🤖 Agent Mode for PopSQL and more

March 7, 2025

Agent Mode for PopSQL

Introducing Agent Mode, a new feature in Timescale Console SQL Assistant. SQL Assistant lets you query your database using natural language. However, if you ran into errors, you have to approve the implementation of the Assistant's suggestions.

With Agent Mode on, SQL Assistant automatically adjusts and executes your query without intervention. It runs, diagnoses, and fixes any errors that it runs into until you get your desired results.

Below you can see SQL Assistant run into an error, identify the resolution, execute the fixed query, display results, and even change the title of the query:

To use Agent Mode, make sure you have SQL Assistant enabled, then click on the model selector dropdown, and tick the Agent Mode checkbox.

Improved AWS Marketplace integration for a smoother experience

We've enhanced the AWS Marketplace workflow to make your experience even better! Now, everything is fully automated, ensuring a seamless process from setup to billing. If you're using the AWS Marketplace integration, you'll notice a smoother transition and clearer billing visibility—your Timescale Cloud subscription will be reflected directly in AWS Marketplace!

Timescale Console recommendations

Sometimes it can be hard to know if you are getting the best use out of your service. To help with this, Timescale Cloud now provides recommendations based on your service's context, assisting with onboarding or notifying if there is a configuration concern with your service, such as consistently failing jobs.

To start, recommendations are focused primarily on onboarding or service health, though we will regularly add new ones. You can see if you have any existing recommendations for your service by going to the Actions tab in Timescale Console.

🛣️ Configuration Options for Secure Connections and More

February 28, 2025

Edit VPC and AWS Transit Gateway CIDRs

You can now modify the CIDRs blocks for your VPC or Transit Gateway directly from Timescale Console, giving you greater control over network access and security. This update makes it easier to adjust your private networking setup without needing to recreate your VPC or contact support.

Improved log filtering

We’ve enhanced the Logs screen with the new Warning and Log filters to help you quickly find the logs you need. These additions complement the existing Fatal, Error, and Detail filters, making it easier to pinpoint specific events and troubleshoot issues efficiently.

TimescaleDB v2.18.2 on Timescale Cloud

New services created in Timescale Cloud now use TimescaleDB v2.18.2. Existing services are in the process of being automatically upgraded to this version.

This new release fixes a number of bugs including:

• Fix ExplainHook breaking the call chain.
• Respect ExecutorStart hooks of other extensions.
• Block dropping internal compressed chunks with drop_chunk().

SQL Assistant improvements

• Support for Claude 3.7 Sonnet and extended thinking including reasoning tokens.
• Ability to abort SQL Assistant requests while the response is streaming.

🤖 SQL Assistant Improvements and Pgai Docs Reorganization

February 21, 2025

New models and improved UX for SQL Assistant

We have added fireworks.ai and Groq as service providers, and several new LLM options for SQL Assistant:

• OpenAI o1
• DeepSeek R1
• Llama 3.3 70B
• Llama 3.1 405B
• DeepSeek R1 Distill - Llama 3.3

We've also improved the model picker by adding descriptions for each model:

Updated and reorganized docs for pgai

We have improved the GitHub docs for pgai. Now relevant sections have been grouped into their own folders and we've created a comprehensive summary doc. Check it out here.

💘 TimescaleDB v2.18.1 and AWS Transit Gateway Support Generally Available

February 14, 2025

TimescaleDB v2.18.1

New services created in Timescale Cloud now use TimescaleDB v2.18.1. Existing services will be automatically upgraded in their next maintenance window starting next week.

This new release includes a number of bug fixes and small improvements including:

• Faster columnar scans when using the hypercore table access method
• Ensure all constraints are always applied when deleting data on the columnstore
• Pushdown all filters on scans for UPDATE/DELETE operations on the columnstore

AWS Transit Gateway support is now generally available!

Timescale Cloud now fully supports AWS Transit Gateway, making it even easier to securely connect your database to multiple VPCs across different environments—including AWS, on-prem, and other cloud providers.

With this update, you can establish a peering connection between your Timescale Cloud services and an AWS Transit Gateway in your AWS account. This keeps your Timescale Cloud services safely behind a VPC while allowing seamless access across complex network setups.

🤖 TimescaleDB v2.18 and SQL Assistant Improvements in Data Mode and PopSQL

February 6, 2025

TimescaleDB v2.18 - dense indexes in the columnstore and query vectorization improvements

Starting this week, all new services created on Timescale Cloud use TimescaleDB v2.18. Existing services will be upgraded gradually during their maintenance window.

Highlighted features in TimescaleDB v2.18.0 include:

• The ability to add dense indexes (btree and hash) to the columnstore through the new hypercore table access method.
• Significant performance improvements through vectorization (SIMD) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
• Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
• Updated methods to manage Timescale's hybrid row-columnar store (hypercore). These methods highlight columnstore usage. The columnstore includes an optimized columnar format as well as compression.

SQL Assistant improvements

We made a few improvements to SQL Assistant:

Dedicated SQL Assistant threads 🧵

Each query, notebook, and dashboard now gets its own conversation thread, keeping your chats organized.

Delete messages ❌

Made a typo? Asked the wrong question? You can now delete individual messages from your thread to keep the conversation clean and relevant.

Support for OpenAI o3-mini ⚡

We’ve added support for OpenAI’s latest o3-mini model, bringing faster response times and improved reasoning for SQL queries.

🌐 IP Allowlists in Data Mode and PopSQL

January 31, 2025

For enhanced network security, you can now also create IP allowlists in the Timescale Console data mode and PopSQL. Similarly to the ops mode IP allowlists, this feature grants access to your data only to certain IP addresses. For example, you might require your employees to use a VPN and add your VPN static egress IP to the allowlist.

This feature is available in:

• Timescale Console data mode, for all pricing tiers
• PopSQL web
• PopSQL desktop

Enable this feature in PopSQL/Timescale Console data mode > Project > Settings > IP Allowlist:

🤖 pgai Extension and Python Library Updates

January 24, 2025

AI — pgai Postgres extension 0.7.0

This release enhances the Vectorizer functionality by adding configurable base_url support for OpenAI API. This enables pgai Vectorizer to use all OpenAI-compatible models and APIs via the OpenAI integration simply by changing the base_url. This release also includes public granting of vectorizers, superuser creation on any table, an upgrade to the Ollama client to 0.4.5, a new docker-start command, and various fixes for struct handling, schema qualification, and system package management. See all changes on Github.

AI - pgai python library 0.5.0

This release adds comprehensive SQLAlchemy and Alembic support for vector embeddings, including operations for migrations and improved model inheritance patterns. You can now seamlessly integrate vector search capabilities with SQLAlchemy models while utilizing Alembic for database migrations. This release also adds key improvements to the Ollama integration and self-hosted Vectorizer configuration. See all changes on Github.

AWS Transit Gateway Support

January 17, 2025

AWS Transit Gateway Support (Early Access)

Timescale Cloud now enables you to connect to your Timescale Cloud services through AWS Transit Gateway. This feature is available to Scale and Enterprise customers. It will be in Early Access for a short time and available in the Timescale Console very soon. If you are interested in implementing this Early Access Feature, reach out to your Rep.

🇮🇳 New region in India, Postgres 17 upgrades, and TimescaleDB on AWS Marketplace

January 10, 2025

Welcome India! (Support for a new region: Mumbai)

Timescale Cloud now supports the Mumbai region. Starting today, you can run Timescale Cloud services in Mumbai, bringing our database solutions closer to users in India.

Postgres major version upgrades to PG 17

Timescale Cloud services can now be upgraded directly to Postgres 17 from versions 14, 15, or 16. Users running versions 12 or 13 must first upgrade to version 15 or 16, before upgrading to 17.

Timescale Cloud available on AWS Marketplace

Timescale Cloud is now available in the AWS Marketplace. This allows you to keep billing centralized on your AWS account, use your already committed AWS Enterprise Discount Program spend to pay your Timescale Cloud bill and simplify procurement and vendor management.

🎅 Postgres 17, feature requests, and Postgres Livesync

December 20, 2024

Postgres 17

All new Timescale Cloud services now come with Postgres 17.2, the latest version. Upgrades to Postgres 17 for services running on prior versions will be available in January. Postgres 17 adds new capabilities and improvements to Timescale like:

• System-wide Performance Improvements. Significant performance boosts, particularly in high-concurrency workloads. Enhancements in the I/O layer, including improved Write-Ahead Log (WAL) processing, can result in up to a 2x increase in write throughput under heavy loads.
• Enhanced JSON Support. The new JSON_TABLE allows developers to convert JSON data directly into relational tables, simplifying the integration of JSON and SQL. The release also adds new SQL/JSON constructors and query functions, offering powerful tools to manipulate and query JSON data within a traditional relational schema.
• More Flexible MERGE Operations. The MERGE command now includes a RETURNING clause, making it easier to track and work with modified data. You can now also update views using MERGE, unlocking new use cases for complex queries and data manipulation.

Submit feature requests from Timescale Console

You can now submit feature requests directly from Console and see the list of feature requests you have made. Just click on Feature Requests on the right sidebar. All feature requests are automatically published to the Timescale Forum and are reviewed by the product team, providing more visibility and transparency on their status as well as allowing other customers to vote for them.

Postgres Livesync (Alpha release)

We have built a new solution that helps you continuously replicate all or some of your Postgres tables directly into Timescale Cloud.

Livesync allows you to keep a current Postgres instance such as RDS as your primary database, and easily offload your real-time analytical queries to Timescale Cloud to boost their performance. If you have any questions or feedback, talk to us in #livesync in Timescale Community.

This is just the beginning—you'll see more from livesync in 2025!

In-Console import from S3, I/O Boost, and Jobs Explorer

December 13, 2024

In-Console import from S3 (CSV and Parquet files)

Connect your S3 buckets to import data into Timescale Cloud. We support CSV (including .zip and .gzip) and Parquet files, with a 10 GB size limit in this initial release. This feature is accessible in the Import your data section right after service creation and through the Actions tab.

Self-Serve I/O Boost 📈

I/O Boost is an add-on for customers on Scale or Enterprise tiers that maximizes the I/O capacity of EBS storage to 16,000 IOPS and 1,000 MBps throughput per service. To enable I/O Boost, navigate to Services > Operations in Timescale Console. A simple toggle allows you to enable the feature, with pricing clearly displayed at $0.41/hour per node.

Jobs Explorer

See all the jobs associated with your service through a new Jobs tab. You can see the type of job, its status (Running, Paused, and others), and a detailed history of the last 100 runs, including success rates and runtime statistics.

🛝 New service creation flow

December 6, 2024

• AI and Vector: the UI now lets you choose an option for creating AI and Vector-ready services right from the start. You no longer need to add the pgai, pgvector, and pgvectorscale extensions manually. You can combine this with time-series capabilities as well!

  

• Compute size recommendations: new (and old) users were sometimes unsure about what compute size to use for their workload. We now offer compute size recommendations based on how much data you plan to have in your service.

  

• More information about configuration options: we've made it clearer what each configuration option does, so that you can make more informed choices about how you want your service to be set up.

🗝️ IP Allow Lists!

November 21, 2024

IP Allow Lists let you specify a list of IP addresses that have access to your Timescale Cloud services and block any others. IP Allow Lists are a lightweight but effective solution for customers concerned with security and compliance. They enable you to prevent unauthorized connections without the need for a Virtual Private Cloud (VPC).

To get started, in Timescale Console, select a service, then click Operations > Security > IP Allow List, then create an IP Allow List.

For more information, see our docs.

🤩 SQL Assistant, TimescaleDB v2.17, HIPAA compliance, and better logging

November 14, 2024

🤖 New AI companion: SQL Assistant

SQL Assistant uses AI to help you write SQL faster and more accurately.

• Real-time help: chat with models like OpenAI 4o and Claude 3.5 Sonnet to get help writing SQL. Describe what you want in natural language and have AI write the SQL for you.

• Error resolution: when you run into an error, SQL Assistant proposes a recommended fix that you can choose to accept.

  

• Generate titles and descriptions: click a button and SQL Assistant generates a title and description for your query. No more untitled queries!

  

See our blog post or docs for full details!

🏄 TimescaleDB v2.17 - performance improvements for analytical queries and continuous aggregate refreshes

Starting this week, all new services created on Timescale Cloud use TimescaleDB v2.17. Existing services are upgraded gradually during their maintenance windows.

TimescaleDB v2.17 significantly improves the performance of continuous aggregate refreshes, and contains performance improvements for analytical queries and delete operations over compressed hypertables.

Best practice is to upgrade at the next available opportunity.

Highlighted features in TimescaleDB v2.17 are:

• Significant performance improvements for continuous aggregate policies:

  • Continuous aggregate refresh now uses merge instead of deleting old materialized data and re-inserting.

  • Continuous aggregate policies are now more lightweight, use less system resources, and complete faster. This update:

    • Decreases dramatically the amount of data that must be written on the continuous aggregate in the presence of a small number of changes
    • Reduces the i/o cost of refreshing a continuous aggregate
    • Generates fewer Write-Ahead Logs (WAL)

• Increased performance for real-time analytical queries over compressed hypertables:

  • We are excited to introduce additional Single Instruction, Multiple Data (SIMD) vectorization optimization to TimescaleDB. This release supports vectorized execution for queries that group by using the segment_by column(s), and aggregate using the sum, count, avg, min, and max basic aggregate functions.

  • Stay tuned for more to come in follow-up releases! Support for grouping on additional columns, filtered aggregation, vectorized expressions, and time_bucket is coming soon.

  • Improved performance of deletes on compressed hypertables when a large amount of data is affected.

    This improvement speeds up operations that delete whole segments by skipping the decompression step. It is enabled for all deletes that filter by the segment_by column(s).

HIPAA compliance

Timescale Cloud's Enterprise plan is now HIPAA (Health Insurance Portability and Accountability Act) compliant. This allows organizations to securely manage and analyze sensitive healthcare data, ensuring they meet regulatory requirements while building compliant applications.

Expanded logging within Timescale Console

Customers can now access more than just the most recent 500 logs within the Timescale Console. We've updated the user experience, including scrollbar with infinite scrolling capabilities.

✨ Connect to Timescale from .NET Stack and check status of recent jobs

November 07, 2024

Connect to Timescale with your .NET stack

We've added instructions for connecting to Timescale using your .NET workflow. In Console after service creation, or in the Actions tab, you can now select .NET from the developer library list. The guide demonstrates how to use Npgsql to integrate Timescale with your existing software stack.

✅ Last 5 jobs status

In the Jobs section of the Explorer, users can now see the status (completed/failed) of the last 5 runs of each job.

🎃 New AI, data integration, and performance enhancements

October 31, 2024

Pgai Vectorizer: vector embeddings as database indexes (early access)

This early access feature enables you to automatically create, update, and maintain embeddings as your data changes. Just like an index, Timescale handles all the complexity: syncing, versioning, and cleanup happen automatically. This means no manual tracking, zero maintenance burden, and the freedom to rapidly experiment with different embedding models and chunking strategies without building new pipelines. Navigate to the AI tab in your service overview and follow the instructions to add your OpenAI API key and set up your first vectorizer or read our guide to automate embedding generation with pgai Vectorizer for more details.

Postgres-to-Postgres foreign data wrappers:

Fetch and query data from multiple Postgres databases, including time-series data in hypertables, directly within Timescale Cloud using foreign data wrappers (FDW). No more complicated ETL processes or external tools—just seamless integration right within your SQL editor. This feature is ideal for developers who manage multiple Postgres and time-series instances and need quick, easy access to data across databases.

Faster queries over tiered data

This release adds support for runtime chunk exclusion for queries that need to access tiered storage. Chunk exclusion now works with queries that use stable expressions in the WHERE clause. The most common form of this type of query is:

SELECT * FROM  hypertable WHERE timestamp_col > now() - '100 days'::interval

For more info on queries with immutable/stable/volatile filters, check our blog post on Implementing constraint exclusion for faster query performance.

If you no longer want to use tiered storage for a particular hypertable, you can now disable tiering and drop the associated tiering metadata on the hypertable with a call to disable_tiering function.

Chunk interval recommendations

Timescale Console now shows recommendations for services with too many small chunks in their hypertables. Recommendations for new intervals that improve service performance are displayed for each underperforming service and hypertable. Users can then change their chunk interval and boost performance within Timescale Console.

💡 Help with hypertables and faster notebooks

October 18, 2024

🧙Hypertable creation wizard

After creating a service, users can now create a hypertable directly in Timescale Console by first creating a table, then converting it into a hypertable. This is possible using the in-console SQL editor. All standard hypertable configuration options are supported, along with any customization of the underlying table schema.

🍭 PopSQL Notebooks

The newest version of Data Mode Notebooks is now waaaay faster. Why? We've incorporated the newly developed v3 of our query engine that currently powers Timescale Console's SQL Editor. Check out the difference in query response times.

✨ Production-Ready Low-Downtime Migrations, MySQL Import, Actions Tab, and Current Lock Contention Visibility in SQL Editor

October 10, 2024

🏗️ Live Migrations v1.0 Release

Last year, we began developing a solution for low-downtime migration from Postgres and TimescaleDB. Since then, this solution has evolved significantly, featuring enhanced functionality, improved reliability, and performance optimizations. We're now proud to announce that live migration is production-ready with the release of version 1.0.

Many of our customers have successfully migrated databases to Timescale using live migration, with some databases as large as a few terabytes in size.

🔁 Actions Tab

As part of the service creation flow, we offer the following:

• Connect to services from different sources
• Import and migrate data from various sources
• Create hypertables

Previously, these actions were only visible during the service creation process and couldn't be accessed later. Now, these actions are persisted within the service, allowing users to leverage them on-demand whenever they're ready to perform these tasks.

🧭 Import Data from MySQL

We've noticed users struggling to convert their MySQL schema and data into their Timescale Cloud services. This was due to the semantic differences between MySQL and Postgres. To simplify this process, we now offer easy-to-follow instructions to import data from MySQL to Timescale Cloud. This feature is available as part of the data import wizard, under the Import from MySQL option.

🔐 Current Lock Contention

In Timescale Console, we offer the SQL editor so you can directly querying your service. As a new improvement, if a query is waiting on locks and can't complete execution, Timescale Console now displays the current lock contention in the results section .

CIDR & VPC Updates

October 3, 2024

Timescale now supports multiple CIDRs on the customer VPC. Customers who want to take advantage of multiple CIDRs will need to recreate their peering.

🤝 New modes in Timescale Console: Ops and Data mode, and Console based Parquet File Import

September 19, 2024

We've been listening to your feedback and noticed that Timescale Console users have diverse needs. Some of you are focused on operational tasks like adding replicas or changing parameters, while others are diving deep into data analysis to gather insights.

To better serve you, we've introduced new modes to the Timescale Console UI—tailoring the experience based on what you're trying to accomplish.

Ops mode is where you can manage your services, add replicas, configure compression, change parameters, and so on.

Data mode is the full PopSQL experience: write queries with autocomplete, visualize data with charts and dashboards, schedule queries and dashboards to create alerts or recurring reports, share queries and dashboards, and more.

Try it today and let us know what you think!

Console based Parquet File Import

Now users can upload from Parquet to Timescale Cloud by uploading the file from their local file system. For files larger than 250 MB, or if you want to do it yourself, follow the three-step process to upload Parquet files to Timescale.

SQL editor improvements

• In the Ops mode SQL editor, you can now highlight a statement to run a specific statement.

High availability, usability, and migrations improvements

September 12, 2024

Multiple HA replicas

Scale and Enterprise customers can now configure two new multiple high availability (HA) replica options directly through Timescale Console:

• Two HA replicas (both asynchronous) - our highest availability configuration.
• Two HA replicas (one asynchronous, one synchronous) - our highest data integrity configuration.

Previously, Timescale offered only a single synchronous replica for customers seeking high availability. The single HA option is still available.

For more details on multiple HA replicas, see Manage high availability.

Other improvements

• In the Console SQL editor, we now indicate if your database session is healthy or has been disconnected. If it's been disconnected, the session will reconnect on your next query execution.

  

• Released live-migration v0.0.26 and then v0.0.27 which includes multiple performance improvements and bugfixes as well as better support for Postgres 12.

One-click SQL statement execution from Timescale Console, and session support in the SQL editor

September 05, 2024

One-click SQL statement execution from Timescale Console

Now you can simply click to run SQL statements in various places in the Console. This requires that the SQL Editor is enabled for the service.

• Enable Continuous Aggregates from the CAGGs wizard by clicking Run below the SQL statement.

• Enable database extensions by clicking Run below the SQL statement.

• Query data instantly with a single click in the Console after successfully uploading a CSV file.

Session support in the SQL editor

Last week we announced the new in-console SQL editor. However, there was a limitation where a new database session was created for each query execution.

Today we removed that limitation and added support for keeping one database session for each user logged in, which means you can do things like start transactions:

begin;
insert into users (name, email) values ('john doe', 'john@example.com');
abort; -- nothing inserted

Or work with temporary tables:

create temporary table temp_users (email text);
insert into temp_sales (email) values ('john@example.com');
-- table will automatically disappear after your session ends

Or use the set command:

set search_path to 'myschema', 'public';

😎 Query your database directly from the Console and enhanced data import and migration options

August 30, 2024

SQL Editor in Timescale Console

We've added a new tab to the service screen that allows users to query their database directly, without having to leave the console interface.

• For existing services on Timescale, this is an opt-in feature. For all newly created services, the SQL Editor will be enabled by default.
• Users can disable the SQL Editor at any time by toggling the option under the Operations tab.
• The editor supports all DML and DDL operations (any single-statement SQL query), but doesn't support multiple SQL statements in a single query.

Enhanced Data Import Options for Quick Evaluation

After service creation, we now offer a dedicated section for data import, including options to import from Postgres as a source or from CSV files.

The enhanced Postgres import instructions now offer several options: single table import, schema-only import, partial data import (allowing selection of a specific time range), and complete database import. Users can execute any of these data imports with just one or two simple commands provided in the data import section.

Improvements to Live migration

We've released v0.0.25 of Live migration that includes the following improvements:

• Support migrating tsdb on non public schema to public schema
• Pre-migration compatibility checks
• Docker compose build fixes

🛠️ Improved tooling in Timescale Cloud and new AI and Vector extension releases

August 22, 2024

CSV import

We have added a CSV import tool to the Timescale Console. For all TimescaleDB services, after service creation you can:

• Choose a local file
• Select the name of the data collection to be uploaded (default is file name)
• Choose data types for each column
• Upload the file as a new hypertable within your service Look for the Import data from .csv tile in the Import your data step of service creation.

Replica lag

Customers now have more visibility into the state of replicas running on Timescale Cloud. We’ve released a new parameter called Replica Lag within the Service Overview for both Read and High Availability Replicas. Replica lag is measured in bytes against the current state of the primary database. For questions or concerns about the relative lag state of your replica, reach out to Customer Support.

Adjust chunk interval

Customers can now adjust their chunk interval for their hypertables and continuous aggregates through the Timescale UI. In the Explorer, select the corresponding hypertable you would like to adjust the chunk interval for. Under Chunk information, you can change the chunk interval. Note that this only changes the chunk interval going forward, and does not retroactively change existing chunks.

CloudWatch permissions via role assumption

We've released permission granting via role assumption to CloudWatch. Role assumption is both more secure and more convenient for customers who no longer need to rotate credentials and update their exporter config.

For more details take a look at our documentation.

Two-factor authentication (2FA) indicator

We’ve added a 2FA status column to the Members page, allowing customers to easily see whether each project member has 2FA enabled or disabled.

Anthropic and Cohere integrations in pgai

The pgai extension v0.3.0 now supports embedding creation and LLM reasoning using models from Anthropic and Cohere. For details and examples, see this post for pgai and Cohere, and this post for pgai and Anthropic.

pgvectorscale extension: ARM builds and improved recall for low dimensional vectors

pgvectorscale extension v0.3.0 adds support for ARM processors and improves recall when using StreamingDiskANN indexes with low dimensionality vectors. We recommend updating to this version if you are self-hosting.

🏄 Optimizations for compressed data and extended join support in continuous aggregates

August 15, 2024

TimescaleDB v2.16.0 contains significant performance improvements when working with compressed data, extended join support in continuous aggregates, and the ability to define foreign keys from regular tables towards hypertables. We recommend upgrading at the next available opportunity.

Any new service created on Timescale Cloud starting today uses TimescaleDB v2.16.0.

In TimescaleDB v2.16.0 we:

• Introduced multiple performance focused optimizations for data manipulation operations (DML) over compressed chunks.

  Improved upsert performance by more than 100x in some cases and more than 500x in some update/delete scenarios.

• Added the ability to define chunk skipping indexes on non-partitioning columns of compressed hypertables.

  TimescaleDB v2.16.0 extends chunk exclusion to use these skipping (sparse) indexes when queries filter on the relevant columns, and prune chunks that do not include any relevant data for calculating the query response.

• Offered new options for use cases that require foreign keys defined.

  You can now add foreign keys from regular tables towards hypertables. We have also removed some really annoying locks in the reverse direction that blocked access to referenced tables while compression was running.

• Extended Continuous Aggregates to support more types of analytical queries.

  More types of joins are supported, additional equality operators on join clauses, and support for joins between multiple regular tables.

Highlighted features in this release

• Improved query performance through chunk exclusion on compressed hypertables.

  You can now define chunk skipping indexes on compressed chunks for any column with one of the following integer data types: smallint, int, bigint, serial, bigserial, date, timestamp, timestamptz.

  After calling enable_chunk_skipping on a column, TimescaleDB tracks the min and max values for that column, using this information to exclude chunks for queries filtering on that column, where no data would be found.

• Improved upsert performance on compressed hypertables.

  By using index scans to verify constraints during inserts on compressed chunks, TimescaleDB speeds up some ON CONFLICT clauses by more than 100x.

• Improved performance of updates, deletes, and inserts on compressed hypertables.

  By filtering data while accessing the compressed data and before decompressing, TimescaleDB has improved performance for updates and deletes on all types of compressed chunks, as well as inserts into compressed chunks with unique constraints.

  By signaling constraint violations without decompressing, or decompressing only when matching records are found in the case of updates, deletes and upserts, TimescaleDB v2.16.0 speeds up those operations more than 1000x in some update/delete scenarios, and 10x for upserts.

• You can add foreign keys from regular tables to hypertables, with support for all types of cascading options. This is useful for hypertables that partition using sequential IDs, and need to reference these IDs from other tables.

• Lower locking requirements during compression for hypertables with foreign keys

  Advanced foreign key handling removes the need for locking referenced tables when new chunks are compressed. DML is no longer blocked on referenced tables while compression runs on a hypertable.

• Improved support for queries on Continuous Aggregates

  INNER/LEFT and LATERAL joins are now supported. Plus, you can now join with multiple regular tables, and have more than one equality operator on join clauses.

Postgres 13 support removal announcement

Following the deprecation announcement for Postgres 13 in TimescaleDB v2.13, Postgres 13 is no longer supported in TimescaleDB v2.16.

The currently supported Postgres major versions are 14, 15, and 16.

📦 Performance, packaging and stability improvements for Timescale Cloud

August 8, 2024

New plans

To support evolving customer needs, Timescale Cloud now offers three plans to provide more value, flexibility, and efficiency.

• Performance: for cost-focused, smaller projects. No credit card required to start.
• Scale: for developers handling critical and demanding apps.
• Enterprise: for enterprises with mission-critical apps.

Each plan continues to bill based on hourly usage, primarily for compute you run and storage you consume. You can upgrade or downgrade between Performance and Scale plans via the Console UI at any time. More information about the specifics and differences between these pricing plans can be found here in the docs.

Improvements to the Timescale Console

The individual tiles on the services page have been enhanced with new information, including high-availability status. This will let you better assess the state of your services at a glance.

Live migration release v0.0.24

Improvements:

• Automatic retries are now available for the initial data copy of the migration
• Now uses pgcopydb for initial data copy for PG to TSDB migrations also (already did for TS to TS) which has a significant performance boost.
• Fixes issues with TimescaleDB v2.13.x migrations
• Support for chunk mapping for hypertables with custom schema and table prefixes

⚡ Performance and stability improvements for Timescale Cloud and TimescaleDB

July 12, 2024

The following improvements have been made to Timescale products:

• Timescale Cloud:

  • The connection pooler has been updated and now avoids multiple reloads
  • The tsdbadmin user can now grant the following roles to other users: pg_checkpoint,pg_monitor,pg_signal_backend,pg_read_all_stats,pg_stat_scan_tables
  • Timescale Console is far more reliable.

• TimescaleDB

  • The TimescaleDB v2.15.3 patch release improves handling of multiple unique indexes in a compressed INSERT, removes the recheck of ORDER when querying compressed data, improves memory management in DML functions, improves the tuple lock acquisition for tiered chunks on replicas, and fixes an issue with ORDER BY/GROUP BY in our HashAggregate optimization on PG16. For more information, see the release note.
  • The TimescaleDB v2.15.2 patch release improves sort pushdown for partially compressed chunks, and compress_chunk with a primary space partition. The metadata function is removed from the update script, and hash partitioning on a primary column is disallowed. For more information, see the release note.

⚡ Performance improvements for live migration to Timescale Cloud

June 27, 2024

The following improvements have been made to the Timescale live-migration docker image:

• Table-based filtering is now available during live migration.
• Improvements to pbcopydb increase performance and remove unhelpful warning messages.
• The user notification log enables you to always select the most recent release for a migration run.

For improved stability and new features, update to the latest timescale/live-migration docker image. To learn more, see the live migration docs.

🦙Ollama integration in pgai

June 21, 2024

Ollama is now integrated with pgai.

Ollama is the easiest and most popular way to get up and running with open-source language models. Think of Ollama as Docker for LLMs, enabling easy access and usage of a variety of open-source models like Llama 3, Mistral, Phi 3, Gemma, and more.

With the pgai extension integrated in your database, embed Ollama AI into your app using SQL. For example:

select ollama_generate
( 'llava:7b'
, 'Please describe this image.'
, _images=> array[pg_read_binary_file('/pgai/tests/postgresql-vs-pinecone.jpg')]
, _system=>'you are a helpful assistant'
, _options=> jsonb_build_object
  ( 'seed', 42
  , 'temperature', 0.9
  )
)->>'response'
;

To learn more, see the pgai Ollama documentation.

🧙 Compression Wizard

June 13, 2024

The compression wizard is now available on Timescale Cloud. Select a hypertable and be guided through enabling compression through the UI!

To access the compression wizard, navigate to Explorer, and select the hypertable you would like to compress. In the top right corner, hover where it says Compression off, and open the wizard. You will then be guided through the process of configuring compression for your hypertable, and can compress it directly through the UI.

🏎️💨 High Performance AI Apps With pgvectorscale

June 11, 2024

The vectorscale extension is now available on Timescale Cloud.

pgvectorscale complements pgvector, the open-source vector data extension for Postgres, and introduces the following key innovations for pgvector data:

• A new index type called StreamingDiskANN, inspired by the DiskANN algorithm, based on research from Microsoft.
• Statistical Binary Quantization: developed by Timescale researchers, This compression method improves on standard Binary Quantization.

On benchmark dataset of 50 million Cohere embeddings (768 dimensions each), Postgres with pgvector and pgvectorscale achieves 28x lower p95 latency and 16x higher query throughput compared to Pinecone's storage optimized (s1) index for approximate nearest neighbor queries at 99% recall, all at 75% less cost when self-hosted on AWS EC2.

To learn more, see the pgvectorscale documentation.

🧐Integrate AI Into Your Database Using pgai

June 11, 2024

The pgai extension is now available on Timescale Cloud.

pgai brings embedding and generation AI models closer to the database. With pgai, you can now do the following directly from within Postgres in a SQL query:

• Create embeddings for your data.
• Retrieve LLM chat completions from models like OpenAI GPT4o.
• Reason over your data and facilitate use cases like classification, summarization, and data enrichment on your existing relational data in Postgres.

To learn more, see the pgai documentation.

🐅Continuous Aggregate and Hypertable Improvements for TimescaleDB

June 7, 2024

The 2.15.x releases contains performance improvements and bug fixes. Highlights in these releases are:

• Continuous Aggregate now supports time_bucket with origin and/or offset.
• Hypertable compression has the following improvements:
  • Recommend optimized defaults for segment by and order by when configuring compression through analysis of table configuration and statistics.
  • Added planner support to check more kinds of WHERE conditions before decompression. This reduces the number of rows that have to be decompressed.
  • You can now use minmax sparse indexes when you compress columns with btree indexes.
  • Vectorize filters in the WHERE clause that contain text equality operators and LIKE expressions.

To learn more, see the TimescaleDB release notes.

🔍 Database Audit Logging with pgaudit

May 31, 2024

The Postgres Audit extension(pgaudit) is now available on Timescale Cloud. pgaudit provides detailed database session and object audit logging in the Timescale Cloud logs.

If you have strict security and compliance requirements and need to log all operations on the database level, pgaudit can help. You can also export these audit logs to Amazon CloudWatch.

To learn more, see the pgaudit documentation.

🌡 International System of Unit Support with postgresql-unit

May 31, 2024

The SI Units for Postgres extension(unit) provides support for the ISU in Timescale Cloud.

You can use Timescale Cloud to solve day-to-day questions. For example, to see what 50°C is in °F, run the following query in your Timescale Cloud service:

SELECT '50°C'::unit @ '°F' as temp;
  temp
--------
 122 °F
(1 row)

To learn more, see the postgresql-unit documentation.

===== PAGE: https://docs.tigerdata.com/about/timescaledb-editions/ =====

Compare TimescaleDB editions

The following versions of TimescaleDB are available:

• TimescaleDB Apache 2 Edition
• TimescaleDB Community Edition

TimescaleDB Apache 2 Edition

TimescaleDB Apache 2 Edition is available under the Apache 2.0 license. This is a classic open source license, meaning that it is completely unrestricted - anyone can take this code and offer it as a service.

You can install TimescaleDB Apache 2 Edition on your own on-premises or cloud infrastructure and run it for free.

You can sell TimescaleDB Apache 2 Edition as a service, even if you're not the main contributor.

You can modify the TimescaleDB Apache 2 Edition source code and run it for production use.

TimescaleDB Community Edition

TimescaleDB Community Edition is the advanced, best, and most feature complete version of TimescaleDB, available under the terms of the Tiger Data License (TSL).

For more information about the Tiger Data license, see this blog post.

Many of the most recent features of TimescaleDB are only available in TimescaleDB Community Edition.

You can install TimescaleDB Community Edition in your own on-premises or cloud infrastructure and run it for free. TimescaleDB Community Edition is completely free if you manage your own service.

You cannot sell TimescaleDB Community Edition as a service, even if you are the main contributor.

You can modify the TimescaleDB Community Edition source code and run it for production use. Developers using TimescaleDB Community Edition have the "right to repair" and make modifications to the source code and run it in their own on-premises or cloud infrastructure. However, you cannot make modifications to the TimescaleDB Community Edition source code and offer it as a service.

You can access a hosted version of TimescaleDB Community Edition through Tiger Cloud, a cloud-native platform for time-series and real-time analytics.

Feature comparison

Features TimescaleDB Apache 2 Edition TimescaleDB Community Edition Hypertables and chunks CREATE TABLE ✅ ✅ create_hypertable ✅ ✅ show_chunks ✅ ✅ drop_chunks ✅ ✅ split_chunk ❌ ✅ reorder_chunk ❌ ✅ move_chunk ❌ ✅ add_reorder_policy ❌ ✅ attach_tablespace ✅ ✅ detach_tablespace() ✅ ✅ detach_tablespaces() ✅ ✅ show_tablespaces ✅ ✅ set_chunk_time_interval ✅ ✅ set_integer_now_func ✅ ✅ add_dimension() ✅ ✅ create_index (Transaction Per Chunk) ✅ ✅ hypertable_size ✅ ✅ hypertable_detailed_size ✅ ✅ hypertable_index_size ✅ ✅ chunks_detailed_size ✅ ✅ SkipScan ❌ ✅ Distributed hypertables: This feature is sunsetted in all editions in TimescaleDB v2.14.x Hypercore Since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0) ALTER TABLE (Hypercore) ❌ ✅ add_columnstore_policy ❌ ✅ remove_columnstore_policy ❌ ✅ convert_to_columnstore ❌ ✅ convert_to_rowstore ❌ ✅ hypertable_columnstore_settings ❌ ✅ hypertable_columnstore_stats ❌ ✅ chunk_columnstore_settings ❌ ✅ chunk_columnstore_stats ❌ ✅ Continuous aggregates CREATE MATERIALIZED VIEW (Continuous Aggregate) ❌ ✅ ALTER MATERIALIZED VIEW (Continuous Aggregate) ❌ ✅ DROP MATERIALIZED VIEW (Continuous Aggregate) ❌ ✅ add_continuous_aggregate_policy() ❌ ✅ refresh_continuous_aggregate ❌ ✅ remove_continuous_aggregate_policy() ❌ ✅ Data retention add_retention_policy ❌ ✅ remove_retention_policy ❌ ✅ Jobs and automation add_job ❌ ✅ alter_job ❌ ✅ delete_job ❌ ✅ run_job ❌ ✅ Hyperfunctions approximate_row_count ✅ ✅ first ✅ ✅ last ✅ ✅ histogram ✅ ✅ time_bucket ✅ ✅ time_bucket_ng (experimental feature) ✅ ✅ time_bucket_gapfill ❌ ✅ locf ❌ ✅ interpolate ❌ ✅ percentile_agg ❌ ✅ approx_percentile ❌ ✅ approx_percentile_rank ❌ ✅ rollup ❌ ✅ max_val ❌ ✅ mean ❌ ✅ error ❌ ✅ min_val ❌ ✅ num_vals ❌ ✅ uddsketch ❌ ✅ tdigest ❌ ✅ time_weight ❌ ✅ rollup ❌ ✅ average ❌ ✅ Informational Views timescaledb_information.chunks ✅ ✅ timescaledb_information.continuous_aggregates ✅ ✅ timescaledb_information.compression_settings ✅ ✅ timescaledb_information.data_nodes ✅ ✅ timescaledb_information.dimension ✅ ✅ timescaledb_information.hypertables ✅ ✅ timescaledb_information.jobs ✅ ✅ timescaledb_information.job_stats ✅ ✅ Administration functions timescaledb_pre_restore ✅ ✅ timescaledb_post_restore ✅ ✅ get_telemetry_report ✅ ✅ dump_meta_data ✅ ✅ Compression Old API since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0) replaced by Hypercore ALTER TABLE (Compression) ❌ ✅ add_compression_policy ❌ ✅ remove_compression_policy ❌ ✅ compress_chunk ❌ ✅ decompress_chunk ❌ ✅ hypertable_compression_stats ❌ ✅ chunk_compression_stats ❌ ✅

===== PAGE: https://docs.tigerdata.com/about/supported-platforms/ =====

Supported platforms

This page lists the platforms and systems that Tiger Data products have been tested on for the following options:

• Tiger Cloud: all the latest features that just work. A reliable and worry-free Postgres cloud for all your workloads.
• Self-hosted products: create your best app from the comfort of your own developer environment.

Tiger Cloud

Tiger Cloud always runs the latest version of all Tiger Data products. With Tiger Cloud you:

• Build everything on one service, and each service hosts one database
• Get faster queries using less compute
• Compress data without sacrificing performance
• View insights on performance, queries, and more
• Reduce storage with automated retention policies

See the available service capabilities and regions.

Available service capabilities

<Availability products={['cloud']} price_plans={['enterprise', 'scale', 'performance']} />

Tiger Cloud services run optimized Tiger Data extensions on latest Postgres, in a highly secure cloud environment. Each service is a specialized database instance tuned for your workload. Available capabilities are:

<thead>
    <tr>
        <th>Capability</th>
        <th>Extensions</th>
    </tr>
</thead>
<tbody>
    <tr>
        <td><strong>Real-time analytics</strong> <p>Lightning-fast ingest and querying of time-based and event data.</p></td>
        <td><ul><li>TimescaleDB</li><li>TimescaleDB Toolkit</li></ul>   </td>
    </tr>
    <tr>
        <td ><strong>AI and vector </strong><p>Seamlessly build RAG, search, and AI agents.</p></td>
        <td><ul><li>TimescaleDB</li><li>pgvector</li><li>pgvectorscale</li><li>pgai</li></ul></td>
    </tr>
    <tr>
        <td ><strong>Hybrid</strong><p>Everything for real-time analytics and AI workloads, combined.</p></td>
        <td><ul><li>TimescaleDB</li><li>TimescaleDB Toolkit</li><li>pgvector</li><li>pgvectorscale</li><li>pgai</li></ul></td>
    </tr>
    <tr>
        <td ><strong>Support</strong></td>
        <td><ul><li>24/7 support no matter where you are.</li><li> Continuous incremental backup/recovery. </li><li>Point-in-time forking/branching.</li><li>Zero-downtime upgrades. </li><li>Multi-AZ high availability. </li><li>An experienced global ops and support team that can build and manage Postgres at scale.</li></ul></td>
    </tr>
</tbody>

Available regions

<Availability products={['cloud']} price_plans={['enterprise', 'scale', 'performance']} />

Tiger Cloud services run in the following Amazon Web Services (AWS) regions:

Region	Zone	Location
ap-south-1	Asia Pacific	Mumbai
ap-southeast-1	Asia Pacific	Singapore
ap-southeast-2	Asia Pacific	Sydney
ap-northeast-1	Asia Pacific	Tokyo
ca-central-1	Canada	Central
eu-central-1	Europe	Frankfurt
eu-west-1	Europe	Ireland
eu-west-2	Europe	London
sa-east-1	South America	São Paulo
us-east-1	United States	North Virginia
us-east-2	United States	Ohio
us-west-2	United States	Oregon

Self-hosted products

You use Tiger Data's open-source products to create your best app from the comfort of your own developer environment.

See the available services and supported systems.

Available services

Tiger Data offers the following services for your self-hosted installations:

<thead>
    <tr>
        <th>Service type</th>
        <th>Description</th>
    </tr>
</thead>
<tbody>
    <tr>
        <td><strong>Self-hosted support</strong></td>
        <td><ul><li>24/7 support no matter where you are.</li><li>An experienced global ops and support team that
        can build and manage Postgres at scale.</li></ul>
        Want to try it out? <a href="https://www.tigerdata.com/self-managed-support">See how we can help</a>.
        </td>
    </tr>
</tbody>

Postgres, TimescaleDB support matrix

TimescaleDB and TimescaleDB Toolkit run on Postgres v10, v11, v12, v13, v14, v15, v16, and v17. Currently Postgres 15 and higher are supported.

TimescaleDB version	Postgres 17	Postgres 16	Postgres 15	Postgres 14	Postgres 13	Postgres 12	Postgres 11	Postgres 10
2.22.x	✅	✅	✅	❌	❌	❌	❌	❌
2.21.x	✅	✅	✅	❌	❌	❌	❌	❌
2.20.x	✅	✅	✅	❌	❌	❌	❌	❌
2.17 - 2.19	✅	✅	✅	✅	❌	❌	❌	❌
2.16.x	❌	✅	✅	✅	❌	❌	❌	❌
2.13 - 2.15	❌	✅	✅	✅	✅	❌	❌	❌
2.12.x	❌	❌	✅	✅	✅	❌	❌	❌
2.10.x	❌	❌	✅	✅	✅	✅	❌	❌
2.5 - 2.9	❌	❌	❌	✅	✅	✅	❌	❌
2.4	❌	❌	❌	❌	✅	✅	❌	❌
2.1 - 2.3	❌	❌	❌	❌	✅	✅	✅	❌
2.0	❌	❌	❌	❌	❌	✅	✅	❌
1.7	❌	❌	❌	❌	❌	✅	✅	✅

We recommend not using TimescaleDB with Postgres 17.1, 16.5, 15.9, 14.14, 13.17, 12.21. These minor versions introduced a breaking binary interface change that, once identified, was reverted in subsequent minor Postgres versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with Postgres 17.2, 16.6, etc and higher. Users of Tiger Cloud and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected.

Supported operating system

You can deploy TimescaleDB and TimescaleDB Toolkit on the following systems:

Operation system	Version
Debian	13 Trixe, 12 Bookworm, 11 Bullseye
Ubuntu	24.04 Noble Numbat, 22.04 LTS Jammy Jellyfish
Red Hat Enterprise	Linux 9, Linux 8
Fedora	Fedora 35, Fedora 34, Fedora 33
Rocky Linux	Rocky Linux 9 (x86_64), Rocky Linux 8
ArchLinux (community-supported)	Check the available packages

Operation system	Version
Microsoft Windows	10, 11
Microsoft Windows Server	2019, 2020

Operation system	Version
macOS	From 10.15 Catalina to 14 Sonoma

===== PAGE: https://docs.tigerdata.com/about/contribute-to-timescale/ =====

Contribute to Tiger Data

TimescaleDB, pgai, pgvectorscale, TimescaleDB Toolkit, and the Tiger Data documentation are all open source. They are available in GitHub for you to use, review, and update. This page shows you where you can add to Tiger Data products.

Contribute to the code for Tiger Data products

Tiger Data appreciates any help the community can provide to make its products better! You can:

• Open an issue with a bug report, build issue, feature request or suggestion.
• Fork a corresponding repository and submit a pull request.

Head over to the Tiger Data source repositories to learn, review, and help improve our products!

• TimescaleDB: a Postgres extension for high-performance real-time analytics on time-series and event data.
• pgai: a suite of tools to develop RAG, semantic search, and other AI applications more easily with Postgres.
• pgvectorscale: a complement to pgvector for higher performance embedding search and cost-efficient storage for AI applications.
• TimescaleDB Toolkit: all things analytics when using TimescaleDB, with a particular focus on developer ergonomics and performance.

Contribute to Tiger Data documentation

Tiger Data documentation is hosted in the docs GitHub repository and open for contribution from all community members.

See the README and contribution guide for details.

===== PAGE: https://docs.tigerdata.com/about/release-notes/ =====

Release notes

For information about new updates and improvement to Tiger Data products, see the Changelog. For release notes about our downloadable products, see:

• TimescaleDB - an open-source database that makes SQL scalable for time-series data, packaged as a Postgres extension.
• TimescaleDB Toolkit - additional functions to ease all things analytics when using TimescaleDB.
• pgai - brings AI workflows to your Postgres database.
• pgvectorscale - higher performance embedding search and cost-efficient storage for AI applications on Postgres.
• pgspot - spot vulnerabilities in Postgres extension scripts.
• live-migration - a Docker image to migrate data to a Tiger Cloud service.

Want to stay up-to-date with new releases? On the main page for each repository click Watch, select Custom and then check Releases.

===== PAGE: https://docs.tigerdata.com/migrate/livesync-for-postgresql/ =====

Sync data from Postgres to your service

You use the source Postgres connector in Tiger Cloud to synchronize all data or specific tables from a Postgres database instance to your service, in real time. You run the connector continuously, turning Postgres into a primary database with your service as a logical replica. This enables you to leverage Tiger Cloud’s real-time analytics capabilities on your replica data.

The source Postgres connector in Tiger Cloud leverages the well-established Postgres logical replication protocol. By relying on this protocol, Tiger Cloud ensures compatibility, familiarity, and a broader knowledge base—making it easier for you to adopt the connector and integrate your data.

You use the source Postgres connector for data synchronization, rather than migration. This includes:

• Copy existing data from a Postgres instance to a Tiger Cloud service:

  • Copy data at up to 150 GB/hr.

    You need at least a 4 CPU/16 GB source database, and a 4 CPU/16 GB target service.

  • Copy the publication tables in parallel.

    Large tables are still copied using a single connection. Parallel copying is in the backlog.

  • Forget foreign key relationships.

    The connector disables foreign key validation during the sync. For example, if a metrics table refers to the id column on the tags table, you can still sync only the metrics table without worrying about their foreign key relationships.

  • Track progress.

    Postgres exposes COPY progress under pg_stat_progress_copy.

• Synchronize real-time changes from a Postgres instance to a Tiger Cloud service.

• Add and remove tables on demand using the Postgres PUBLICATION interface.

• Enable features such as hypertables, columnstore, and continuous aggregates on your logical replica.

Early access: this source Postgres connector is not yet supported for production use. If you have any questions or feedback, talk to us in #livesync in the Tiger Community.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

  You need your connection details.

• Install the Postgres client tools on your sync machine.

• Ensure that the source Postgres instance and the target Tiger Cloud service have the same extensions installed.

  The source Postgres connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table.

Limitations

• The source Postgres instance must be accessible from the Internet.

  Services hosted behind a firewall or VPC are not supported. This functionality is on the roadmap.

• Indexes, including the primary key and unique constraints, are not migrated to the target Tiger Cloud service.

  We recommend that, depending on your query patterns, you create only the necessary indexes on the target Tiger Cloud service.

• This works for Postgres databases only as source. TimescaleDB is not yet supported.

• The source must be running Postgres 13 or later.

• Schema changes must be co-ordinated.

  Make compatible changes to the schema in your Tiger Cloud service first, then make the same changes to the source Postgres instance.

• Ensure that the source Postgres instance and the target Tiger Cloud service have the same extensions installed.

  The source Postgres connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table.

• There is WAL volume growth on the source Postgres instance during large table copy.

• Continuous aggregate invalidation

  The connector uses session_replication_role=replica during data replication, which prevents table triggers from firing. This includes the internal triggers that mark continuous aggregates as invalid when underlying data changes.

  If you have continuous aggregates on your target database, they do not automatically refresh for data inserted during the migration. This limitation only applies to data below the continuous aggregate's materialization watermark. For example, backfilled data. New rows synced above the continuous aggregate watermark are used correctly when refreshing.

  This can lead to:

  • Missing data in continuous aggregates for the migration period.
  • Stale aggregate data.
  • Queries returning incomplete results.

  If the continuous aggregate exists in the source database, best practice is to add it to the Postgres connector publication. If it only exists on the target database, manually refresh the continuous aggregate using the force option of refresh_continuous_aggregate.

Set your connection string

This variable holds the connection information for the source database. In the terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Tune your source database

Updating parameters on a Postgres instance will cause an outage. Choose a time that will cause the least issues to tune this database.

1. Tune the Write Ahead Log (WAL) on the RDS/Aurora Postgres source database

   1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

   2. Click Configuration, scroll down and note the DB instance parameter group, then click Parameter Groups

      

   3. Click Create parameter group, fill in the form with the following values, then click Create.

      • Parameter group name - whatever suits your fancy.
      • Description - knock yourself out with this one.
      • Engine type - PostgreSQL
      • Parameter group family - the same as DB instance parameter group in your Configuration.

   4. In Parameter groups, select the parameter group you created, then click Edit.

   5. Update the following parameters, then click Save changes.

      • rds.logical_replication set to 1: record the information needed for logical decoding.
      • wal_sender_timeout set to 0: disable the timeout for the sender process.

   6. In RDS, navigate back to your databases, select the RDS instance to migrate, and click Modify.

   7. Scroll down to Database options, select your new parameter group, and click Continue.

   8. Click Apply immediately or choose a maintenance window, then click Modify DB instance.

      Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.

   9. Verify that the settings are live in your database.

2. Create a user for the source Postgres connector and assign permissions

   1. Create <pg connector username>:

      psql source -c "CREATE USER <pg connector username> PASSWORD '<password>'"
      

      You can use an existing user. However, you must ensure that the user has the following permissions.

   2. Grant permissions to create a replication slot:

      psql source -c "GRANT rds_replication TO <pg connector username>"
      

   3. Grant permissions to create a publication:

      psql source -c "GRANT CREATE ON DATABASE <database name> TO <pg connector username>"
      

   4. Assign the user permissions on the source database:

      psql source <<EOF
      GRANT USAGE ON SCHEMA "public" TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

      If the tables you are syncing are not in the public schema, grant the user permissions for each schema you are syncing:

      psql source <<EOF
      GRANT USAGE ON SCHEMA <schema> TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA <schema> GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

   5. On each table you want to sync, make <pg connector username> the owner:

      psql source -c 'ALTER TABLE  OWNER TO <pg connector username>;'
      

      You can skip this step if the replicating user is already the owner of the tables.

3. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

1. Tune the Write Ahead Log (WAL) on the Postgres source database

   psql source <<EOF
   ALTER SYSTEM SET wal_level='logical';
   ALTER SYSTEM SET max_wal_senders=10;
   ALTER SYSTEM SET wal_sender_timeout=0;
   EOF
   

   • GUC “wal_level” as “logical”
   • GUC “max_wal_senders” as 10
   • GUC “wal_sender_timeout” as 0

   This will require a restart of the Postgres source database.

2. Create a user for the connector and assign permissions

   1. Create <pg connector username>:

      psql source -c "CREATE USER <pg connector username> PASSWORD '<password>'"
      

      You can use an existing user. However, you must ensure that the user has the following permissions.

   2. Grant permissions to create a replication slot:

      psql source -c "ALTER ROLE <pg connector username> REPLICATION"
      

   3. Grant permissions to create a publication:

      psql source -c "GRANT CREATE ON DATABASE <database name> TO <pg connector username>"
      

   4. Assign the user permissions on the source database:

      psql source <<EOF
      GRANT USAGE ON SCHEMA "public" TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

      If the tables you are syncing are not in the public schema, grant the user permissions for each schema you are syncing:

      psql source <<EOF
      GRANT USAGE ON SCHEMA <schema> TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA <schema> GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

   5. On each table you want to sync, make <pg connector username> the owner:

      psql source -c 'ALTER TABLE  OWNER TO <pg connector username>;'
      

      You can skip this step if the replicating user is already the owner of the tables.

3. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

Synchronize data to your Tiger Cloud service

To sync data from your Postgres database to your Tiger Cloud service using Tiger Cloud Console:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console, select the service to sync live data to.

2. Connect the source database and the target service

   

   1. Click Connectors > PostgreSQL.
   2. Set the name for the new connector by clicking the pencil icon.
   3. Check the boxes for Set wal_level to logical and Update your credentials, then click Continue.
   4. Enter your database credentials or a Postgres connection string, then click Connect to database. This is the connection string for <pg connector username>. Tiger Cloud Console connects to the source database and retrieves the schema information.

3. Optimize the data to synchronize in hypertables

   

   1. In the Select table dropdown, select the tables to sync.

   2. Click Select tables + .

      Tiger Cloud Console checks the table schema and, if possible, suggests the column to use as the time dimension in a hypertable.

   3. Click Create Connector.

      Tiger Cloud Console starts source Postgres connector between the source database and the target service and displays the progress.

4. Monitor synchronization

   

   1. To view the amount of data replicated, click Connectors. The diagram in Connector data flow gives you an overview of the connectors you have created, their status, and how much data has been replicated.

   2. To review the syncing progress for each table, click Connectors > Source connectors, then select the name of your connector in the table.

5. Manage the connector

   

   1. To edit the connector, click Connectors > Source connectors, then select the name of your connector in the table. You can rename the connector, delete or add new tables for syncing.

   2. To pause a connector, click Connectors > Source connectors, then open the three-dot menu on the right and select Pause.

   3. To delete a connector, click Connectors > Source connectors, then open the three-dot menu on the right and select Delete. You must pause the connector before deleting it.

And that is it, you are using the source Postgres connector to synchronize all the data, or specific tables, from a Postgres database instance to your Tiger Cloud service, in real time.

Prerequisites

Best practice is to use an Ubuntu EC2 instance hosted in the same region as your Tiger Cloud service to move data. That is, the machine you run the commands on to move your data from your source database to your target Tiger Cloud service.

Before you move your data:

• Create a target Tiger Cloud service.

  Each Tiger Cloud service has a single Postgres instance that supports the most popular extensions. Tiger Cloud services do not support tablespaces, and there is no superuser associated with a service. Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window.

• To ensure that maintenance does not run while migration is in progress, best practice is to adjust the maintenance window.

• Ensure that the source Postgres instance and the target Tiger Cloud service have the same extensions installed.

  The source Postgres connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table.

• Install Docker on your sync machine.

  For a better experience, use a 4 CPU/16GB EC2 instance or greater to run the source Postgres connector.

• Install the Postgres client tools on your sync machine.

  This includes psql, pg_dump, pg_dumpall, and vacuumdb commands.

Limitations

• The schema is not migrated by the source Postgres connector, you use pg_dump/pg_restore to migrate it.

• This works for Postgres databases only as source. TimescaleDB is not yet supported.

• The source must be running Postgres 13 or later.

• Schema changes must be co-ordinated.

  Make compatible changes to the schema in your Tiger Cloud service first, then make the same changes to the source Postgres instance.

• Ensure that the source Postgres instance and the target Tiger Cloud service have the same extensions installed.

  The source Postgres connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table.

• There is WAL volume growth on the source Postgres instance during large table copy.

• Continuous aggregate invalidation

  The connector uses session_replication_role=replica during data replication, which prevents table triggers from firing. This includes the internal triggers that mark continuous aggregates as invalid when underlying data changes.

  If you have continuous aggregates on your target database, they do not automatically refresh for data inserted during the migration. This limitation only applies to data below the continuous aggregate's materialization watermark. For example, backfilled data. New rows synced above the continuous aggregate watermark are used correctly when refreshing.

  This can lead to:

  • Missing data in continuous aggregates for the migration period.
  • Stale aggregate data.
  • Queries returning incomplete results.

  If the continuous aggregate exists in the source database, best practice is to add it to the Postgres connector publication. If it only exists on the target database, manually refresh the continuous aggregate using the force option of refresh_continuous_aggregate.

Set your connection strings

The <user> in the SOURCE connection must have the replication role granted in order to create a replication slot.

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Tune your source database

Updating parameters on a Postgres instance will cause an outage. Choose a time that will cause the least issues to tune this database.

1. Update the DB instance parameter group for your source database

   1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

   2. Click Configuration, scroll down and note the DB instance parameter group, then click Parameter groups

      

   3. Click Create parameter group, fill in the form with the following values, then click Create.

      • Parameter group name - whatever suits your fancy.
      • Description - knock yourself out with this one.
      • Engine type - PostgreSQL
      • Parameter group family - the same as DB instance parameter group in your Configuration.

   4. In Parameter groups, select the parameter group you created, then click Edit.

   5. Update the following parameters, then click Save changes.

      • rds.logical_replication set to 1: record the information needed for logical decoding.
      • wal_sender_timeout set to 0: disable the timeout for the sender process.

   6. In RDS, navigate back to your databases, select the RDS instance to migrate, and click Modify.

   7. Scroll down to Database options, select your new parameter group, and click Continue.

   8. Click Apply immediately or choose a maintenance window, then click Modify DB instance.

      Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.

   9. Verify that the settings are live in your database.

2. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

1. Tune the Write Ahead Log (WAL) on the Postgres source database

   psql source <<EOF
   ALTER SYSTEM SET wal_level='logical';
   ALTER SYSTEM SET max_wal_senders=10;
   ALTER SYSTEM SET wal_sender_timeout=0;
   EOF
   

   • GUC “wal_level” as “logical”
   • GUC “max_wal_senders” as 10
   • GUC “wal_sender_timeout” as 0

   This will require a restart of the Postgres source database.

2. Create a user for the connector and assign permissions

   1. Create <pg connector username>:

      psql source -c "CREATE USER <pg connector username> PASSWORD '<password>'"
      

      You can use an existing user. However, you must ensure that the user has the following permissions.

   2. Grant permissions to create a replication slot:

      psql source -c "ALTER ROLE <pg connector username> REPLICATION"
      

   3. Grant permissions to create a publication:

      psql source -c "GRANT CREATE ON DATABASE <database name> TO <pg connector username>"
      

   4. Assign the user permissions on the source database:

      psql source <<EOF
      GRANT USAGE ON SCHEMA "public" TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

      If the tables you are syncing are not in the public schema, grant the user permissions for each schema you are syncing:

      psql source <<EOF
      GRANT USAGE ON SCHEMA <schema> TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA <schema> GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

   5. On each table you want to sync, make <pg connector username> the owner:

      psql source -c 'ALTER TABLE  OWNER TO <pg connector username>;'
      

      You can skip this step if the replicating user is already the owner of the tables.

3. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

Migrate the table schema to the Tiger Cloud service

Use pg_dump to:

1. Download the schema from the source database

pg_dump source \
--no-privileges \
--no-owner \
--no-publications \
--no-subscriptions \
--no-table-access-method \
--no-tablespaces \
--schema-only \
--file=schema.sql

1. Apply the schema on the target service

psql target -f schema.sql

Convert partitions and tables with time-series data into hypertables

For efficient querying and analysis, you can convert tables which contain time-series or events data, and tables that are already partitioned using Postgres declarative partition into hypertables.

1. Convert tables to hypertables

   Run the following on each table in the target Tiger Cloud service to convert it to a hypertable:

   psql -X -d target -c "SELECT public.create_hypertable('', by_range('<partition column>', '<chunk interval>'::interval));"
   

   For example, to convert the metrics table into a hypertable with time as a partition column and 1 day as a partition interval:

   psql -X -d target -c "SELECT public.create_hypertable('public.metrics', by_range('time', '1 day'::interval));"
   

2. Convert Postgres partitions to hypertables

   Rename the partition and create a new regular table with the same name as the partitioned table, then convert to a hypertable:

   psql target -f - <<'EOF'
      BEGIN;
      ALTER TABLE public.events RENAME TO events_part;
      CREATE TABLE public.events(LIKE public.events_part INCLUDING ALL);
      SELECT create_hypertable('public.events', by_range('time', '1 day'::interval));
      COMMIT;
   

EOF

## Specify the tables to synchronize

After the schema is migrated, you [`CREATE PUBLICATION`][create-publication] on the source database that
specifies the tables to synchronize.

1. **Create a publication that specifies the table to synchronize**

A `PUBLICATION` enables you to synchronize some or all the tables in the schema or database.

```sql
CREATE PUBLICATION <publication_name> FOR TABLE , ;

To add tables after to an existing publication, use [ALTER PUBLICATION][alter-publication]**

ALTER PUBLICATION <publication_name> ADD TABLE ;

1. Publish the Postgres declarative partitioned table

   ALTER PUBLICATION <publication_name> SET(publish_via_partition_root=true);
   

   To convert partitioned table to hypertable, follow Convert partitions and tables with time-series data into hypertables.

2. Stop syncing a table in the PUBLICATION, use DROP TABLE

   ALTER PUBLICATION <publication_name> DROP TABLE ;
   

Synchronize data to your Tiger Cloud service

You use the source Postgres connector docker image to synchronize changes in real time from a Postgres database instance to a Tiger Cloud service:

1. Start the source Postgres connector

   As you run the source Postgres connector continuously, best practice is to run it as a Docker daemon.

   docker run -d --rm --name livesync timescale/live-sync:v0.1.25 run \
      --publication <publication_name> --subscription <subscription_name> \
      --source source --target target --table-map
   

   --publication: The name of the publication as you created in the previous step. To use multiple publications, repeat the --publication flag.

   --subscription: The name that identifies the subscription on the target Tiger Cloud service.

   --source: The connection string to the source Postgres database.

   --target: The connection string to the target Tiger Cloud service.

   --table-map: (Optional) A JSON string that maps source tables to target tables. If not provided, the source and target table names are assumed to be the same. For example, to map the source table metrics to the target table metrics_data:

   --table-map '{"source": {"schema": "public", "table": "metrics"}, "target": {"schema": "public", "table": "metrics_data"}}'
   

   To map only the schema, use:

   --table-map '{"source": {"schema": "public"}, "target": {"schema": "analytics"}}'
   

   This flag can be repeated for multiple table mappings.

2. Capture logs

   Once the source Postgres connector is running as a docker daemon, you can also capture the logs:

   docker logs -f livesync
   

3. View the progress of tables being synchronized

   List the tables being synchronized by the source Postgres connector using the _ts_live_sync.subscription_rel table in the target Tiger Cloud service:

   psql target -c "SELECT * FROM _ts_live_sync.subscription_rel"
   

   You see something like the following:

   subname	pubname	schemaname	tablename	rrelid	state	lsn	updated_at	last_error	created_at	rows_copied	approximate_rows	bytes_copied	approximate_size	target_schema	target_table
   livesync	analytics	public	metrics	20856	r	6/1A8CBA48	2025-06-24 06:16:21.434898+00		2025-06-24 06:03:58.172946+00	18225440	18225440	1387359359	1387359359	public	metrics

   The state column indicates the current state of the table synchronization. Possible values for state are:

   state	description
   d	initial table data sync
   f	initial table data sync completed
   s	catching up with the latest changes
   r	table is ready, syncing live changes

   To see the replication lag, run the following against the SOURCE database:

   psql source -f - <<'EOF'
   SELECT
      slot_name,
      pg_size_pretty(pg_current_wal_flush_lsn() - confirmed_flush_lsn) AS lag
   FROM pg_replication_slots
   WHERE slot_name LIKE 'live_sync_%' AND slot_type = 'logical'
   

EOF

1. **Add or remove tables from the publication**

To add tables, use [ALTER PUBLICATION .. ADD TABLE][alter-publication]**

```sql
ALTER PUBLICATION <publication_name> ADD TABLE ;

To remove tables, use ALTER PUBLICATION .. DROP TABLE**

ALTER PUBLICATION <publication_name> DROP TABLE ;

1. Update table statistics

   If you have a large table, you can run ANALYZE on the target Tiger Cloud service to update the table statistics after the initial sync is complete.

   This helps the query planner make better decisions for query execution plans.

   vacuumdb --analyze --verbose --dbname=target
   

2. Stop the source Postgres connector

   docker stop live-sync
   

3. (Optional) Reset sequence nextval on the target Tiger Cloud service

   The source Postgres connector does not automatically reset the sequence nextval on the target Tiger Cloud service.

   Run the following script to reset the sequence for all tables that have a serial or identity column in the target Tiger Cloud service:

   psql target -f - <<'EOF'
      DO $$
   DECLARE
     rec RECORD;
   BEGIN
     FOR rec IN (
       SELECT
         sr.target_schema  AS table_schema,
         sr.target_table   AS table_name,
         col.column_name,
         pg_get_serial_sequence(
           sr.target_schema || '.' || sr.target_table,
           col.column_name
         ) AS seqname
       FROM _ts_live_sync.subscription_rel AS sr
       JOIN information_schema.columns AS col
         ON col.table_schema = sr.target_schema
        AND col.table_name   = sr.target_table
       WHERE col.column_default LIKE 'nextval(%'  -- only serial/identity columns
     ) LOOP
       EXECUTE format(
         'SELECT setval(%L,
            COALESCE((SELECT MAX(%I) FROM %I.%I), 0) + 1,
            false
          );',
         rec.seqname,       -- the sequence identifier
         rec.column_name,   -- the column to MAX()
         rec.table_schema,  -- schema for MAX()
         rec.table_name     -- table for MAX()
       );
     END LOOP;
   END;
   $$ LANGUAGE plpgsql;
   

EOF

1. **Clean up**

Use the `--drop` flag to remove the replication slots created by the source Postgres connector on the source database.

```shell
docker run -it --rm --name livesync timescale/live-sync:v0.1.25 run \
   --publication <publication_name> --subscription <subscription_name> \
   --source source --target target \
   --drop

===== PAGE: https://docs.tigerdata.com/migrate/livesync-for-s3/ =====

Sync data from S3 to your service

You use the source S3 connector in Tiger Cloud to synchronize CSV and Parquet files from an S3 bucket to your Tiger Cloud service in real time. The connector runs continuously, enabling you to leverage Tiger Cloud as your analytics database with data constantly synced from S3. This lets you take full advantage of Tiger Cloud's real-time analytics capabilities without having to develop or manage custom ETL solutions between S3 and Tiger Cloud.

You can use the source S3 connector to synchronize your existing and new data. Here's what the connector can do:

• Sync data from an S3 bucket instance to a Tiger Cloud service:

  • Use glob patterns to identify the objects to sync.
  • Watch an S3 bucket for new files and import them automatically. It runs on a configurable schedule and tracks processed files.
  • Important: The connector processes files in lexicographical order. It uses the name of the last file processed as a marker and fetches only files later in the alphabet in subsequent queries. Files added with names earlier in the alphabet than the marker are skipped and never synced. For example, if you add the file Bob when the marker is at Elephant, Bob is never processed.
  • For large backlogs, check every minute until caught up.

• Sync data from multiple file formats:

  • CSV: check for compression in GZ and ZIP format, then process using timescaledb-parallel-copy.
  • Parquet: convert to CSV, then process using timescaledb-parallel-copy.

• The source S3 connector offers an option to enable a hypertable during the file-to-table schema mapping setup. You can enable columnstore and continuous aggregates through the SQL editor once the connector has started running.

• The connector offers a default 1-minute polling interval. This means that Tiger Cloud checks the S3 source every minute for new data. You can customize this interval by setting up a cron expression.

The source S3 connector continuously imports data from an Amazon S3 bucket into your database. It monitors your S3 bucket for new files matching a specified pattern and automatically imports them into your designated database table.

Note: the connector currently only syncs existing and new files—it does not support updating or deleting records based on updates and deletes from S3 to tables in a Tiger Cloud service.

Early access: this source S3 connector is not supported for production use. If you have any questions or feedback, talk to us in #livesync in the Tiger Community.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

  You need your connection details.

• Ensure access to a standard Amazon S3 bucket containing your data files.

  Directory buckets are not supported.

• Configure access credentials for the S3 bucket. The following credentials are supported:

  • IAM Role.

    • Configure the trust policy. Set the:

      • Principal: arn:aws:iam::142548018081:role/timescale-s3-connections.

      • ExternalID: set to the Tiger Cloud project and Tiger Cloud service ID of the service you are syncing to in the format <projectId>/<serviceId>.

        This is to avoid the confused deputy problem.

    • Give the following access permissions:

      • s3:GetObject.
      • s3:ListBucket.

  • Public anonymous user.

Limitations

• File naming: Files must follow lexicographical ordering conventions. Files with names that sort earlier than already-processed files are permanently skipped. Example: if file_2024_01_15.csv has been processed, a file named file_2024_01_10.csv added later will never be synced. Recommended naming patterns: timestamps (for example, YYYY-MM-DD-HHMMSS), sequential numbers with fixed padding (for example, file_00001, file_00002).

• CSV:

  • Maximum file size: 1 GB

    To increase this limit, contact sales@tigerdata.com

  • Maximum row size: 2 MB

  • Supported compressed formats:

    • GZ
    • ZIP

  • Advanced settings:

    • Delimiter: the default character is ,, you can choose a different delimiter
    • Skip header: skip the first row if your file has headers

• Parquet:

  • Maximum file size: 1 GB
  • Maximum row size: 2 MB

• Sync iteration:

  To prevent system overload, the connector tracks up to 100 files for each sync iteration. Additional checks only fill empty queue slots.

Synchronize data to your Tiger Cloud service

To sync data from your S3 bucket to your Tiger Cloud service using Tiger Cloud Console:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console, select the service to sync live data to.

2. Connect the source S3 bucket to the target service

   

   1. Click Connectors > Amazon S3.

   2. Click the pencil icon, then set the name for the new connector.

   3. Set the Bucket name and Authentication method, then click Continue.

      For instruction on creating the IAM role to connect your S3 bucket, click Learn how. Tiger Cloud Console connects to the source bucket.

   4. In Define files to sync, choose the File type and set the Glob pattern.

      Use the following patterns:

      • <folder name>/*: match all files in a folder. Also, any pattern ending with / is treated as /*.
      • <folder name>/**: match all recursively.
      • <folder name>/**/*.csv: match a specific file type.

      The source S3 connector uses prefix filters where possible, place patterns carefully at the end of your glob expression. AWS S3 doesn't support complex filtering. If your expression filters too many files, the list operation may time out.

   5. Click the search icon. You see the files to sync. Click Continue.

3. Optimize the data to synchronize in hypertables

   

   Tiger Cloud Console checks the file schema and, if possible, suggests the column to use as the time dimension in a hypertable.

   1. Choose Create a new table for your data or Ingest data to an existing table.

   2. Choose the Data type for each column, then click Continue.

   3. Choose the interval. This can be a minute, an hour, or use a cron expression.

   4. Click Start Connector.

      Tiger Cloud Console starts the connection between the source database and the target service and displays the progress.

4. Monitor synchronization

   1. To view the amount of data replicated, click Connectors. The diagram in Connector data flow gives you an overview of the connectors you have created, their status, and how much data has been replicated.

      

   2. To view file import statistics and logs, click Connectors > Source connectors, then select the name of your connector in the table.

      

5. Manage the connector

   1. To pause the connector, click Connectors > Source connectors. Open the three-dot menu next to your connector in the table, then click Pause.

   

   1. To edit the connector, click Connectors > Source connectors. Open the three-dot menu next to your connector in the table, then click Edit and scroll down to Modify your Connector. You must pause the connector before editing it.

   

   1. To pause or delete the connector, click Connectors > Source connectors, then open the three-dot menu on the right and select an option. You must pause the connector before deleting it.

And that is it, you are using the source S3 connector to synchronize all the data, or specific files, from an S3 bucket to your Tiger Cloud service in real time.

===== PAGE: https://docs.tigerdata.com/migrate/livesync-for-kafka/ =====

Stream data from Kafka into your service

You use the Kafka source connector in Tiger Cloud to stream events from Kafka into your service. Tiger Cloud connects to your Confluent Cloud Kafka cluster and Schema Registry using SASL/SCRAM authentication and service account–based API keys. Only the Avro format is currently supported with some limitations.

This page explains how to connect Tiger Cloud to your Confluence Cloud Kafka cluster.

Early access: the Kafka source connector is not yet supported for production use.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

  You need your connection details.

• Sign up for Confluence Cloud.
• Create a Kafka cluster in Confluence Cloud.

Access your Kafka cluster in Confluent Cloud

Take the following steps to prepare your Kafka cluster for connection to Tiger Cloud:

1. Create a service account

   If you already have a service account for Tiger Cloud, you can reuse it.  To create a new service account:
   
   1. Log in to [Confluent Cloud][confluent-cloud].
   1. Click the burger menu at the top-right of the pane, then press
      `Access control` > `Service accounts` >`Add service account`.
   1. Enter the following details:
   
      - Name: `tigerdata-access`
      - Description: `Service account for the Tiger Cloud source connector`
   
   1. Add the service account owner role, then click `Next`.
   
   1. Select a role assignment, then click `Add`
   
   1. Click `Next`, then click `Create service account`.
   

   1. Create API keys

      1. In Confluent Cloud, click Home > Environments > Select your environment > Select your cluster.
      2. Under Cluster overview in the left sidebar, select API Keys.
      3. Click Add key, choose Service Account and click Next.
      4. Select tigerdata-access, then click Next.
      5. For your cluster, choose the Operation and select the following Permissions, then click Next:
         • Resource type: Cluster
         • Operation: DESCRIBE
         • Permission: ALLOW
      6. Click Download and continue, then securely store the ACL.
      7. Use the same procedure to add the following keys:
         • ACL 2: Topic access
           • Resource type: Topic
           • Topic name: Select the topics that Tiger Cloud should read
           • Pattern type: LITERAL
           • Operation: READ
           • Permission: ALLOW
         • ACL 3: Consumer group access
           • Resource type: Consumer group
           • Consumer group ID: tigerdata-kafka/<tiger_cloud_project_id>. See Find your connection details for where to find your project ID
           • Pattern type: PREFIXED
           • Operation: READ
           • Permission: ALLOW You need these to configure your Kafka source connector in Tiger Cloud.

Configure Confluent Cloud Schema Registry

Tiger Cloud requires access to the Schema Registry to fetch schemas for Kafka topics. To configure the Schema Registry:

1. Navigate to Schema Registry

   In Confluent Cloud, click Environments and select your environment, then click Stream Governance.

   1. Create a Schema Registry API key

      1. Click API Keys, then click Add API Key.

      2. Choose Service Account, select tigerdata-access, then click Next.

      3. Under Resource scope, choose Schema Registry, select the default environment, then click Next.

      4. In Create API Key, add the following, then click Create API Key :

         • Name: tigerdata-schema-registry-access
         • Description: API key for Tiger Cloud schema registry access

      5. Click Download API Key and securely store the API key and secret, then click Complete.

   2. Assign roles for Schema Registry

      1. Click the burger menu at the top-right of the pane, then press Access control > Accounts & access > Service accounts.

      2. Select the tigerdata-access service account.

      3. In the Access tab, add the following role assignments for All schema subjects:

         • ResourceOwner on the service account.

         • DeveloperRead on schema subjects.

           Choose All schema subjects or restrict to specific subjects as required.

      4. Save the role assignments.

Your Confluent Cloud Schema Registry is now accessible to Tiger Cloud using the API key and secret.

Add Kafka source connector in Tiger Cloud

Take the following steps to create a Kafka source connector in Tiger Cloud Console.

1. In Console, select your service

2. Go to Connectors > Source connectors. Click New Connector, then select Kafka

3. Click the pencil icon, then set the connector name

4. Set up Kafka authentication

   Enter the name of your cluster in Confluent Cloud and the information from the first api-key-*.txt that you downloaded, then click Authenticate.

5. Set up the Schema Registry

   Enter the service account ID and the information from the second api-key-*.txt that you downloaded, then click Authenticate.

6. Select topics to sync

   Add the schema and table, map the columns in the table, and click Create connector.

Your Kafka connector is configured and ready to stream events.

Known limitations and unsupported types

The following Avro schema types are not supported:

Union types

Multi-type non-nullable unions are blocked.

Examples:

• Multiple type union:

  {
    "type": "record",
    "name": "Message",
    "fields": [
      {"name": "content", "type": ["string", "bytes", "null"]}
    ]
  }
  

• Union as root schema:

  ["null", "string"]
  

Reference types (named type references)

Referencing a previously defined named type by name, instead of inline, is not supported.

Examples:

• Named type definition:

  {
    "type": "record",
    "name": "Address",
    "fields": [
      {"name": "street", "type": "string"},
      {"name": "city", "type": "string"}
    ]
  }
  

• Failing reference:

  {
    "type": "record",
    "name": "Person",
    "fields": [
      {"name": "name", "type": "string"},
      {"name": "address", "type": "Address"}
    ]
  }
  

Unsupported logical types

Only the logical types in the hardcoded supported list are supported. This includes:

• decimal, date, time-millis, time-micros

• timestamp-millis, timestamp-micros, timestamp-nanos

• local-timestamp-millis, local-timestamp-micros, local-timestamp-nanos

• uuid, duration

Unsupported examples:

{
  "type": "int",
  "logicalType": "date-time"
}

{
  "type": "string",
  "logicalType": "json"
}

{
  "type": "bytes",
  "logicalType": "custom-type"
}

===== PAGE: https://docs.tigerdata.com/migrate/upload-file-using-console/ =====

Upload a file into your service using Tiger Cloud Console

You can upload files into your service using Tiger Cloud Console. This page explains how to upload CSV, Parquet, and text files, from your local machine and from an S3 bucket.

Tiger Cloud Console enables you to drag and drop files to upload from your local machine.

Early access

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

To upload a CSV file to your service:

1. Select your service in Console, then click Actions > Import data > Upload your files > Upload CSV file

   

2. Click to browse, or drag the file to import

3. Configure the import

   

   • Set a delimiter.
   • Toggle to skip or keep the header.
   • Select to ingest the data into an existing table or create a new one.
   • Provide the new or existing table name.
   • For a new table with a time column, toggle the time column to create a hypertable instead of a regular table.

4. Click Process CSV file

   When the processing is completed, to find the data your imported, click Explorer.

To upload a Parquet file to your service:

1. Select your service in Console, then click Actions > Import data > Upload your files > Upload Parquet file

   

2. Click to browse, or drag the file to import

3. Configure the import

   

   • Select to ingest the data into an existing table or create a new one.
   • Provide the new or existing table name.
   • For a new table with a time column, toggle the time column to create a hypertable instead of a regular table.

4. Click Process Parquet file

   When the processing is completed, to find the data your imported, click Explorer.

To upload a TXT or MD file to your service:

1. Select your service in Console, then click Actions > Import data > Upload your files > Upload Text file

   

2. Click to browse, or drag and drop the file to import

3. Configure the import

   Provide a name to create a new table, or select an existing table to add data to.

   

4. Click Upload files

   When the upload is finished, find your data imported to a new or existing table in Explorer.

Tiger Cloud Console enables you to upload CSV and Parquet files, including archives compressed using GZIP and ZIP, by connecting to an S3 bucket.

This feature is not available under the Free pricing plan.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

• Ensure access to a standard Amazon S3 bucket containing your data files.
• Configure access credentials for the S3 bucket. The following credentials are supported:
  • IAM Role.
  • Public anonymous user.

To import a CSV file from an S3 bucket:

1. Select your service in Console, then click Actions > Import data > Explore import options > Import from S3

2. Select your file in the S3 bucket

   

   1. Provide your file path.
   2. Select CSV in the file type dropdown.
   3. Select the authentication method:
      • IAM role and provide the role.
      • Public.
   4. Click Continue.

3. Configure the import

   

   • Set a delimiter.
   • Toggle to skip or keep the header.
   • Select to ingest the data into an existing table or create a new one.
   • Provide the new or existing table name.
   • For a new table with a time column, toggle the time column to create a hypertable instead of a regular table.

4. Click Process CSV file

   When the processing is completed, to find the data your imported, click Explorer.

To import a Parquet file from an S3 bucket:

1. Select your service in Console, then click Actions > Import from S3

2. Select your file in the S3 bucket

   

   1. Provide your file path.
   2. Select Parquet in the file type dropdown.
   3. Select the authentication method:
   • IAM role and provide the role.
   • Public.
   1. Click Continue.

3. Configure the import

   • Select Create a new table for your data or Ingest data to an existing table.
   • Provide the new or existing table name.
   • For a new table with a time column, toggle the time column to create a hypertable instead of a regular table.

4. Click Process Parquet file

   When the processing is completed, to find the data your imported, click Explorer.

And that is it, you have imported your data to your Tiger Cloud service.

===== PAGE: https://docs.tigerdata.com/migrate/upload-file-using-terminal/ =====

Upload a file into your service using the terminal

This page shows you how to upload CSV, MySQL, and Parquet files from a source machine into your service using the terminal.

The CSV file format is widely used for data migration. This page shows you how to import data into your Tiger Cloud service from a CSV file using the terminal.

Prerequisites

To follow the procedure on this page you need to:

• Create a target Tiger Cloud service.

  This procedure also works for self-hosted TimescaleDB.

• Install Go v1.13 or later

• Install timescaledb-parallel-copy

  timescaledb-parallel-copy improves performance for large datasets by parallelizing the import process. It also preserves row order and uses a round-robin approach to optimize memory management and disk operations.

  To verify your installation, run timescaledb-parallel-copy --version.

• Ensure that the time column in the CSV file uses the TIMESTAMPZ data type.

For faster data transfer, best practice is that your target service and the system running the data import are in the same region.

Import data into your service

To import data from a CSV file:

1. Set up your service connection string

   This variable holds the connection information for the target Tiger Cloud service.

In the terminal on the source machine, set the following:

export TARGET=postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require

See where to find your connection details.

1. Create a hypertable to hold your data

   Create a hypertable with a schema that is compatible with the data in your parquet file. For example, if your parquet file contains the columns ts, location, and temperature with typesTIMESTAMP, STRING, and DOUBLE:

   • TimescaleDB v2.20 and above:

     psql target -c "CREATE TABLE  ( \
     ts          TIMESTAMPTZ         NOT NULL, \
     location    TEXT                NOT NULL, \
     temperature DOUBLE PRECISION    NULL \
     ) WITH (timescaledb.hypertable, timescaledb.partition_column = 'ts');"
     
     

   • TimescaleDB v2.19.3 and below:

     1. Create a new regular table:

        psql target -c  "CREATE TABLE  ( \
           ts          TIMESTAMPTZ         NOT NULL,  \
           location    TEXT                NOT NULL,  \
           temperature DOUBLE PRECISION    NULL  \
        );"
        

     2. Convert the empty table to a hypertable:

        In the following command, replace with the name of the table you just created, and `<COLUMN_NAME>` with the partitioning column in.

        psql target -c  "SELECT create_hypertable('', by_range('<COLUMN_NAME>'))"
        

2. Import your data

   In the folder containing your CSV files, either:

   • Use timescaledb-parallel-copy:

       timescaledb-parallel-copy \
       --connection target \
       --table  \
       --file <FILE_NAME>.csv \
       --workers <NUM_WORKERS> \
       --reporting-period 30s
     

     For the best performances while avoiding resource competition, set <NUM_WORKERS> to twice the number of CPUs in your service, but less than the available CPU cores.

     For self-hosted TimescaleDB, set target to host=localhost user=postgres sslmode=disable

   • Use psql:

     psql target
     \c <DATABASE_NAME>
     \COPY  FROM <FILENAME>.csv CSV"
     

     psql COPY is single-threaded, and may be slower for large datasets.

3. Verify the data was imported correctly into your service

And that is it, you have imported your data from a CSV file.

MySQL is an open-source relational database management system (RDBMS). This page shows you how to import data into your Tiger Cloud service from a database running on MySQL version 8 or earlier.

Prerequisites

To follow the procedure on this page you need to:

• Create a target Tiger Cloud service.

  This procedure also works for self-hosted TimescaleDB.

• Install Docker on your migration machine.

  This machine needs sufficient space to store the buffered changes that occur while your data is being copied. This space is proportional to the amount of new uncompressed data being written to the Tiger Cloud service during migration. A general rule of thumb is between 100GB and 500GB.

For faster data transfer, best practice is for your source database, target service, and the system running the data import are in the same region .

Import data into your service

To import data from a MySQL database:

1. Set up the connection string for your target service

   This variable holds the connection information for the target Tiger Cloud service.

In the terminal on the source machine, set the following:

export TARGET=postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require

See where to find your connection details.

1. Set up the connection string for your source database

   SOURCE="mysql://<mysql_username>:<mysql_password>@<mysql_host>:<mysql_port>/<mysql_database>?sslmode=require"
   

   where:

   • <mysql_username>: your MySQL username
   • <mysql_password>: your MySQL password
   • <mysql_host>: the MySQL server hostname or IP address
   • <mysql_port>: the MySQL server port, the default is 3306
   • <mysql_database>: the name of your MySQL database

2. Import your data

   On your data import machine, run the following command:

   docker run -it ghcr.io/dimitri/pgloader:latest pgloader
   --no-ssl-cert-verification \
   "source" \
   "target"
   

3. Verify the data was imported correctly into your service

And that is it, you have imported your data from MySQL.

Apache Parquet is a free and open-source column-oriented data storage format in the Apache Hadoop ecosystem. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk. This page shows you how to import data into your Tiger Cloud service from a Parquet file.

Prerequisites

To follow the procedure on this page you need to:

• Create a target Tiger Cloud service.

  This procedure also works for self-hosted TimescaleDB.

• Install DuckDB on the source machine where the Parquet file is located.
• Ensure that the time column in the Parquet file uses the TIMESTAMP data type.

For faster data transfer, best practice is that your target service and the system running the data import are in the same region.

Import data into your service

To import data from a Parquet file:

1. Set up your service connection string

   This variable holds the connection information for the target Tiger Cloud service.

In the terminal on the source machine, set the following:

export TARGET=postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require

See where to find your connection details.

1. Create a hypertable to hold your data

   Create a hypertable with a schema that is compatible with the data in your parquet file. For example, if your parquet file contains the columns ts, location, and temperature with typesTIMESTAMP, STRING, and DOUBLE:

   • TimescaleDB v2.20 and above:

     psql target -c "CREATE TABLE  ( \
     ts          TIMESTAMPTZ         NOT NULL, \
     location    TEXT                NOT NULL, \
     temperature DOUBLE PRECISION    NULL \
     ) WITH (timescaledb.hypertable, timescaledb.partition_column = 'ts');"
     
     

   • TimescaleDB v2.19.3 and below:

     1. Create a new regular table:

        psql target -c  "CREATE TABLE  ( \
           ts          TIMESTAMPTZ         NOT NULL,  \
           location    TEXT                NOT NULL,  \
           temperature DOUBLE PRECISION    NULL  \
        );"
        

     2. Convert the empty table to a hypertable:

        In the following command, replace with the name of the table you just created, and `<COLUMN_NAME>` with the partitioning column in.

        psql target -c  "SELECT create_hypertable('', by_range('<COLUMN_NAME>'))"
        

2. Set up a DuckDB connection to your service

   1. In a terminal on the source machine with your Parquet files, start a new DuckDB interactive session:

      duckdb
      

   2. Connect to your service in your DuckDB session:

      ATTACH '<Paste the value of target here' AS db (type postgres);
      

      target is the connection string you used to connect to your service using psql.

3. Import data from Parquet to your service

   1. In DuckDB, upload the table data to your service

      COPY db. FROM '<FILENAME>.parquet' (FORMAT parquet);
      

      Where:

      • ``: the hypertable you created to import data to
      • <FILENAME>: the Parquet file to import data from

   2. Exit the DuckDB session:

      EXIT;
      

4. Verify the data was imported correctly into your service

   In your psql session, view the data in ``:

   SELECT * FROM ;
   

And that is it, you have imported your data from a Parquet file to your Tiger Cloud service.

===== PAGE: https://docs.tigerdata.com/migrate/pg-dump-and-restore/ =====

Migrate with downtime

You use downtime migration to move less than 100GB of data from a self-hosted database to a Tiger Cloud service.

Downtime migration uses the native Postgres pg_dump and pg_restore commands. If you are migrating from self-hosted TimescaleDB, this method works for hypertables compressed into the columnstore without having to convert the data back to the rowstore before you begin.

If you want to migrate more than 400GB of data, create a Tiger Cloud Console support request, or send us an email at support@tigerdata.com saying how much data you want to migrate. We pre-provision your Tiger Cloud service for you.

However, downtime migration for large amounts of data takes a large amount of time. For more than 100GB of data, best practice is to follow live migration.

This page shows you how to move your data from a self-hosted database to a Tiger Cloud service using shell commands.

Prerequisites

Best practice is to use an Ubuntu EC2 instance hosted in the same region as your Tiger Cloud service to move data. That is, the machine you run the commands on to move your data from your source database to your target Tiger Cloud service.

Before you move your data:

• Create a target Tiger Cloud service.

  Each Tiger Cloud service has a single Postgres instance that supports the most popular extensions. Tiger Cloud services do not support tablespaces, and there is no superuser associated with a service. Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window.

• To ensure that maintenance does not run while migration is in progress, best practice is to adjust the maintenance window.

• Install the Postgres client tools on your migration machine.

  This includes psql, pg_dump, and pg_dumpall.

• Install the GNU implementation of sed.

  Run sed --version on your migration machine. GNU sed identifies itself as GNU software, BSD sed returns sed: illegal option -- -.

Migrate to Tiger Cloud

To move your data from a self-hosted database to a Tiger Cloud service:

This section shows you how to move your data from self-hosted TimescaleDB to a Tiger Cloud service using pg_dump and psql from Terminal.

Prepare to migrate

1. Take the applications that connect to the source database offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss.

2. Set your connection strings

   These variables hold the connection information for the source database and target Tiger Cloud service:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

   You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Align the version of TimescaleDB on the source and target

1. Ensure that the source and target databases are running the same version of TimescaleDB.

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target service:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

2. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Migrate the roles from TimescaleDB to your Tiger Cloud service

Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service:

1. Dump the roles from your source database

   Export your role-based security hierarchy. <db_name> has the same value as <db_name> in source. I know, it confuses me as well.

   pg_dumpall -d "source" \
     -l <db_name>
     --quote-all-identifiers \
     --roles-only \
     --file=roles.sql
   

   If you only use the default postgres role, this step is not necessary.

2. Remove roles with superuser access

   Tiger Cloud service do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e '/CREATE ROLE "tsdbadmin";/d' \
   -e '/ALTER ROLE "tsdbadmin"/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

3. Dump the source database schema and data

   The pg_dump flags remove superuser access and tablespaces from your data. When you run pgdump, check the run time, a long-running pg_dump can cause issues.

   pg_dump -d "source" \
   --format=plain \
   --quote-all-identifiers \
   --no-tablespaces \
   --no-owner \
   --no-privileges \
   --file=dump.sql
   

   To dramatically reduce the time taken to dump the source database, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

Upload your data to the target Tiger Cloud service

This command uses the timescaledb_pre_restore and timescaledb_post_restore functions to put your database in the correct state.

psql target -v ON_ERROR_STOP=1 --echo-errors \
-f roles.sql \
-c "SELECT timescaledb_pre_restore();" \
-f dump.sql \
-c "SELECT timescaledb_post_restore();"

Validate your Tiger Cloud service and restart your app

1. Update the table statistics.

   psql target -c "ANALYZE;"
   

2. Verify the data in the target Tiger Cloud service.

   Check that your data is correct, and returns the results that you expect,

3. Enable any Tiger Cloud features you want to use.

   Migration from Postgres moves the data only. Now manually enable Tiger Cloud features like hypertables, hypercore or data retention while your database is offline.

4. Reconfigure your app to use the target database, then restart it.

And that is it, you have migrated your data from a self-hosted instance running TimescaleDB to a Tiger Cloud service.

This section shows you how to move your data from self-hosted Postgres to a Tiger Cloud service using pg_dump and psql from Terminal.

Migration from Postgres moves the data only. You must manually enable Tiger Cloud features like hypertables, hypercore or data retention after the migration is complete. You enable Tiger Cloud features while your database is offline.

Prepare to migrate

1. Take the applications that connect to the source database offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss.

2. Set your connection strings

   These variables hold the connection information for the source database and target Tiger Cloud service:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

   You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Align the extensions on the source and target

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Migrate the roles from TimescaleDB to your Tiger Cloud service

Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service:

1. Dump the roles from your source database

   Export your role-based security hierarchy. <db_name> has the same value as <db_name> in source. I know, it confuses me as well.

   pg_dumpall -d "source" \
     -l <db_name>
     --quote-all-identifiers \
     --roles-only \
     --file=roles.sql
   

   If you only use the default postgres role, this step is not necessary.

2. Remove roles with superuser access

   Tiger Cloud service do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e '/CREATE ROLE "tsdbadmin";/d' \
   -e '/ALTER ROLE "tsdbadmin"/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

3. Dump the source database schema and data

   The pg_dump flags remove superuser access and tablespaces from your data. When you run pgdump, check the run time, a long-running pg_dump can cause issues.

   pg_dump -d "source" \
   --format=plain \
   --quote-all-identifiers \
   --no-tablespaces \
   --no-owner \
   --no-privileges \
   --file=dump.sql
   

   To dramatically reduce the time taken to dump the source database, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

Upload your data to the target Tiger Cloud service

psql target -v ON_ERROR_STOP=1 --echo-errors \
-f roles.sql \
-f dump.sql

Validate your Tiger Cloud service and restart your app

1. Update the table statistics.

   psql target -c "ANALYZE;"
   

2. Verify the data in the target Tiger Cloud service.

   Check that your data is correct, and returns the results that you expect,

3. Enable any Tiger Cloud features you want to use.

   Migration from Postgres moves the data only. Now manually enable Tiger Cloud features like hypertables, hypercore or data retention while your database is offline.

4. Reconfigure your app to use the target database, then restart it.

And that is it, you have migrated your data from a self-hosted instance running Postgres to a Tiger Cloud service.

To migrate your data from an Amazon RDS/Aurora Postgres instance to a Tiger Cloud service, you extract the data to an intermediary EC2 Ubuntu instance in the same AWS region as your RDS/Aurora Postgres instance. You then upload your data to a Tiger Cloud service. To make this process as painless as possible, ensure that the intermediary machine has enough CPU and disk space to rapidLy extract and store your data before uploading to Tiger Cloud.

Migration from RDS/Aurora Postgres moves the data only. You must manually enable Tiger Cloud features like hypertables, data compression or data retention after the migration is complete. You enable Tiger Cloud features while your database is offline.

This section shows you how to move your data from a Postgres database running in an Amazon RDS/Aurora Postgres instance to a Tiger Cloud service using pg_dump and psql from Terminal.

Create an intermediary EC2 Ubuntu instance

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.
2. Click Actions > Set up EC2 connection. Press Create EC2 instance and use the following settings:
   • AMI: Ubuntu Server.
   • Key pair: use an existing pair or create a new one that you will use to access the intermediary machine.
   • VPC: by default, this is the same as the database instance.
   • Configure Storage: adjust the volume to at least the size of RDS/Aurora Postgres instance you are migrating from. You can reduce the space used by your data on Tiger Cloud using Hypercore.
3. Click Lauch instance. AWS creates your EC2 instance, then click Connect to instance > SSH client. Follow the instructions to create the connection to your intermediary EC2 instance.

Install the psql client tools on the intermediary instance

1. Connect to your intermediary EC2 instance. For example:

   ssh -i "<key-pair>.pem" ubuntu@<EC2 instance's Public IPv4>
   

2. On your intermediary EC2 instance, install the Postgres client.

   sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
   wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo tee /etc/apt/trusted.gpg.d/pgdg.asc &>/dev/null
   sudo apt update
   sudo apt install postgresql-client-16 -y # "postgresql-client-16" if your source DB is using PG 16.
   psql --version && pg_dump --version
   

Keep this terminal open, you need it to connect to the RDS/Aurora Postgres instance for migration.

Set up secure connectivity between your RDS/Aurora Postgres and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.

2. Scroll down to Security group rules (1) and select the EC2 Security Group - Inbound group. The Security Groups (1) window opens. Click the Security group ID, then click Edit inbound rules

   

3. On your intermediary EC2 instance, get your local IP address:

   ec2metadata --local-ipv4
   

   Bear with me on this one, you need this IP address to enable access to your RDS/Aurora Postgres instance.

4. In Edit inbound rules, click Add rule, then create a PostgreSQL, TCP rule granting access to the local IP address for your EC2 instance (told you :-)). Then click Save rules.

   

Test the connection between your RDS/Aurora Postgres and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.

2. On your intermediary EC2 instance, use the values of Endpoint, Port, Master username, and DB name to create the postgres connectivity string to the SOURCE variable.

   

   export SOURCE="postgres://<Master username>:<Master password>@<Endpoint>:<Port>/<DB name>"
   

   The value of Master password was supplied when this RDS/Aurora Postgres instance was created.

3. Test your connection:

   psql -d source
   

   You are connected to your RDS/Aurora Postgres instance from your intermediary EC2 instance.

Migrate your data to your Tiger Cloud service

To securely migrate data from your RDS instance:

Prepare to migrate

1. Take the applications that connect to the RDS instance offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss. You should also ensure that your source RDS instance is not receiving any DML queries.

2. Connect to your intermediary EC2 instance

   For example:

   ssh -i "<key-pair>.pem" ubuntu@<EC2 instance's Public IPv4>
   

3. Set your connection strings

   These variables hold the connection information for the RDS instance and target Tiger Cloud service:

   export SOURCE="postgres://<Master username>:<Master password>@<Endpoint>:<Port>/<DB name>"
   export TARGET=postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require
   

   You find the connection information for SOURCE in your RDS configuration. For TARGET in the configuration file you downloaded when you created the Tiger Cloud service.

Align the extensions on the source and target

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Migrate roles from RDS to your Tiger Cloud service

Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service:

1. Dump the roles from your RDS instance

   Export your role-based security hierarchy. If you only use the default postgres role, this step is not necessary.

   pg_dumpall -d "source" \
     --quote-all-identifiers \
     --roles-only \
     --no-role-passwords \
     --file=roles.sql
   

   AWS RDS does not allow you to export passwords with roles. You assign passwords to these roles when you have uploaded them to your Tiger Cloud service.

2. Remove roles with superuser access

   Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e '/CREATE ROLE "rds/d' \
   -e '/ALTER ROLE "rds/d' \
   -e '/TO "rds/d' \
   -e '/GRANT "rds/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

3. Upload the roles to your Tiger Cloud service

   psql -X -d "target" \
     -v ON_ERROR_STOP=1 \
     --echo-errors \
     -f roles.sql
   

4. Manually assign passwords to the roles

   AWS RDS did not allow you to export passwords with roles. For each role, use the following command to manually assign a password to a role:

    psql target -c "ALTER ROLE <role name> WITH PASSWORD '<highly secure password>';"
   

Migrate data from your RDS instance to your Tiger Cloud service

1. Dump the data from your RDS instance to your intermediary EC2 instance

   The pg_dump flags remove superuser access and tablespaces from your data. When you run pgdump, check the run time, a long-running pg_dump can cause issues.

   pg_dump -d "source" \
   --format=plain \
   --quote-all-identifiers \
   --no-tablespaces \
   --no-owner \
   --no-privileges \
   --file=dump.sql
   

   To dramatically reduce the time taken to dump the RDS instance, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

2. Upload your data to your Tiger Cloud service

   psql -d target -v ON_ERROR_STOP=1 --echo-errors \
     -f dump.sql
   

Validate your Tiger Cloud service and restart your app

1. Update the table statistics.

   psql target -c "ANALYZE;"
   

2. Verify the data in the target Tiger Cloud service.

   Check that your data is correct, and returns the results that you expect,

3. Enable any Tiger Cloud features you want to use.

   Migration from Postgres moves the data only. Now manually enable Tiger Cloud features like hypertables, hypercore or data retention while your database is offline.

4. Reconfigure your app to use the target database, then restart it.

And that is it, you have migrated your data from an RDS/Aurora Postgres instance to a Tiger Cloud service.

This section shows you how to move your data from a Managed Service for TimescaleDB instance to a Tiger Cloud service using pg_dump and psql from Terminal.

Prepare to migrate

1. Take the applications that connect to the source database offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss.

2. Set your connection strings

   These variables hold the connection information for the source database and target Tiger Cloud service:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

   You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Align the version of TimescaleDB on the source and target

1. Ensure that the source and target databases are running the same version of TimescaleDB.

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target service:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

2. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Migrate the roles from TimescaleDB to your Tiger Cloud service

Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service:

1. Dump the roles from your source database

   Export your role-based security hierarchy. <db_name> has the same value as <db_name> in source. I know, it confuses me as well.

   pg_dumpall -d "source" \
     -l <db_name>  \
     --quote-all-identifiers \
     --roles-only \
     --no-role-passwords \
     --file=roles.sql
   

   MST does not allow you to export passwords with roles. You assign passwords to these roles when you have uploaded them to your Tiger Cloud service.

2. Remove roles with superuser access

   Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   

-e '/DROP ROLE IF EXISTS "postgres";/d'
-e '/DROP ROLE IF EXISTS "tsdbadmin";/d'
-e '/CREATE ROLE "postgres";/d'
-e '/ALTER ROLE "postgres"/d'
-e '/CREATE ROLE "rds/d'
-e '/ALTER ROLE "rds/d'
-e '/TO "rds/d'
-e '/GRANT "rds/d'
-e '/GRANT "pg_read_all_stats" TO "tsdbadmin"/d'
-e 's/(NO)SUPERUSER//g'
-e 's/(NO)REPLICATION//g'
-e 's/(NO)BYPASSRLS//g'
-e 's/GRANTED BY "[^"]"//g'
-e '/CREATE ROLE "tsdbadmin";/d'
-e '/ALTER ROLE "tsdbadmin"/d'
-e 's/WITH ADMIN OPTION,/WITH /g'
-e 's/WITH ADMIN OPTION//g'
-e 's/GRANTED BY "."//g'
-e '/GRANT "pg_." TO/d'
-e '/CREATE ROLE "_aiven";/d'
-e '/ALTER ROLE "_aiven"/d'
-e '/GRANT SET ON PARAMETER "pgaudit.[^"]+" TO "_tsdbadmin_auditing"/d'
-e '/GRANT SET ON PARAMETER "anon.[^"]+" TO "tsdbadmin_group"/d'
roles.sql

1. **Dump the source database schema and data**

The `pg_dump` flags remove superuser access and tablespaces from your data. When you run
`pgdump`, check the run time, [a long-running `pg_dump` can cause issues][long-running-pgdump].

```bash
pg_dump -d "source" \
--format=plain \
--quote-all-identifiers \
--no-tablespaces \
--no-owner \
--no-privileges \
--file=dump.sql

To dramatically reduce the time taken to dump the source database, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

Upload your data to the target Tiger Cloud service

This command uses the timescaledb_pre_restore and timescaledb_post_restore functions to put your database in the correct state.

1. Upload your data

   psql target -v ON_ERROR_STOP=1 --echo-errors \
   -f roles.sql \
   -c "SELECT timescaledb_pre_restore();" \
   -f dump.sql \
   -c "SELECT timescaledb_post_restore();"
   

2. Manually assign passwords to the roles

   MST did not allow you to export passwords with roles. For each role, use the following command to manually assign a password to a role:

    psql target -c "ALTER ROLE <role name> WITH PASSWORD '<highly secure password>';"
   

Validate your Tiger Cloud service and restart your app

1. Update the table statistics.

   psql target -c "ANALYZE;"
   

2. Verify the data in the target Tiger Cloud service.

   Check that your data is correct, and returns the results that you expect,

3. Enable any Tiger Cloud features you want to use.

   Migration from Postgres moves the data only. Now manually enable Tiger Cloud features like hypertables, hypercore or data retention while your database is offline.

4. Reconfigure your app to use the target database, then restart it.

And that is it, you have migrated your data from a Managed Service for TimescaleDB instance to a Tiger Cloud service.

===== PAGE: https://docs.tigerdata.com/migrate/live-migration/ =====

Live migration

Live migration is an end-to-end solution that copies the database schema and data to your target Tiger Cloud service, then replicates the database activity in your source database to the target service in real time. Live migration uses the Postgres logical decoding functionality and leverages pgcopydb.

You use the live migration Docker image to move 100GB-10TB+ of data to a Tiger Cloud service seamlessly with only a few minutes downtime.

If you want to migrate more than 400GB of data, create a Tiger Cloud Console support request, or send us an email at support@tigerdata.com saying how much data you want to migrate. We pre-provision your Tiger Cloud service for you.

Best practice is to use live migration when:

• Modifying your application logic to perform dual writes is a significant effort.

• The insert workload does not exceed 20,000 rows per second, and inserts are batched.

  Use Dual write and backfill for greater workloads.

• Your source database:

  • Uses UPDATE and DELETE statements on uncompressed time-series data.

    Live-migration does not support replicating INSERT/UPDATE/DELETE statements on compressed data.

  • Has large, busy tables with primary keys.

  • Does not have many UPDATE or DELETE statements.

This page shows you how to move your data from a self-hosted database to a Tiger Cloud service using the live-migration Docker image.

Prerequisites

Best practice is to use an Ubuntu EC2 instance hosted in the same region as your Tiger Cloud service to move data. That is, the machine you run the commands on to move your data from your source database to your target Tiger Cloud service.

Before you move your data:

• Create a target Tiger Cloud service.

  Each Tiger Cloud service has a single Postgres instance that supports the most popular extensions. Tiger Cloud services do not support tablespaces, and there is no superuser associated with a service. Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window.

• To ensure that maintenance does not run while migration is in progress, best practice is to adjust the maintenance window.

• Install Docker on your migration machine.

  This machine needs sufficient space to store the buffered changes that occur while your data is being copied. This space is proportional to the amount of new uncompressed data being written to the Tiger Cloud service during migration. A general rule of thumb is between 100GB and 500GB. The CPU specifications of this EC2 instance should match those of your Tiger Cloud service for optimal performance. For example, if your service has an 8-CPU configuration, then your EC2 instance should also have 8 CPUs.

• Before starting live migration, read the Frequently Asked Questions.

Migrate to Tiger Cloud

To move your data from a self-hosted database to a Tiger Cloud service:

This section shows you how to move your data from self-hosted TimescaleDB to a Tiger Cloud service using live migration from Terminal.

Set your connection strings

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Align the version of TimescaleDB on the source and target

1. Ensure that the source and target databases are running the same version of TimescaleDB.

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target service:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

2. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Tune your source database

You need admin rights to to update the configuration on your source database. If you are using a managed service, follow the instructions in the From MST tab on this page.

1. Install the wal2json extension on your source database

   Install wal2json on your source database.

2. Prevent Postgres from treating the data in a snapshot as outdated

   psql -X -d source -c 'alter system set old_snapshot_threshold=-1'
   

   This is not applicable if the source database is Postgres 17 or later.

3. Set the write-Ahead Log (WAL) to record the information needed for logical decoding

   psql -X -d source -c 'alter system set wal_level=logical'
   

4. Restart the source database

   Your configuration changes are now active. However, verify that the settings are live in your database.

5. Enable live-migration to replicate DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

Migrate your data, then start downtime

2. Pull the live-migration docker image to you migration machine

   sudo docker pull timescale/live-migration:latest
   

   To list the available commands, run:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest --help
   

   To see the available flags for each command, run --help for that command. For example:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest migrate --help
   

3. Create a snapshot image of your source database in your Tiger Cloud service

   This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine:

   docker run --rm -it --name live-migration-snapshot \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest snapshot
   

   Live-migration supplies information about updates you need to make to the source database and target service. For example:

   2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX)
   2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB
   2024-03-25T12:40:40.884 WARNING:        - public.metrics
   

   If you have warnings, stop live-migration, make the suggested changes and start again.

4. Synchronize data between your source database and your Tiger Cloud service

   This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target.

   docker run --rm -it --name live-migration-migrate \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest migrate
   

   If the source Postgres version is 17 or later, you need to pass additional flag -e PGVERSION=17 to the migrate command.

   During this process, you see the migration process:

   Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB)
   

   If migrate stops add --resume to start from where it left off.

   Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message:

   Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB)
       To stop replication, hit 'c' and then ENTER
   

   Wait until replay_lag is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished.

5. Start app downtime

   1. Stop your app writing to the source database, then let the the remaining transactions finish to fully sync with the target. You can use tools like the pg_top CLI or pg_stat_activity to view the current transaction on the source database.

   2. Stop Live-migration.

      hit 'c' and then ENTER
      

      Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message:

      Migration successfully completed
      

Validate your data, then restart your app

1. Validate the migrated data

   The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app.

2. Stop app downtime

   Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service.

3. Cleanup resources associated with live-migration from your migration machine

   This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration.

   docker run --rm -it --name live-migration-clean \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest clean --prune
   

This section shows you how to move your data from self-hosted Postgres to a Tiger Cloud service using live migration from Terminal.

Set your connection strings

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Align the extensions on the source and target

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Tune your source database

You need admin rights to to update the configuration on your source database. If you are using a managed service, follow the instructions in the From AWS RDS/Aurora tab on this page.

1. Install the wal2json extension on your source database

   Install wal2json on your source database.

2. Prevent Postgres from treating the data in a snapshot as outdated

   psql -X -d source -c 'alter system set old_snapshot_threshold=-1'
   

   This is not applicable if the source database is Postgres 17 or later.

3. Set the write-Ahead Log (WAL) to record the information needed for logical decoding

   psql -X -d source -c 'alter system set wal_level=logical'
   

4. Restart the source database

   Your configuration changes are now active. However, verify that the settings are live in your database.

5. Enable live-migration to replicate DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

Migrate your data, then start downtime

1. Pull the live-migration docker image to you migration machine

   sudo docker pull timescale/live-migration:latest
   

   To list the available commands, run:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest --help
   

   To see the available flags for each command, run --help for that command. For example:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest migrate --help
   

2. Create a snapshot image of your source database in your Tiger Cloud service

   This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine:

   docker run --rm -it --name live-migration-snapshot \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest snapshot
   

   Live-migration supplies information about updates you need to make to the source database and target service. For example:

   2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX)
   2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB
   2024-03-25T12:40:40.884 WARNING:        - public.metrics
   

   If you have warnings, stop live-migration, make the suggested changes and start again.

3. Synchronize data between your source database and your Tiger Cloud service

   This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target.

   docker run --rm -it --name live-migration-migrate \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest migrate
   

   If the source Postgres version is 17 or later, you need to pass additional flag -e PGVERSION=17 to the migrate command.

   After migrating the schema, live-migration prompts you to create hypertables for tables that contain time-series data in your Tiger Cloud service. Run create_hypertable() to convert these table. For more information, see the Hypertable docs.

   During this process, you see the migration process:

   Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB)
   

   If migrate stops add --resume to start from where it left off.

   Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message:

   Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB)
       To stop replication, hit 'c' and then ENTER
   

   Wait until replay_lag is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished.

4. Start app downtime

   1. Stop your app writing to the source database, then let the the remaining transactions finish to fully sync with the target. You can use tools like the pg_top CLI or pg_stat_activity to view the current transaction on the source database.

   2. Stop Live-migration.

      hit 'c' and then ENTER
      

      Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message:

      Migration successfully completed
      

Validate your data, then restart your app

1. Validate the migrated data

   The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app.

2. Stop app downtime

   Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service.

3. Cleanup resources associated with live-migration from your migration machine

   This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration.

   docker run --rm -it --name live-migration-clean \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest clean --prune
   

To migrate your data from an Amazon RDS/Aurora Postgres instance to a Tiger Cloud service, you extract the data to an intermediary EC2 Ubuntu instance in the same AWS region as your RDS/Aurora instance. You then upload your data to a Tiger Cloud service. To make this process as painless as possible, ensure that the intermediary machine has enough CPU and disk space to rapidly extract and store your data before uploading to Tiger Cloud.

Migration from RDS/Aurora gives you the opportunity to create hypertables before copying the data. Once the migration is complete, you can manually enable Tiger Cloud features like data compression or data retention.

This section shows you how to move your data from an Amazon RDS/Aurora instance to a Tiger Cloud service using live migration.

Create an intermediary EC2 Ubuntu instance

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.
2. Click Actions > Set up EC2 connection. Press Create EC2 instance and use the following settings:
   • AMI: Ubuntu Server.
   • Key pair: use an existing pair or create a new one that you will use to access the intermediary machine.
   • VPC: by default, this is the same as the database instance.
   • Configure Storage: adjust the volume to at least the size of RDS/Aurora Postgres instance you are migrating from. You can reduce the space used by your data on Tiger Cloud using Hypercore.
3. Click Lauch instance. AWS creates your EC2 instance, then click Connect to instance > SSH client. Follow the instructions to create the connection to your intermediary EC2 instance.

Install the psql client tools on the intermediary instance

1. Connect to your intermediary EC2 instance. For example:

   ssh -i "<key-pair>.pem" ubuntu@<EC2 instance's Public IPv4>
   

2. On your intermediary EC2 instance, install the Postgres client.

   sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
   wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo tee /etc/apt/trusted.gpg.d/pgdg.asc &>/dev/null
   sudo apt update
   sudo apt install postgresql-client-16 -y # "postgresql-client-16" if your source DB is using PG 16.
   psql --version && pg_dump --version
   

Keep this terminal open, you need it to connect to the RDS/Aurora Postgres instance for migration.

Set up secure connectivity between your RDS/Aurora Postgres and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.

2. Scroll down to Security group rules (1) and select the EC2 Security Group - Inbound group. The Security Groups (1) window opens. Click the Security group ID, then click Edit inbound rules

   

3. On your intermediary EC2 instance, get your local IP address:

   ec2metadata --local-ipv4
   

   Bear with me on this one, you need this IP address to enable access to your RDS/Aurora Postgres instance.

4. In Edit inbound rules, click Add rule, then create a PostgreSQL, TCP rule granting access to the local IP address for your EC2 instance (told you :-)). Then click Save rules.

   

Test the connection between your RDS/Aurora Postgres and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.

2. On your intermediary EC2 instance, use the values of Endpoint, Port, Master username, and DB name to create the postgres connectivity string to the SOURCE variable.

   

   export SOURCE="postgres://<Master username>:<Master password>@<Endpoint>:<Port>/<DB name>"
   

   The value of Master password was supplied when this RDS/Aurora Postgres instance was created.

3. Test your connection:

   psql -d source
   

   You are connected to your RDS/Aurora Postgres instance from your intermediary EC2 instance.

Set your connection strings

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Align the extensions on the source and target

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Tune your source database

Updating parameters on a Postgres instance will cause an outage. Choose a time that will cause the least issues to tune this database.

1. Update the DB instance parameter group for your source database

   1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

   2. Click Configuration, scroll down and note the DB instance parameter group, then click Parameter groups

      

   3. Click Create parameter group, fill in the form with the following values, then click Create.

      • Parameter group name - whatever suits your fancy.
      • Description - knock yourself out with this one.
      • Engine type - PostgreSQL
      • Parameter group family - the same as DB instance parameter group in your Configuration.

   4. In Parameter groups, select the parameter group you created, then click Edit.

   5. Update the following parameters, then click Save changes.

      • rds.logical_replication set to 1: record the information needed for logical decoding.
      • wal_sender_timeout set to 0: disable the timeout for the sender process.

   6. In RDS, navigate back to your databases, select the RDS instance to migrate, and click Modify.

   7. Scroll down to Database options, select your new parameter group, and click Continue.

   8. Click Apply immediately or choose a maintenance window, then click Modify DB instance.

      Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.

   9. Verify that the settings are live in your database.

2. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

Migrate your data, then start downtime

1. Pull the live-migration docker image to you migration machine

   sudo docker pull timescale/live-migration:latest
   

   To list the available commands, run:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest --help
   

   To see the available flags for each command, run --help for that command. For example:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest migrate --help
   

2. Create a snapshot image of your source database in your Tiger Cloud service

   This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine:

   docker run --rm -it --name live-migration-snapshot \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest snapshot
   

   Live-migration supplies information about updates you need to make to the source database and target service. For example:

   2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX)
   2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB
   2024-03-25T12:40:40.884 WARNING:        - public.metrics
   

   If you have warnings, stop live-migration, make the suggested changes and start again.

3. Synchronize data between your source database and your Tiger Cloud service

   This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target.

   docker run --rm -it --name live-migration-migrate \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest migrate
   

   If the source Postgres version is 17 or later, you need to pass additional flag -e PGVERSION=17 to the migrate command.

   After migrating the schema, live-migration prompts you to create hypertables for tables that contain time-series data in your Tiger Cloud service. Run create_hypertable() to convert these table. For more information, see the Hypertable docs.

   During this process, you see the migration process:

   Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB)
   

   If migrate stops add --resume to start from where it left off.

   Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message:

   Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB)
       To stop replication, hit 'c' and then ENTER
   

   Wait until replay_lag is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished.

4. Start app downtime

   1. Stop your app writing to the source database, then let the the remaining transactions finish to fully sync with the target. You can use tools like the pg_top CLI or pg_stat_activity to view the current transaction on the source database.

   2. Stop Live-migration.

      hit 'c' and then ENTER
      

      Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message:

      Migration successfully completed
      

Validate your data, then restart your app

1. Validate the migrated data

   The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app.

2. Stop app downtime

   Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service.

3. Cleanup resources associated with live-migration from your migration machine

   This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration.

   docker run --rm -it --name live-migration-clean \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest clean --prune
   

This section shows you how to move your data from a MST instance to a Tiger Cloud service using live migration from Terminal.

Set your connection strings

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Align the version of TimescaleDB on the source and target

1. Ensure that the source and target databases are running the same version of TimescaleDB.

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target service:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

2. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Tune your source database

1. Enable live-migration to replicate DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

Migrate your data, then start downtime

2. Pull the live-migration docker image to you migration machine

   sudo docker pull timescale/live-migration:latest
   

   To list the available commands, run:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest --help
   

   To see the available flags for each command, run --help for that command. For example:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest migrate --help
   

3. Create a snapshot image of your source database in your Tiger Cloud service

   This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine:

   docker run --rm -it --name live-migration-snapshot \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest snapshot
   

   Live-migration supplies information about updates you need to make to the source database and target service. For example:

   2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX)
   2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB
   2024-03-25T12:40:40.884 WARNING:        - public.metrics
   

   If you have warnings, stop live-migration, make the suggested changes and start again.

4. Synchronize data between your source database and your Tiger Cloud service

   This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target.

   docker run --rm -it --name live-migration-migrate \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest migrate
   

   If the source Postgres version is 17 or later, you need to pass additional flag -e PGVERSION=17 to the migrate command.

   During this process, you see the migration process:

   Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB)
   

   If migrate stops add --resume to start from where it left off.

   Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message:

   Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB)
       To stop replication, hit 'c' and then ENTER
   

   Wait until replay_lag is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished.

5. Start app downtime

   1. Stop your app writing to the source database, then let the the remaining transactions finish to fully sync with the target. You can use tools like the pg_top CLI or pg_stat_activity to view the current transaction on the source database.

   2. Stop Live-migration.

      hit 'c' and then ENTER
      

      Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message:

      Migration successfully completed
      

Validate your data, then restart your app

1. Validate the migrated data

   The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app.

2. Stop app downtime

   Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service.

3. Cleanup resources associated with live-migration from your migration machine

   This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration.

   docker run --rm -it --name live-migration-clean \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest clean --prune
   

And you are done, your data is now in your Tiger Cloud service.

Troubleshooting

This section shows you how to work around frequently seen issues when using live migration.

ERROR: relation "xxx.yy" does not exist

This may happen when a relation is removed after executing the snapshot command. A relation can be a table, index, view, or materialized view. When you see you this error:

• Do not perform any explicit DDL operation on the source database during the course of migration.

• If you are migrating from self-hosted TimescaleDB or MST, disable the chunk retention policy on your source database until you have finished migration.

FATAL: remaining connection slots are reserved for non-replication superuser connections

This may happen when the number of connections exhaust max_connections defined in your target Tiger Cloud service. By default, live-migration needs around ~6 connections on the source and ~12 connections on the target.

Migration seems to be stuck with “x GB copied to Target DB (Source DB is y GB)”

When you are migrating a lot of data involved in aggregation, or there are many materialized views taking time to complete the materialization, this may be due to REFRESH MATERIALIZED VIEWS happening at the end of initial data migration.

To resolve this issue:

1. See what is happening on the target Tiger Cloud service:

   psql target -c "select * from pg_stat_activity where application_name ilike '%pgcopydb%';"
   

2. When you run the migrate, add the following flags to exclude specific materialized views being materialized:

   --skip-table-data <matview1> <matview2>”
   

3. When migrate has finished, manually refresh the materialized views you excluded.

Restart migration from scratch after a non-resumable failure

If the migration halts due to a failure, such as a misconfiguration of the source or target database, you may need to restart the migration from scratch. In such cases, you can reuse the original target Tiger Cloud service created for the migration by utilizing the --drop-if-exists flag with the migrate command.

This flag ensures that the existing target objects created by the previous migration are dropped, allowing the migration to proceed without trouble.

Note: This flag also requires you to manually recreate the TimescaleDB extension on the target.

Here’s an example command sequence to restart the migration:

psql target -c "DROP EXTENSION timescaledb CASCADE"

psql target -c 'CREATE EXTENSION timescaledb VERSION "<desired version>"'

docker run --rm -it --name live-migration-migrate \
    -e PGCOPYDB_SOURCE_PGURI=source \
    -e PGCOPYDB_TARGET_PGURI=target \
    --pid=host \
    -v ~/live-migration:/opt/timescale/ts_cdc \
    timescale/live-migration:latest migrate --drop-if-exists

This approach provides a clean slate for the migration process while reusing the existing target instance.

Inactive or lagging replication slots

If you encounter an “Inactive or lagging replication slots” warning on your cloud provider console after using live-migration, it might be due to lingering replication slots created by the live-migration tool on your source database.

To clean up resources associated with live migration, use the following command:

docker run --rm -it --name live-migration-clean \
    -e PGCOPYDB_SOURCE_PGURI=source \
    -e PGCOPYDB_TARGET_PGURI=target \
    --pid=host \
    -v ~/live-migration:/opt/timescale/ts_cdc \
    timescale/live-migration:latest clean --prune

The --prune flag is used to delete temporary files in the ~/live-migration directory that were needed for the migration process. It's important to note that executing the clean command means you cannot resume the interrupted live migration.

Role passwords

Because of issues dumping passwords from various managed service providers, Live-migration migrates roles without passwords. You have to migrate passwords manually.

Table privileges

Live-migration does not migrate table privileges. After completing Live-migration:

1. Grant all roles to tsdbadmin.

   psql -d source -t -A -c "SELECT FORMAT('GRANT %I TO tsdbadmin;', rolname) FROM
   pg_catalog.pg_roles WHERE rolname not like 'pg_%' AND rolname != 'tsdbadmin'
   AND NOT rolsuper" | psql -d target -f -
   

2. On your migration machine, edit /tmp/grants.psql to match table privileges on your source database.

   pg_dump --schema-only --quote-all-identifiers
   --exclude-schema=_timescaledb_catalog --format=plain --dbname "source" | grep
   "(ALTER.*OWNER.*|GRANT|REVOKE)"  > /tmp/grants.psql
   

3. Run grants.psql on your target Tiger Cloud service.

   psql -d target -f /tmp/grants.psql
   

Postgres to Tiger Cloud: “live-replay not keeping up with source load”

1. Go to Tiger Cloud Console -> Monitoring -> Insights tab and find the query which takes significant time
2. If the query is either UPDATE/DELETE, make sure the columns used on the WHERE clause have necessary indexes.
3. If the query is either UPDATE/DELETE on the tables which are converted as hypertables, make sure the REPLIDA IDENTITY(defaults to primary key) on the source is compatible with the target primary key. If not, create an UNIQUE index source database by including the hypertable partition column and make it as a REPLICA IDENTITY. Also, create the same UNIQUE index on target.

ERROR: out of memory (or) Failed on request of size xxx in memory context "yyy" on a Tiger Cloud service

This error occurs when the Out of Memory (OOM) guard is triggered due to memory allocations exceeding safe limits. It typically happens when multiple concurrent connections to the TimescaleDB instance are performing memory-intensive operations. For example, during live migrations, this error can occur when large indexes are being created simultaneously.

The live-migration tool includes a retry mechanism to handle such errors. However, frequent OOM crashes may significantly delay the migration process.

One of the following can be used to avoid the OOM errors:

1. Upgrade to Higher Memory Spec Instances: To mitigate memory constraints, consider using a TimescaleDB instance with higher specifications, such as an instance with 8 CPUs and 32 GB RAM (or more). Higher memory capacity can handle larger workloads and reduce the likelihood of OOM errors.

2. Reduce Concurrency: If upgrading your instance is not feasible, you can reduce the concurrency of the index migration process using the --index-jobs=<value> flag in the migration command. By default, the value of --index-jobs matches the GUC max_parallel_workers. Lowering this value reduces the memory usage during migration but may increase the total migration time.

By taking these steps, you can prevent OOM errors and ensure a smoother migration experience with TimescaleDB.

===== PAGE: https://docs.tigerdata.com/migrate/dual-write-and-backfill/ =====

Low-downtime migrations with dual-write and backfill

Dual-write and backfill is a migration strategy to move a large amount of time-series data (100 GB-10 TB+) with low downtime (on the order of minutes of downtime). It is significantly more complicated to execute than a migration with downtime using pg_dump/restore, and has some prerequisites on the data ingest patterns of your application, so it may not be universally applicable.

In the context of migrations, your existing production database is referred to as the SOURCE database, the Tiger Cloud service that you are migrating your data to is the TARGET.

Roughly, it consists of three steps:

1. Clone schema and relational data from source to target
2. Dual-write to source and target
3. Backfill time-series data

Dual-write and backfill can be used for any source database type, as long as it can provide data in csv format. It can be used to move data from a PostgresSQL source, and from TimescaleDB to TimescaleDB.

Dual-write and backfill works well when:

1. The bulk of the (on-disk) data is in time-series tables.
2. Writes by the application do not reference historical time-series data.
3. Writes to time-series data are append-only.
4. No UPDATE or DELETE queries will be run on time-series data in the source database during the migration process (or if they are, it happens in a controlled manner, such that it's possible to either ignore, or re-backfill).
5. Either the relational (non-time-series) data is small enough to be copied from source to target in an acceptable amount of time for this to be done with downtime, or the relational data can be copied asynchronously while the application continues to run (that is, changes relatively infrequently).

For more information, consult the step-by-step guide for your source database:

• Dual-write and backfill from TimescaleDB
• Dual-write and backfill from Postgres
• Dual-write and backfill from other

If you get stuck, you can get help by either opening a support request, or take your issue to the #migration channel in the community slack, where the developers of this migration method are there to help.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

===== PAGE: https://docs.tigerdata.com/migrate/troubleshooting/ =====

FAQ and troubleshooting

Unsupported in live migration

Live migration tooling is currently experimental. You may run into the following shortcomings:

• Live migration does not yet support mutable columnstore compression (INSERT, UPDATE, DELETE on data in the columnstore).
• By default, numeric fields containing NaN/+Inf/-Inf values are not correctly replicated, and will be converted to NULL. A workaround is available, but is not enabled by default.

Should you run into any problems, please open a support request before losing any time debugging issues. You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

Where can I find logs for processes running during live migration?

Live migration involves several background processes to manage different stages of the migration. The logs of these processes can be helpful for troubleshooting unexpected behavior. You can find these logs in the <volume_mount>/logs directory.

Source and target databases have different TimescaleDB versions

When you migrate a self-hosted or Managed Service for TimescaleDB (MST) database to Tiger Cloud, the source database and the destination Tiger Cloud service must run the same version of TimescaleDB.

Before you start live migration:

1. Check the version of TimescaleDB running on the source database and the target Tiger Cloud service:

   select extversion from pg_extension where extname = 'timescaledb';
   

2. If the version of TimescaleDB on the source database is lower than your Tiger Cloud service, either:

   • Downgrade: reinstall an older version of TimescaleDB on your Tiger Cloud service that matches the source database:

     1. Connect to your Tiger Cloud service and check the versions of TimescaleDB available:

        SELECT version FROM pg_available_extension_versions WHERE name = 'timescaledb' ORDER BY 1 DESC;
        

     2. If an available TimescaleDB release matches your source database:

        1. Uninstall TimescaleDB from your Tiger Cloud service:

           DROP EXTENSION timescaledb;
           

        2. Reinstall the correct version of TimescaleDB:

           CREATE EXTENSION timescaledb VERSION '<version>';
           

        You may need to reconnect to your Tiger Cloud service using psql -X when you're creating the TimescaleDB extension.

   • Upgrade: for self-hosted databases, upgrade TimescaleDB to match your Tiger Cloud service.

Why does live migration log "no tuple identifier" warning?

Live migration logs a warning WARNING: no tuple identifier for UPDATE in table when it cannot determine which specific rows should be updated after receiving an UPDATE statement from the source database during replication. This occurs when tables in the source database that receive UPDATE statements lack either a PRIMARY KEY or a REPLICA IDENTITY setting. For live migration to successfully replicate UPDATE and DELETE statements, tables must have either a PRIMARY KEY or REPLICA IDENTITY set as a prerequisite.

Set REPLICA IDENTITY on Postgres partitioned tables

If your Postgres tables use native partitioning, setting REPLICA IDENTITY on the root (parent) table will not automatically apply it to the partitioned child tables. You must manually set REPLICA IDENTITY on each partitioned child table.

Can I use read/failover replicas as source database for live migration?

Live migration does not support replication from read or failover replicas. You must provide a connection string that points directly to your source database for live migration.

Can I use live migration with a Postgres connection pooler like PgBouncer?

Live migration does not support connection poolers. You must provide a connection string that points directly to your source and target databases for live migration to work smoothly.

Can I use Tiger Cloud instance as source for live migration?

No, Tiger Cloud cannot be used as a source database for live migration.

How can I exclude a schema/table from being replicated in live migration?

At present, live migration does not allow for excluding schemas or tables from replication, but this feature is expected to be added in future releases. However, a workaround is available for skipping table data using the --skip-table-data flag. For more information, please refer to the help text under the migrate subcommand.

Large migrations blocked

Tiger Cloud automatically manages the underlying disk volume. Due to platform limitations, it is only possible to resize the disk once every six hours. Depending on the rate at which you're able to copy data, you may be affected by this restriction. Affected instances are unable to accept new data and error with: FATAL: terminating connection due to administrator command.

If you intend on migrating more than 400&nbspGB of data to Tiger Cloud, open a support request requesting the required storage to be pre-allocated in your Tiger Cloud service.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

Dumping and locks

When pg_dump starts, it takes an ACCESS SHARE lock on all tables which it dumps. This ensures that tables aren't dropped before pg_dump is able to drop them. A side effect of this is that any query which tries to take an ACCESS EXCLUSIVE lock on a table is be blocked by the ACCESS SHARE lock.

A number of Tiger Cloud-internal processes require taking ACCESS EXCLUSIVE locks to ensure consistency of the data. The following is a non-exhaustive list of potentially affected operations:

• converting a chunk into the columnstore/rowstore and back
• continuous aggregate refresh (before 2.12)
• create hypertable with foreign keys, truncate hypertable
• enable hypercore on a hypertable
• drop chunks

The most likely impact of the above is that background jobs for retention policies, columnstore compression policies, and continuous aggregate refresh policies are blocked for the duration of the pg_dump command. This may have unintended consequences for your database performance.

Dumping with concurrency

When using the pg_dump directory format, it is possible to use concurrency to use multiple connections to the source database to dump data. This speeds up the dump process. Due to the fact that there are multiple connections, it is possible for pg_dump to end up in a deadlock situation. When it detects a deadlock it aborts the dump.

In principle, any query which takes an ACCESS EXCLUSIVE lock on a table causes such a deadlock. As mentioned above, some common operations which take an ACCESS EXCLUSIVE lock are:

• retention policies
• columnstore compression policies
• continuous aggregate refresh policies

If you would like to use concurrency nonetheless, turn off all background jobs in the source database before running pg_dump, and turn them on once the dump is complete. If the dump procedure takes longer than the continuous aggregate refresh policy's window, you must manually refresh the continuous aggregate in the correct time range. For more information, consult the refresh policies documentation.

To turn off the jobs:

SELECT public.alter_job(id::integer, scheduled=>false)
FROM _timescaledb_config.bgw_job
WHERE id >= 1000;

To turn on the jobs:

SELECT public.alter_job(id::integer, scheduled=>true)
FROM _timescaledb_config.bgw_job
WHERE id >= 1000;

Restoring with concurrency

If the directory format is used for pg_dump and pg_restore, concurrency can be employed to speed up the process. Unfortunately, loading the tables in the timescaledb_catalog schema concurrently causes errors. Furthermore, the tsdbadmin user does not have sufficient privileges to turn off triggers in this schema. To get around this limitation, load this schema serially, and then load the rest of the database concurrently.

pg_restore -d "target" \
    --format=directory \
    --schema='_timescaledb_catalog' \
    --exit-on-error \
    --no-tablespaces \
    --no-owner \
    --no-privileges \
    dump

pg_restore -d "target" \
    --format=directory \
    --jobs=8 \
    --exclude-schema='_timescaledb_catalog' \
    --exit-on-error \
    --disable-triggers \
    --no-tablespaces \
    --no-owner \
    --no-privileges \
    dump

Ownership of background jobs

The _timescaledb_config.bgw_jobs table is used to manage background jobs. This includes custom jobs, columnstore compression policies, retention policies, and continuous aggregate refresh policies. On Tiger Cloud, this table has a trigger which ensures that no database user can create or modify jobs owned by another database user. This trigger can provide an obstacle for migrations.

If the --no-owner flag is used with pg_dump and pg_restore, all objects in the target database are owned by the user that ran pg_restore, likely tsdbadmin.

If all the background jobs in the source database were owned by a user of the same name as the user running the restore (again likely tsdbadmin), then loading the _timescaledb_config.bgw_jobs table should work.

If the background jobs in the source were owned by the postgres user, they are be automatically changed to be owned by the tsdbadmin user. In this case, one just needs to verify that the jobs do not make use of privileges that the tsdbadmin user does not possess.

If background jobs are owned by one or more users other than the user employed in restoring, then there could be issues. To work around this issue, do not dump this table with pg_dump. Provide either --exclude-table-data='_timescaledb_config.bgw_job' or --exclude-table='_timescaledb_config.bgw_job' to pg_dump to skip this table. Then, use psql and the COPY command to dump and restore this table with modified values for the owner column.

psql -d "source" -X -v ON_ERROR_STOP=1 --echo-errors -f - <<'EOF'
begin;
select string_agg
( case attname
    when 'owner' then $$'tsdbadmin' as "owner"$$
    else format('%I', attname)
  end
, ', '
) as cols
from pg_namespace n
inner join pg_class c
on (n.nspname = '_timescaledb_config'
and n.oid = c.relnamespace
and c.relname = 'bgw_job')
inner join pg_attribute a
on (c.oid = a.attrelid and a.attnum > 0)
\gset
copy
(
    select :cols
    from _timescaledb_config.bgw_job
    where id >= 1000
) to stdout with (format csv, header true)
\g bgw_job.csv
rollback;
\q
EOF

psql -X -d "target" -v ON_ERROR_STOP=1 --echo-errors \
    -c "\copy _timescaledb_config.bgw_job from 'bgw_job.csv' with (format csv, header match)"

Once the table has been loaded and the restore completed, you may then use SQL to adjust the ownership of the jobs and/or the associated stored procedures and functions as you wish.

Extension availability

There are a vast number of Postgres extensions available in the wild. Tiger Cloud supports many of the most popular extensions, but not all extensions. Before migrating, check that the extensions you are using are supported on Tiger Cloud. Consult the list of supported extensions.

TimescaleDB extension in the public schema

When self-hosting, the TimescaleDB extension may be installed in an arbitrary schema. Tiger Cloud only supports installing the TimescaleDB extension in the public schema. How to go about resolving this depends heavily on the particular details of the source schema and the migration approach chosen.

Tablespaces

Tiger Cloud does not support using custom tablespaces. Providing the --no-tablespaces flag to pg_dump and pg_restore when dumping/restoring the schema results in all objects being in the default tablespace as desired.

Only one database per instance

While Postgres clusters can contain many databases, Tiger Cloud services are limited to a single database. When migrating a cluster with multiple databases to Tiger Cloud, one can either migrate each source database to a separate Tiger Cloud service or "merge" source databases to target schemas.

Superuser privileges

The tsdbadmin database user is the most powerful available on Tiger Cloud, but it is not a true superuser. Review your application for use of superuser privileged operations and mitigate before migrating.

Migrate partial continuous aggregates

In order to improve the performance and compatibility of continuous aggregates, TimescaleDB v2.7 replaces partial continuous aggregates with finalized continuous aggregates.

To test your database for partial continuous aggregates, run the following query:

SELECT exists (SELECT 1 FROM timescaledb_information.continuous_aggregates WHERE NOT finalized);

If you have partial continuous aggregates in your database, migrate them from partial to finalized before you migrate your database.

If you accidentally migrate partial continuous aggregates across Postgres versions, you see the following error when you query any continuous aggregates:

ERROR:  insufficient data left in message.

===== PAGE: https://docs.tigerdata.com/ai/mcp-server/ =====

Integrate Tiger Cloud with your AI Assistant

The Tiger Model Context Protocol Server provides access to your Tiger Cloud resources through Claude and other AI Assistants. Tiger MCP Server mirrors the functionality of Tiger CLI and is integrated directly into the CLI binary. You manage your Tiger Cloud resources using natural language from your AI Assistant. As Tiger MCP Server is integrated with the Tiger Data documentation, ask any question and you will get the best answer.

This page shows you how to install Tiger CLI and set up secure authentication for Tiger MCP Server, then manage the resources in your Tiger Data account through the Tiger Model Context Protocol Server using your AI Assistant.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Data account.

• Install an AI Assistant on your developer device with an active API key.

  The following AI Assistants are automatically configured by the Tiger Model Context Protocol Server: claude-code, cursor, windsurf, codex, gemini/gemini-cli, vscode/code/vs-code. You can also manually configure Tiger MCP Server.

Install and configure Tiger MCP Server

The Tiger MCP Server is bundled with Tiger CLI:

1. Install Tiger CLI

   Use the terminal to install the CLI:

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   brew install --cask timescale/tap/tiger-cli
   

   curl -fsSL https://cli.tigerdata.com | sh
   

2. Set up API credentials

   1. Log Tiger CLI into your Tiger Data account:

      tiger auth login
      

      Tiger CLI opens Console in your browser. Log in, then click Authorize.

      You can have a maximum of 10 active client credentials. If you get an error, open credentials and delete an unused credential.

   2. Select a Tiger Cloud project:

      Auth URL is: https://console.cloud.timescale.com/oauth/authorize?client_id=lotsOfURLstuff
      Opening browser for authentication...
      Select a project:
      
      > 1. Tiger Project (tgrproject)
      2. YourCompany (Company wide project) (cpnproject)
      3. YourCompany Department (dptproject)
      
      Use ↑/↓ arrows or number keys to navigate, enter to select, q to quit
      

      If only one project is associated with your account, this step is not shown.

      Where possible, Tiger CLI stores your authentication information in the system keychain/credential manager. If that fails, the credentials are stored in ~/.config/tiger/credentials with restricted file permissions (600). By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml.

3. Test your authenticated connection to Tiger Cloud by listing services

   tiger service list
   

   This call returns something like:

   • No services:

     🏜️  No services found! Your project is looking a bit empty.
     🚀 Ready to get started? Create your first service with: tiger service create
     

   • One or more services:

     ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐
     │ SERVICE ID │        NAME         │ STATUS │    TYPE     │    REGION    │     CREATED      │
     ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤
     │ tgrservice │ tiger-agent-service │ READY  │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │
     └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘
     

4. Configure your AI Assistant to interact with the project and services in your Tiger Data account

   For example:

   tiger mcp install
   

5. **Choose the client to integrate with, then press Enter **

   Select an MCP client to configure:
   
   > 1. Claude Code
   2. Codex
   3. Cursor
   4. Gemini CLI
   5. VS Code
   6. Windsurf
   
   Use ↑/↓ arrows or number keys to navigate, enter to select, q to quit
   

And that is it, you are ready to use the Tiger Model Context Protocol Server to manage your services in Tiger Cloud.

Manage the resources in your Tiger Data account through your AI Assistant

Your AI Assistant is connected to your Tiger Data account and the Tiger Data documentation, you can now use it to manage your services and learn more about how to implement Tiger Cloud features. For example:

1. Run your AI Assistant

   claude
   

   Claude automatically runs the Tiger MCP Server server that enables you to interact with Tiger Cloud from your AI Assistant.

2. Check your Tiger Model Context Protocol Server configuration

   > is the tigerdata mcp server active for you?
   

   You see something like:

   MCP server is active. I can see the following Tiger Data-related tools available:
   
   - mcp__tiger__get_guide - Retrieve TimescaleDB guides and best practices
   - mcp__tiger__semantic_search_postgres_docs - Search PostgreSQL documentation
   - mcp__tiger__semantic_search_tiger_docs - Search Tiger Cloud and TimescaleDB documentation
   - mcp__tiger__tiger_service_create - Create new database services
   - mcp__tiger__tiger_service_list - List all database services
   - mcp__tiger__tiger_service_show - Show detailed service information
   - mcp__tiger__tiger_service_update_password - Update service passwords
   
   Is there something specific you'd like to do with the Tiger Data MCP server?
   

3. Ask a basic question about your services

   > can you list my active services please
   

   You see something like:

   ⏺ You have 3 active services, all with the same name "my-first-service":
   
   1. tgrservice1 - READY - TimescaleDB - us-east-1 - 1 CPU/4GB - Created: 2025-09-30
   2. tgrservice1 - READY - TimescaleDB - us-east-1 - 1 CPU/4GB - Created: 2025-09-30
   3. tgrservice1 - READY - TimescaleDB - us-east-1 - 1 CPU/4GB - Created: 2025-09-30
   

4. Manage your services without having to learn how to

   For example:

   Can you create a new ai service called bob with a replica for me please
   

   You see something like:

    tiger - Create Database Service (MCP)(name: "bob", addons: ["ai"], replicas: 1, wait: false)
   

5. Find best practice for things you need to do

   For example:

   >  I need to migrate a database with 25 million rows of data to Tiger Cloud,
   please tell me what service optimizations i should do before migration.
   
   

   You see something like:

   ⏺ For migrating a database with 25 million rows to Tiger Cloud, here are the key service optimizations to do before migration:
   
   Pre-Migration Service Optimizations
   
   1. Right-size your compute resources:
     - For 25M rows, start with at least 2 CPU/8GB or 4 CPU/16GB
     - You can scale up during migration if needed, then scale down after
   
   2. Enable the appropriate addons:
     - Use time-series addon if you are working with time-series data
     - Use ai addon if you need vector/AI capabilities
   
   3. Choose optimal region:
     - Select the region closest to your current database to minimize network transfer time
     - This significantly reduces migration duration
   
   4. Consider enhanced storage (Enterprise only):
     - If you expect high write throughput during migration, enhanced storage provides better IOPS (up to 32,000)
   
   5. Skip replicas initially:
     - Start with 0 replicas during migration
     - Add replicas after migration completes to avoid duplicate data copying
   
   6. Plan for storage space:
     - Estimate your uncompressed data size
     - Allocate sufficient high-performance storage (scales automatically but good to plan)
     - You can enable tiered storage later for older data
   
   During Migration Best Practices
   
   Migration method selection:
     - 25M rows: Use schema-then-data migration (migrate schema first, then copy data in chunks)
     - Allows restart if a copy operation fails
     - Better for databases >10GB
   
   Database configuration to set AFTER migration:
     - Set up compression policies (once data is loaded)
     - Configure retention policies
     - Create continuous aggregates
     - Enable tiered storage for cost optimization
   
   Connection settings:
     - Use the pooled connection endpoint for better performance during bulk inserts
     - Disable unnecessary constraints/indexes during migration, re-enable after
   
   Would you like me to help you create an optimized service for this migration?
   

That beats working. Let the Tiger MCP Server do it all for you.

Manually configure the Tiger MCP Server

If your MCP client is not supported by tiger mcp install, follow the client's instructions to install MCP servers. For example, many clients use a JSON file like the following that use tiger mcp start to start Tiger Model Context Protocol Server:

{
   "mcpServers": {
      "tiger": {
         "command": "tiger",
         "args": [
            "mcp",
            "start"
         ]
      }
   }
}

Tiger Model Context Protocol Server commands

Tiger Model Context Protocol Server exposes the following MCP tools to your AI Assistant:

Command	Parameter	Required	Description
service_list	-	-	Returns a list of the services in the current project.
service_get	-	-	Returns detailed information about a service.
	service_id	✓	The unique identifier of the service (10-character alphanumeric string).
	with_password	-	Set to true to include the password in the response and connection string. WARNING: never do this unless the user explicitly requests the password.
service_create	-	-	Create a new service in Tiger Cloud. WARNING: creates billable resources.
	name	-	Set the human-readable name of up to 128 characters for this service.
	addons	-	Set the array of addons to enable for the service. Options: time-series: enables TimescaleDB ai: enables the AI and vector extensions Set an empty array for Postgres-only.
	region	-	Set the AWS region to deploy this service in.
	cpu_memory	-	CPU and memory allocation combination. Available configurations are: shared/shared 0.5 CPU/2 GB 1 CPU/4 GB 2 CPU/8 GB 4 CPU/16 GB 8 CPU/32 GB 16 CPU/64 GB 32 CPU/128 GB
	replicas	-	Set the number of high-availability replicas for fault tolerance.
	wait	-	Set to true to wait for service to be fully ready before returning.
	timeout_minutes	-	Set the timeout in minutes to wait for service to be ready. Only used when wait=true. Default: 30 minutes
	set_default	-	By default, the new service is the default for following commands in CLI. Set to false to keep the previous service as the default.
	with_password	-	Set to true to include the password for this service in response and connection string. WARNING: never set to true unless user explicitly requests the password.
service_update_password	-	-	Update the password for the tsdbadmin for this service. The password change takes effect immediately and may terminate existing connections.
	service_id	✓	The unique identifier of the service you want to update the password for.
	password	✓	The new password for the tsdbadmin user.
db_execute_query	-	-	Execute a single SQL query against a service. This command returns column metadata, result rows, affected row count, and execution time. Multi-statement queries are not supported. WARNING: can execute destructive SQL including INSERT, UPDATE, DELETE, and DDL commands.
	service_id	✓	The unique identifier of the service. Use tiger_service_list to find service IDs.
	query	✓	The SQL query to execute. Single statement queries are supported.
	parameters	-	Query parameters for parameterized queries. Values are substituted for the $n placeholders in the query.
	timeout_seconds	-	The query timeout in seconds. Default: 30.
	role	-	The service role/username to connect as. Default: tsdbadmin.
	pooled	-	Use connection pooling. This is only available if you have already enabled it for the service. Default: false.

Tiger CLI commands for Tiger MCP Server

You can use the following Tiger CLI commands to run Tiger MCP Server:

Usage: tiger mcp [subcommand] --<flags>

Command	Subcommand	Description
mcp		Manage the Tiger Model Context Protocol Server
	install [client]	Install and configure Tiger MCP Server for a specific client installed on your developer device. Supported clients are: claude-code, cursor, windsurf, codex, gemini/gemini-cli, vscode/code/vs-code. Flags: --no-backup: do not back up the existing configuration --config-path: open the configuration file at a specific location
	start	Start the Tiger MCP Server. This is the same as tiger mcp start stdio
	start stdio	Start the Tiger MCP Server with stdio transport
	start http	Start the Tiger MCP Server with HTTP transport. This option is for users who wish to access Tiger Model Context Protocol Server without using stdio. For example, your AI Assistant does not support stdio, or you do not want to run CLI on your device. Flags are: --port <port number>: the default is 8000 --host <hostname>: the default is localhost

Global flags

You can use the following Tiger CLI global flags when you run the Tiger MCP Server:

Flag	Default	Description
--analytics	true	Set to false to disable usage analytics
--color	true	Set to false to disable colored output
--config-dir string	.config/tiger	Set the directory that holds config.yaml
--debug	No debugging	Enable debug logging
--help	-	Print help about the current command. For example, tiger service --help
--password-storage string	keyring	Set the password storage method. Options are keyring, pgpass, or none
--service-id string	-	Set the Tiger Cloud service to manage
--skip-update-check	-	Do not check if a new version of Tiger CLI is available

===== PAGE: https://docs.tigerdata.com/ai/tiger-eon/ =====

Aggregate organizational data with AI agents

Your business already has the answers in Slack threads, GitHub pull requests, Linear tasks, your own docs, Salesforce service tickets, anywhere you store data. However, those answers are scattered, hard to find, and often forgotten. Tiger Eon automatically integrates Tiger Agents for Work with your organizational data so you can let AI Assistants analyze your company data and give you the answers you need. For example:

• What did we ship last week?
• What's blocking the release?
• Summarize the latest GitHub pull requests.

Eon responds instantly, pulling from the tools you already use. No new UI, no new workflow, just answers in Slack.

Tiger Eon:

• Unlocks hidden value: your data in Slack, GitHub, and Linear already contains the insights you need. Eon makes them accessible.
• Enables faster decisions: no need to search or ask around, you get answers in seconds.
• Is easy to use: Eon runs a Tiger Agent and MCP servers statelessly in lightweight Docker containers.
• Integrates seamlessly with Tiger Cloud: Eon uses a Tiger Cloud service so you securely and reliably store your company data. Prefer to self-host? Use a Postgres instance with TimescaleDB.

Tiger Eon's real-time ingestion system connects to Slack and captures everything: every message, reaction, edit, and channel update. It can also process historical Slack exports. Eon had instant access to years of institutional knowledge from the very beginning.

All of this data is stored in your Tiger Cloud service as time-series data: conversations are events unfolding over time, and Tiger Cloud is purpose-built for precisely this. Your data is optimized by:

• Automatically partitioning the data into 7-day chunks for efficient queries
• Compressing the data after 45 days to save space
• Segmenting by channel for faster retrieval

When someone asks Eon a question, it uses simple SQL to instantly retrieve the full thread context, related conversations, and historical decisions. No rate limits. No API quotas. Just direct access to your data.

This page shows you how to install and run Eon.

Prerequisites

To follow the procedure on this page you need to:

• Create a Tiger Data account.

  This procedure also works for self-hosted TimescaleDB.

• Install Docker on your developer device
• Install Tiger CLI
• Have rights to create an Anthropic API key
• Optionally:
  • Have rights to create a GitHub token
  • Have rights to create a Logfire token
  • Have rights to create a Linear token

Interactive setup

Tiger Eon is a production-ready repository running Tiger CLI and Tiger Agents for Work that creates and runs the following components for you:

• An ingest Slack app that consumes all messages and reactions from public channels in your Slack workspace
• A Tiger Agent that analyzes your company data for you
• A Tiger Cloud service instance that stores data from the Slack apps
• MCP servers that connect data sources to Eon
• A listener Slack app that passes questions to the Tiger Agent when you @tag it in a public channel, and returns the AI analysis on your data

All local components are run in lightweight Docker containers via Docker Compose.

This section shows you how to run the Eon setup to configure Eon to connect to your Slack app, and give it access to your data and analytics stored in Tiger Cloud.

1. Install Tiger Eon to manage and run your AI-powered Slack bots

   In a local folder, run the following command from the terminal:

   git clone git@github.com:timescale/tiger-eon.git
   

2. Start the Eon setup

   cd tiger-eon
   ./setup-tiger-eon.sh
   

   You see a summary of the setup procedure. Type y and press Enter.

3. Create the Tiger Cloud service to use with Eon

   You see Do you want to use a free tier Tiger Cloud Database? [y/N]:. Press Y to create a free Tiger Cloud service.

   Eon opens the Tiger Cloud authentication page in your browser. Click Authorize. Eon creates a Tiger Cloud service called tiger-eon and stores the credentials in your local keychain.

   If you press N, the Eon setup creates and runs TimescaleDB in a local Docker container.

4. Create the ingest Slack app

   1. In the terminal, name your ingest Slack app:

      1. Eon proposes to create an ingest app called tiger-slack-ingest, press Enter.
      2. Do the same for the App description.

      Eon opens Your Apps in https://api.slack.com/apps/.

   2. Start configuring your ingest app in Slack:

      In the Slack Your Apps page:

      1. Click Create New App, click From an manifest, then select a workspace.
      2. Click Next. Slack opens Create app from manifest.

   3. Add the Slack app manifest:

      1. In terminal press Enter. The setup prints the Slack app manifest to terminal and adds it to your clipboard.
      2. In the Slack Create app from manifest window, paste the manifest.
      3. Click Next, then click Create.

   4. Configure an app-level token:

      1. In your app settings, go to Basic Information.
      2. Scroll to App-Level Tokens.
      3. Click Generate Token and Scopes.
      4. Add a Token Name, then click Add Scope add connections:write, then click Generate.
      5. Copy the xapp-* token and click Done.
      6. In the terminal, paste the token, then press Enter.

   5. Configure a bot user OAuth token:

      1. In your app settings, under Features, click App Home.
      2. Scroll down, then enable Allow users to send Slash commands and messages from the messages tab.
      3. In your app settings, under Settings, click Install App.
      4. Click Install to <workspace name>, then click Allow.
      5. Copy the xoxb- Bot User OAuth Token locally.
      6. In the terminal, paste the token, then press Enter.

5. Create the Eon Slack app

   Follow the same procedure as you did for the ingest Slack app.

6. Integrate Eon with Anthropic

   The Eon setup opens https://console.anthropic.com/settings/keys. Create a Claude Code key, then paste it in the terminal.

7. Integrate Eon with Logfire

   If you would like to integrate logfire with Eon, paste your token and press Enter. If not, press Enter.

8. Integrate Eon with GitHub

   The Eon setup asks if you would like to Enable github MCP server?". For Eon to answer questions about the activity in your Github organization. Press y to integrate with GitHub.

9. Integrate Eon with Linear

   The Eon setup asks if you would like to Enable linear MCP server? [y/N]:. Press y to integrate with Linear.

10. Give Eon access to private repositories

    1. The setup asks if you would like to include access to private repositories. Press y.
    2. Follow the GitHub token creation process.
    3. In the Eon setup add your organization name, then paste the GitHub token.

    The setup sets up a new Tiger Cloud service for you called tiger-eon, then starts Eon in Docker.

    

You have created:

• The Eon ingest and chat apps in Slack
• A private MCP server connecting Eon to your data in GitHub
• A Tiger Cloud service that securely stores the data used by Eon

Integrate Eon in your Slack workspace

To enable your AI Assistant to analyze your data for you when you ask a question, open a public channel, invite @eon to join, then ask a question:

===== PAGE: https://docs.tigerdata.com/ai/tiger-agents-for-work/ =====

Integrate a slack-native AI agent

Tiger Agents for Work is a Slack-native AI agent that you use to unify the knowledge in your company. This includes your Slack history, docs, GitHub repositories, Salesforce and so on. You use your Tiger Agent to get instant answers for real business, technical, and operations questions in your Slack channels.

Tiger Agents for Work can handle concurrent conversations with enterprise-grade reliability. They have the following features:

• Durable and atomic event handling: Postgres-backed event claiming ensures exactly-once processing, even under high concurrency and failure conditions
• Bounded concurrency: fixed worker pools prevent resource exhaustion while maintaining predictable performance under load
• Immediate event processing: Tiger Agents for Work provide real-time responsiveness. Events are processed within milliseconds of arrival rather than waiting for polling cycles
• Resilient retry logic: automatic retry with visibility thresholds, plus stuck or expired event cleanup
• Horizontal scalability: run multiple Tiger Agent instances simultaneously with coordinated work distribution across all instances
• AI-Powered Responses: use the AI model of your choice, you can also integrate with MCP servers
• Extensible architecture: zero code integration for basic agents. For more specialized use cases, easily customize your agent using Jinja templates
• Complete observability: detailed tracing of event flow, worker activity, and database operations with full Logfire instrumentation

This page shows you how to install the Tiger Agent CLI, connect to the Tiger Data MCP server, and customize prompts for your specific needs.

Prerequisites

To follow the procedure on this page you need to:

• Create a Tiger Data account.

  This procedure also works for self-hosted TimescaleDB.

• Install the uv package manager

• Get an Anthropic API key

• Optional: get a Logfire token

Create a Slack app

Before installing Tiger Agents for Work, you need to create a Slack app that the Tiger Agent will connect to. This app provides the security tokens for Slack integration with your Tiger Agent:

1. Create a manifest for your Slack App

   1. In a temporary directory, download the Tiger Agent Slack manifest template:

      curl -O https://raw.githubusercontent.com/timescale/tiger-agents-for-work/main/slack-manifest.json
      

   2. Edit slack-manifest.json and customize your name and description of your Slack App. For example:

      "display_information": {
        "name": "Tiger Agent",
        "description": "Tiger AI Agent helps you easily access your business information, and tune your Tiger services",
        "background_color": "#000000"
      },
      "features": {
        "bot_user": {
          "display_name": "Tiger Agent",
          "always_online": true
        }
      },
      

   3. Copy the contents of slack-manifest.json to the clipboard:

      cat slack-manifest.json| pbcopy
      

2. Create the Slack app

   1. Go to api.slack.com/apps.
   2. Click Create New App.
   3. Select From a manifest.
   4. Choose your workspace, then click Next.
   5. Paste the contents of slack-manifest.json and click Next.
   6. Click Create.

3. Generate an app-level token

   1. In your app settings, go to Basic Information.
   2. Scroll to App-Level Tokens.
   3. Click Generate Token and Scopes.
   4. Add a Token Name, then click Add Scope, add connections:write then click Generate.
   5. Copy the xapp-* token locally and click Done.

4. Install your app to a Slack workspace

   1. In the sidebar, under Settings, click Install App.
   2. Click Install to <workspace name>, then click Allow.
   3. Copy the xoxb- Bot User OAuth Token locally.

You have created a Slack app and obtained the necessary tokens for Tiger Agent integration.

Install and configure your Tiger Agent instance

Tiger Agents for Work are a production-ready library and CLI written in Python that you use to create Slack-native AI agents. This section shows you how to configure a Tiger Agent to connect to your Slack app, and give it access to your data and analytics stored in Tiger Cloud.

1. Create a project directory

   mkdir my-tiger-agent
   cd my-tiger-agent
   

2. Create a Tiger Agent environment with your Slack, AI Assistant, and database configuration

   1. Download .env.sample to a local .env file:

   curl -L -o .env https://raw.githubusercontent.com/timescale/tiger-agent/refs/heads/main/.env.sample
   

   1. In .env, add your Slack tokens and Anthropic API key:

   SLACK_APP_TOKEN=xapp-your-app-token
   SLACK_BOT_TOKEN=xoxb-your-bot-token
   
   ANTHROPIC_API_KEY=sk-ant-your-api-key
   
   LOGFIRE_TOKEN=your-logfire-token
   

   1. Add the connection details for the Tiger Cloud service you are using for this Tiger Agent:

   PGHOST=<host>
   PGDATABASE=tsdb
   PGPORT=<port>
   PGUSER=tsdbadmin
   PGPASSWORD=<password>
   

   1. Save and close .env.

3. Add the default Tiger Agent prompts to your project

   mkdir prompts
   curl -L -o prompts/system_prompt.md https://raw.githubusercontent.com/timescale/tiger-agent/refs/heads/main/prompts/system_prompt.md
   curl -L -o prompts/user_prompt.md https://raw.githubusercontent.com/timescale/tiger-agent/refs/heads/main/prompts/user_prompt.md
   

4. Install Tiger Agents for Work to manage and run your AI-powered Slack bots

   1. Install the Tiger Agent CLI using uv.

      uv tool install --from git+https://github.com/timescale/tiger-agents-for-work.git tiger-agent
      

      tiger-agent is installed in ~/.local/bin/tiger-agent. If necessary, add this folder to your PATH.

   2. Verify the installation.

      tiger-agent --help
      

      You see the Tiger Agent CLI help output with the available commands and options.

5. Connect your Tiger Agent with Slack

   1. Run your Tiger Agent:

      tiger-agent run --prompts prompts/  --env .env
      

      If you open the explorer in Tiger Cloud Console, you can see the tables used by your Tiger Agent.

   2. In Slack, open a public channel app and ask Tiger Agent a couple of questions. You see the response in your public channel and log messages in the terminal.

   

Add information from MCP servers to your Tiger Agent

To increase the amount of specialized information your AI Assistant can use, you can add MCP servers supplying data your users need. For example, to add the Tiger Data MCP server to your Tiger Agent:

1. Copy the example mcp_config.json to your project

   In my-tiger-agent, run the following command:

   ```bash
    curl -L -o mcp_config.json https://raw.githubusercontent.com/timescale/tiger-agent/refs/heads/main/examples/mcp_config.json
    ```
   

2. Configure your Tiger Agent to connect to the most useful MCP servers for your organization

   For example, to add the Tiger Data documentation MCP server to your Tiger Agent, update the docs entry to the following:

   "docs": {
     "tool_prefix": "docs",
     "url": "https://mcp.tigerdata.com/docs",
     "allow_sampling": false
   },
   

   To avoid errors, delete all entries in mcp_config.json with invalid URLs. For example the github entry with http://github-mcp-server/mcp.

3. Restart your Tiger Agent

   tiger-agent run --prompts prompts/ --mcp-config mcp_config.json
   

You have configured your Tiger Agent to connect to the Tiger MCP Server. For more information, see MCP Server Configuration.

Customize prompts for personalization

Tiger Agents for Work uses Jinja2 templates for dynamic, context-aware prompt generation. This system allows for sophisticated prompts that adapt to conversation context, user preferences, and event metadata. Tiger Agents for Work uses the following templates:

• system_prompt.md: defines the AI Assistant's role, capabilities, and behavior patterns. This template sets the foundation for the way your Tiger Agent will respond and interact.
• user_prompt.md: formats the user's request with relevant context, providing the AI Assistant with the information necessary to generate an appropriate response.

To change the way your Tiger Agents interact with users in your Slack app:

1. Update the prompt

   For example, in prompts/system_prompt.md, add another item in the Response Protocol section to fine tune the behavior of your Tiger Agents. For example:

   5. Be snarky but vaguely amusing
   

2. Test your configuration

   Run Tiger Agent with your custom prompt:

   tiger-agent run --mcp-config mcp_config.json --prompts prompts/
   

For more information, see Prompt tempates.

Advanced configuration options

For additional customization, you can modify the following Tiger Agent parameters:

• --model: change AI model (default: anthropic:claude-sonnet-4-20250514)
• --num-workers: adjust concurrent workers (default: 5)
• --max-attempts: set retry attempts per event (default: 3)

Example with custom settings:

tiger-agent run \
  --model claude-3-5-sonnet-latest \
  --mcp-config mcp_config.json \
  --prompts prompts/ \
  --num-workers 10 \
  --max-attempts 5

Your Tiger Agents are now configured with Tiger Data MCP server access and personalized prompts.

===== PAGE: https://docs.tigerdata.com/ai/key-vector-database-concepts-for-understanding-pgvector/ =====

Key vector database concepts for understanding pgvector

Vector data type provided by pgvector

Vectors inside of the database are stored in regular Postgres tables using vector columns. The vector column type is provided by the pgvector extension. A common way to store vectors is alongside the data they have indexed. For example, to store embeddings for documents, a common table structure is:

CREATE TABLE IF NOT EXISTS document_embedding  (
    id BIGINT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
    document_id BIGINT FOREIGN KEY(document.id)
    metadata JSONB,
    contents TEXT,
    embedding VECTOR(1536)
)

This table contains a primary key, a foreign key to the document table, some metadata, the text being embedded (in the contents column), and the embedded vector.

This may seem like a bit of a weird design: why aren't the embeddings simply a separate column in the document table? The answer has to do with context length limits of embedding models and of LLMs. When embedding data, there is a limit to the length of content you can embed (for example, OpenAI's ada-002 has a limit of 8191 tokens ), and so, if you are embedding a long piece of text, you have to break it up into smaller chunks and embed each chunk individually. Therefore, when thinking about this at the database layer, there is usually a one-to-many relationship between the thing being embedded and the embeddings which is represented by a foreign key from the embedding to the thing.

Of course, if you do not want to store the original data in the database and you are just storing only the embeddings, that's totally fine too. Just omit the foreign key from the table. Another popular alternative is to put the foreign key into the metadata JSONB.

Querying vectors using pgvector

The canonical query for vectors is for the closest query vectors to an embedding of the user's query. This is also known as finding the K nearest neighbors.

In the example query below, $1 is a parameter taking a query embedding, and the <=> operator calculates the distance between the query embedding and embedding vectors stored in the database (and returns a float value).

SELECT *
FROM document_embedding
ORDER BY embedding <=> $1
LIMIT 10

The query above returns the 10 rows with the smallest distance between the query's embedding and the row's embedding. Of course, this being Postgres, you can add additional WHERE clauses (such as filters on the metadata), joins, etc.

Vector distance types

The query shown above uses something called cosine distance (using the <=> operator) as a measure of how similar two embeddings are. But, there are multiple ways to quantify how far apart two vectors are from each other.

In practice, the choice of distance measure doesn't matters much and it is recommended to just stick with cosine distance for most applications.

Description of cosine distance, negative inner product, and Euclidean distance

Here's a succinct description of three common vector distance measures

• Cosine distance a.k.a. angular distance: This measures the cosine of the angle between two vectors. It's not a true "distance" in the mathematical sense but a similarity measure, where a smaller angle corresponds to a higher similarity. The cosine distance is particularly useful in high-dimensional spaces where the magnitude of the vectors (their length) is less important, such as in text analysis or information retrieval. It ranges from -1 (meaning exactly opposite) to 1 (exactly the same), with 0 typically indicating orthogonality (no similarity). See here for more on cosine similarity.

• Negative inner product: This is simply the negative of the inner product (also known as the dot product) of two vectors. The inner product measures vector similarity based on the vectors' magnitudes and the cosine of the angle between them. A higher inner product indicates greater similarity. However, it's important to note that, unlike cosine similarity, the magnitude of the vectors influences the inner product.

• Euclidean distance: This is the "ordinary" straight-line distance between two points in Euclidean space. In terms of vectors, it's the square root of the sum of the squared differences between corresponding elements of the vectors. This measure is sensitive to the magnitude of the vectors and is widely used in various fields such as clustering and nearest neighbor search.

Many embedding systems (for example OpenAI's ada-002) use vectors with length 1 (unit vectors). For those systems, the rankings (ordering) of all three measures is the same. In particular,

• The cosine distance is 1−dot product.
• The negative inner product is −dot product.
• The Euclidean distance is related to the dot product, where the squared Euclidean distance is 2(1−dot product).

Recommended vector distance for use in Postgres

Using cosine distance, especially on unit vectors, is recommended. These recommendations are based on OpenAI's recommendation as well as the fact that the ranking of different distances on unit vectors is preserved.

Vector search indexing (approximate nearest neighbor search)

In Postgres and other relational databases, indexing is a way to speed up queries. For vector data, indexes speed up the similarity search query shown above where you find the most similar embedding to some given query embedding. This problem is often referred to as finding the K nearest neighbors.

The term "index" in the context of vector databases has multiple meanings. It can refer to both the storage mechanism for your data and the tool that enhances query efficiency. These docs use the latter meaning.

Finding the K nearest neighbors is not a new problem in Postgres, but existing techniques only work with low-dimensional data. These approaches cease to be effective when dealing with data larger than approximately 10 dimensions due to the "curse of dimensionality." Given that embeddings often consist of more than a thousand dimensions(OpenAI's are 1,536) new techniques had to be developed.

There are no known exact algorithms for efficiently searching in such high-dimensional spaces. Nevertheless, there are excellent approximate algorithms that fall into the category of approximate nearest neighbor algorithms.

There are 3 different indexing algorithms available as part of pgai on Tiger Cloud: StreamingDiskANN, HNSW, and ivfflat. The table below illustrates the high-level differences between these algorithms:

Algorithm	Build Speed	Query Speed	Need to rebuild after updates
StreamingDiskANN	Fast	Fastest	No
HNSW	Fast	Fast	No
ivfflat	Fastest	Slowest	Yes

See the performance benchmarks for details on how the each index performs on a dataset of 1 million OpenAI embeddings.

Recommended index types

For most applications, the StreamingDiskANN index is recommended.

===== PAGE: https://docs.tigerdata.com/ai/sql-interface-for-pgvector-and-timescale-vector/ =====

SQL inteface for pgvector and pgvectorscale

Installing the pgvector and pgvectorscale extensions

If not already installed, install the vector and vectorscale extensions on your Tiger Data database.

CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS vectorscale;

Creating the table for storing embeddings using pgvector

Vectors inside of the database are stored in regular Postgres tables using vector columns. The vector column type is provided by the pgvector extension. A common way to store vectors is alongside the data they are embedding. For example, to store embeddings for documents, a common table structure is:

CREATE TABLE IF NOT EXISTS document_embedding  (
    id BIGINT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
    document_id BIGINT FOREIGN KEY(document.id)
    metadata JSONB,
    contents TEXT,
    embedding VECTOR(1536)
)

This table contains a primary key, a foreign key to the document table, some metadata, the text being embedded (in the contents column) and the embedded vector.

You may ask why not just add an embedding column to the document table? The answer is that there is a limit on the length of text an embedding can encode and so there needs to be a one-to-many relationship between the full document and its embeddings.

The above table is just an illustration, it's totally fine to have a table without a foreign key and/or without a metadata column. The important thing is to have a column with the data being embedded and the vector in the same row, enabling you to return the raw data for a given similarity search query

The vector type can specify an optional number of dimensions (1,538) in the example above). If specified, it enforces the constraint that all vectors in the column have that number of dimensions. A plain VECTOR (without specifying the number of dimensions) column is also possible and allows a variable number of dimensions.

Query the vector embeddings

The canonical query is:

SELECT *
FROM document_embedding
ORDER BY embedding <=> $1
LIMIT 10

Which returns the 10 rows whose distance is the smallest. The distance function used here is cosine distance (specified by using the <=> operator). Other distance functions are available, see the discussion.

The available distance types and their operators are:

Distance type	Operator
Cosine/Angular	<=>
Euclidean	<->
Negative inner product	<#>

If you are using an index, you need to make sure that the distance function used in index creation is the same one used during query (see below). This is important because if you create your index with one distance function but query with another, your index cannot be used to speed up the query.

Indexing the vector data using indexes provided by pgvector and pgvectorscale

Indexing helps speed up similarity queries of the basic form:

SELECT *
FROM document_embedding
ORDER BY embedding <=> $1
LIMIT 10

The key part is that the ORDER BY contains a distance measure against a constant or a pseudo-constant.

Note that if performing a query without an index, you always get an exact result, but the query is slow (it has to read all of the data you store for every query). With an index, your queries are an order-of-magnitude faster, but the results are approximate (because there are no known indexing techniques that are exact see here for more).

Nevertheless, there are excellent approximate algorithms. There are 3 different indexing algorithms available on TimescaleDB: StreamingDiskANN, HNSW, and ivfflat. Below is the trade-offs between these algorithms:

Algorithm	Build Speed	Query Speed	Need to rebuild after updates
StreamingDiskANN	Fast	Fastest	No
HNSW	Fast	Fast	No
ivfflat	Fastest	Slowest	Yes

You can see benchmarks in the blog.

For most use cases, the StreamingDiskANN index is recommended.

Each of these indexes has a set of build-time options for controlling the speed/accuracy trade-off when creating the index and an additional query-time option for controlling accuracy during a particular query.

You can see the details of each index below.

StreamingDiskANN index

The StreamingDiskANN index is a graph-based algorithm that was inspired by the DiskANN algorithm. You can read more about it in How We Made Postgres as Fast as Pinecone for Vector Data.

To create an index named document_embedding_idx on table document_embedding having a vector column named embedding, with cosine distance metric, run:

CREATE INDEX document_embedding_cos_idx ON document_embedding
USING diskann (embedding vector_cosine_ops);

Since this index uses cosine distance, you should use the <=> operator in your queries. StreamingDiskANN also supports L2 distance:

CREATE INDEX document_embedding_l2_idx ON document_embedding
USING diskann (embedding vector_l2_ops);

For L2 distance, use the <-> operator in queries.

These examples create the index with smart defaults for all parameters not listed. These should be the right values for most cases. But if you want to delve deeper, the available parameters are below.

StreamingDiskANN index build-time parameters

These parameters can be set when an index is created.

Parameter name	Description	Default value
storage_layout	memory_optimized which uses SBQ to compress vector data or plain which stores data uncompressed	memory_optimized
num_neighbors	Sets the maximum number of neighbors per node. Higher values increase accuracy but make the graph traversal slower.	50
search_list_size	This is the S parameter used in the greedy search algorithm used during construction. Higher values improve graph quality at the cost of slower index builds.	100
max_alpha	Is the alpha parameter in the algorithm. Higher values improve graph quality at the cost of slower index builds.	1.2
num_dimensions	The number of dimensions to index. By default, all dimensions are indexed. But you can also index less dimensions to make use of Matryoshka embeddings	0 (all dimensions)
num_bits_per_dimension	Number of bits used to encode each dimension when using SBQ	2 for less than 900 dimensions, 1 otherwise

An example of how to set the num_neighbors parameter is:

CREATE INDEX document_embedding_idx ON document_embedding
USING diskann (embedding) WITH(num_neighbors=50);

StreamingDiskANN query-time parameters

You can also set two parameters to control the accuracy vs. query speed trade-off at query time. We suggest adjusting diskann.query_rescore to fine-tune accuracy.

Parameter name	Description	Default value
diskann.query_search_list_size	The number of additional candidates considered during the graph search.	100
diskann.query_rescore	The number of elements rescored (0 to disable rescoring)	50

You can set the value by using SET before executing a query. For example:

SET diskann.query_rescore = 400;

Note the SET command applies to the entire session (database connection) from the point of execution. You can use a transaction-local variant using LOCAL which will be reset after the end of the transaction:

BEGIN;
SET LOCAL diskann.query_search_list_size= 10;
SELECT * FROM document_embedding ORDER BY embedding <=> $1 LIMIT 10
COMMIT;

StreamingDiskANN index-supported queries

You need to use the cosine-distance embedding measure (<=>) in your ORDER BY clause. A canonical query would be:

SELECT *
FROM document_embedding
ORDER BY embedding <=> $1
LIMIT 10

pgvector HNSW

Pgvector provides a graph-based indexing algorithm based on the popular HNSW algorithm.

To create an index named document_embedding_idx on table document_embedding having a vector column named embedding, run:

CREATE INDEX document_embedding_idx ON document_embedding
USING hnsw(embedding vector_cosine_ops);

This command creates an index for cosine-distance queries because of vector_cosine_ops. There are also "ops" classes for Euclidean distance and negative inner product:

Distance type	Query operator	Index ops class
Cosine / Angular	<=>	vector_cosine_ops
Euclidean / L2	<->	vector_ip_ops
Negative inner product	<#>	vector_l2_ops

Pgvector HNSW also includes several index build-time and query-time parameters.

pgvector HNSW index build-time parameters

These parameters can be set at index build time:

Parameter name	Description	Default value
m	Represents the maximum number of connections per layer. Think of these connections as edges created for each node during graph construction. Increasing m increases accuracy but also increases index build time and size.	16
ef_construction	Represents the size of the dynamic candidate list for constructing the graph. It influences the trade-off between index quality and construction speed. Increasing ef_construction enables more accurate search results at the expense of lengthier index build times.	64

An example of how to set the m parameter is:

CREATE INDEX document_embedding_idx ON document_embedding
USING hnsw(embedding vector_cosine_ops) WITH (m = 20);

pgvector HNSW query-time parameters

You can also set a parameter to control the accuracy vs. query speed trade-off at query time. The parameter is called hnsw.ef_search. This parameter specifies the size of the dynamic candidate list used during search. Defaults to 40. Higher values improve query accuracy while making the query slower.

You can set the value by running:

SET hnsw.ef_search = 100;

Before executing the query, note the SET command applies to the entire session (database connection) from the point of execution. You can use a transaction-local variant using LOCAL:

BEGIN;
SET LOCAL hnsw.ef_search = 100;
SELECT * FROM document_embedding ORDER BY embedding <=> $1 LIMIT 10
COMMIT;

pgvector HNSW index-supported queries

You need to use the distance operator (<=>, <->, or <#>) matching the ops class you used during index creation in your ORDER BY clause. A canonical query would be:

SELECT *
FROM document_embedding
ORDER BY embedding <=> $1
LIMIT 10

pgvector ivfflat

Pgvector provides a clustering-based indexing algorithm. The blog post describes how it works in detail. It provides the fastest index-build speed but the slowest query speeds of any indexing algorithm.

To create an index named document_embedding_idx on table document_embedding having a vector column named embedding, run:

CREATE INDEX document_embedding_idx ON document_embedding
USING ivfflat(embedding vector_cosine_ops) WITH (lists = 100);

This command creates an index for cosine-distance queries because of vector_cosine_ops. There are also "ops" classes for Euclidean distance and negative inner product:

Distance type	Query operator	Index ops class
Cosine / Angular	<=>	vector_cosine_ops
Euclidean / L2	<->	vector_ip_ops
Negative inner product	<#>	vector_l2_ops

Note: ivfflat should never be created on empty tables because it needs to cluster data, and that only happens when an index is first created, not when new rows are inserted or modified. Also, if your table undergoes a lot of modifications, you need to rebuild this index occasionally to maintain good accuracy. See the blog post for details.

Pgvector ivfflat has a lists index parameter that should be set. See the next section.

pgvector ivfflat index build-time parameters

Pgvector has a lists parameter that should be set as follows: For datasets with less than one million rows, use lists = rows / 1000. For datasets with more than one million rows, use lists = sqrt(rows). It is generally advisable to have at least 10 clusters.

You can use the following code to simplify creating ivfflat indexes:

def create_ivfflat_index(conn, table_name, column_name, query_operator="<=>"):
    index_method = "invalid"
    if query_operator == "<->":
        index_method = "vector_l2_ops"
    elif query_operator == "<#>":
        index_method = "vector_ip_ops"
    elif query_operator == "<=>":
        index_method = "vector_cosine_ops"
    else:
        raise ValueError(f"unrecognized operator {query_operator}")

    with conn.cursor() as cur:
        cur.execute(f"SELECT COUNT(*) as cnt FROM {table_name};")
        num_records = cur.fetchone()[0]

        num_lists = num_records / 1000
        if num_lists < 10:
            num_lists = 10
        if num_records > 1000000:
            num_lists = math.sqrt(num_records)

        cur.execute(f'CREATE INDEX ON {table_name} USING ivfflat ({column_name} {index_method}) WITH (lists = {num_lists});')
        conn.commit()

pgvector ivfflat query-time parameters

You can also set a parameter to control the accuracy vs. query speed tradeoff at query time. The parameter is called ivfflat.probes. This parameter specifies the number of clusters searched during a query. It is recommended to set this parameter to sqrt(lists) where lists is the parameter used above during index creation. Higher values improve query accuracy while making the query slower.

You can set the value by running:

SET ivfflat.probes = 100;

Before executing the query, note the SET command applies to the entire session (database connection) from the point of execution. You can use a transaction-local variant using LOCAL:

BEGIN;
SET LOCAL ivfflat.probes = 100;
SELECT * FROM document_embedding ORDER BY embedding <=> $1 LIMIT 10
COMMIT;

pgvector ivfflat index-supported queries

You need to use the distance operator (<=>, <->, or <#>) matching the ops class you used during index creation in your ORDER BY clause. A canonical query would be:

SELECT *
FROM document_embedding
ORDER BY embedding <=> $1
LIMIT 10

===== PAGE: https://docs.tigerdata.com/ai/python-interface-for-pgvector-and-timescale-vector/ =====

Python interface for pgvector and pgvectorscale

You use pgai to power production grade AI applications. timescale_vector is the Python interface you use to interact with a pgai on Tiger Cloud service programmatically.

Before you get started with timescale_vector:

• Sign up for pgai on Tiger Cloud: Get 90 days free to try pgai on Tiger Cloud.
• Follow the Get Started Tutorial: Learn how to use pgai on Tiger Cloud for semantic search on a real-world dataset.

If you prefer to use an LLM development or data framework, see pgai's integrations with LangChain and LlamaIndex.

Prerequisites

timescale_vector depends on the source distribution of psycopg2 and adheres to best practices for psycopg2.

Before you install timescale_vector:

• Follow the psycopg2 build prerequisites.

Install

To interact with pgai on Tiger Cloud using Python:

1. Install timescale_vector:

   pip install timescale_vector
   

2. Install dotenv:

   pip install python-dotenv
   

   In these examples, you use dotenv to pass secrets and keys.

That is it, you are ready to go.

Basic usage of the timescale_vector library

First, import all the necessary libraries:

from dotenv import load_dotenv, find_dotenv
import os
from timescale_vector import client
import uuid
from datetime import datetime, timedelta

Load up your Postgres credentials, the safest way is with a .env file:

_ = load_dotenv(find_dotenv(), override=True)
service_url  = os.environ['TIMESCALE_SERVICE_URL']

Next, create the client. This tutorial, uses the sync client. But the library has an async client as well (with an identical interface that uses async functions).

The client constructor takes three required arguments:

name	description
service_url	Tiger Cloud service URL / connection string
table_name	Name of the table to use for storing the embeddings. Think of this as the collection name
num_dimensions	Number of dimensions in the vector

vec  = client.Sync(service_url, "my_data", 2)

Next, create the tables for the collection:

vec.create_tables()

Next, insert some data. The data record contains:

• A UUID to uniquely identify the embedding
• A JSON blob of metadata about the embedding
• The text the embedding represents
• The embedding itself

Because this data includes UUIDs which become primary keys, upserts should be used for ingest.

vec.upsert([\
    (uuid.uuid1(), {"animal": "fox"}, "the brown fox", [1.0,1.3]),\
    (uuid.uuid1(), {"animal": "fox", "action":"jump"}, "jumped over the", [1.0,10.8]),\
])

You can now create a vector index to speed up similarity search:

vec.create_embedding_index(client.TimescaleVectorIndex())

Then, you can query for similar items:

vec.search([1.0, 9.0])

[[UUID('73d05df0-84c1-11ee-98da-6ee10b77fd08'),
  {'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456],
 [UUID('73d05d6e-84c1-11ee-98da-6ee10b77fd08'),
  {'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]

There are many search options which are covered below in the Advanced search section.

A simple search example that returns one item using a similarity search constrained by a metadata filter is shown below:

vec.search([1.0, 9.0], limit=1, filter={"action": "jump"})

[[UUID('73d05df0-84c1-11ee-98da-6ee10b77fd08'),
  {'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]

The returned records contain 5 fields:

name	description
id	The UUID of the record
metadata	The JSON metadata associated with the record
contents	the text content that was embedded
embedding	The vector embedding
distance	The distance between the query embedding and the vector

You can access the fields by simply using the record as a dictionary keyed on the field name:

records = vec.search([1.0, 9.0], limit=1, filter={"action": "jump"})
(records[0]["id"],records[0]["metadata"], records[0]["contents"], records[0]["embedding"], records[0]["distance"])

(UUID('73d05df0-84c1-11ee-98da-6ee10b77fd08'),
 {'action': 'jump', 'animal': 'fox'},
 'jumped over the',
 array([ 1. , 10.8], dtype=float32),
 0.00016793422934946456)

You can delete by ID:

vec.delete_by_ids([records[0]["id"]])

Or you can delete by metadata filters:

vec.delete_by_metadata({"action": "jump"})

To delete all records use:

vec.delete_all()

Advanced usage

This section goes into more detail about the Python interface. It covers:

1. Search filter options - how to narrow your search by additional constraints
2. Indexing - how to speed up your similarity queries
3. Time-based partitioning - how to optimize similarity queries that filter on time
4. Setting different distance types to use in distance calculations

Search options

The search function is very versatile and allows you to search for the right vector in a wide variety of ways. This section describes the search option in 3 parts:

1. Basic similarity search.
2. How to filter your search based on the associated metadata.
3. Filtering on time when time-partitioning is enabled.

The following examples are based on this data:

vec.upsert([\
    (uuid.uuid1(), {"animal":"fox", "action": "sit", "times":1}, "the brown fox", [1.0,1.3]),\
    (uuid.uuid1(),  {"animal":"fox", "action": "jump", "times":100}, "jumped over the", [1.0,10.8]),\
])

The basic query looks like this:

vec.search([1.0, 9.0])

[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456],
 [UUID('7487af14-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]

You could provide a limit for the number of items returned:

vec.search([1.0, 9.0], limit=1)

[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]

Narrowing your search by metadata

There are two main ways to filter results by metadata:

• filters for equality matches on metadata.
• predicates for complex conditions on metadata.

Filters are more limited in what they can express, but are also more performant. You should use filters if your use case allows it.

Using filters for equality matches

You could specify a match on the metadata as a dictionary where all keys have to match the provided values (keys not in the filter are unconstrained):

vec.search([1.0, 9.0], limit=1, filter={"action": "sit"})

[[UUID('7487af14-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]

You can also specify a list of filter dictionaries, where an item is returned if it matches any dict:

vec.search([1.0, 9.0], limit=2, filter=[{"action": "jump"}, {"animal": "fox"}])

[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456],
 [UUID('7487af14-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]

Using predicates for more advanced filtering on metadata

Predicates allow for more complex search conditions. For example, you could use greater than and less than conditions on numeric values.

vec.search([1.0, 9.0], limit=2, predicates=client.Predicates("times", ">", 1))

[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]

Predicates objects are defined by the name of the metadata key, an operator, and a value.

The supported operators are: ==, !=, <, <=, >, >=

The type of the values determines the type of comparison to perform. For example, passing in "Sam" (a string) performs a string comparison while a 10 (an int) performs an integer comparison, and a 10.0 (float) performs a float comparison. It is important to note that using a value of "10" performs a string comparison as well so it's important to use the right type. Supported Python types are: str, int, and float.

One more example with a string comparison:

vec.search([1.0, 9.0], limit=2, predicates=client.Predicates("action", "==", "jump"))

[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]

The real power of predicates is that they can also be combined using the & operator (for combining predicates with AND semantics) and |(for combining using OR semantic). So you can do:

vec.search([1.0, 9.0], limit=2, predicates=client.Predicates("action", "==", "jump") & client.Predicates("times", ">", 1))

[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]

Just for sanity, the next example shows a case where no results are returned because of predicates:

vec.search([1.0, 9.0], limit=2, predicates=client.Predicates("action", "==", "jump") & client.Predicates("times", "==", 1))

[]

And one more example where the predicates are defined as a variable and use grouping with parenthesis:

my_predicates = client.Predicates("action", "==", "jump") & (client.Predicates("times", "==", 1) | client.Predicates("times", ">", 1))
vec.search([1.0, 9.0], limit=2, predicates=my_predicates)

[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]

There is also semantic sugar for combining many predicates with AND semantics. You can pass in multiple 3-tuples to Predicates:

vec.search([1.0, 9.0], limit=2, predicates=client.Predicates(("action", "==", "jump"), ("times", ">", 10)))

[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]

Filter your search by time

When using time-partitioning (see below) you can very efficiently filter your search by time. Time-partitioning associates the timestamp embedded in a UUID-based ID with an embedding. First, create a collection with time partitioning and insert some data (one item from January 2018 and another in January 2019):

tpvec = client.Sync(service_url, "time_partitioned_table", 2, time_partition_interval=timedelta(hours=6))
tpvec.create_tables()

specific_datetime = datetime(2018, 1, 1, 12, 0, 0)
tpvec.upsert([\
    (client.uuid_from_time(specific_datetime), {"animal":"fox", "action": "sit", "times":1}, "the brown fox", [1.0,1.3]),\
    (client.uuid_from_time(specific_datetime+timedelta(days=365)),  {"animal":"fox", "action": "jump", "times":100}, "jumped over the", [1.0,10.8]),\
])

Then, you can filter using the timestamps by specifying a uuid_time_filter:

tpvec.search([1.0, 9.0], limit=4, uuid_time_filter=client.UUIDTimeRange(specific_datetime, specific_datetime+timedelta(days=1)))

[[UUID('33c52800-ef15-11e7-be03-4f1f9a1bde5a'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]

A UUIDTimeRange can specify a start_date or end_date or both(as in the example above). Specifying only the start_date or end_date leaves the other end unconstrained.

tpvec.search([1.0, 9.0], limit=4, uuid_time_filter=client.UUIDTimeRange(start_date=specific_datetime))

[[UUID('ac8be800-0de6-11e9-889a-5eec84ba8a7b'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456],
 [UUID('33c52800-ef15-11e7-be03-4f1f9a1bde5a'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]

You have the option to define whether the start and end dates are inclusive with the start_inclusive and end_inclusive parameters. Setting start_inclusive to true results in comparisons using the >= operator, whereas setting it to false applies the > operator. By default, the start date is inclusive, while the end date is exclusive. One example:

tpvec.search([1.0, 9.0], limit=4, uuid_time_filter=client.UUIDTimeRange(start_date=specific_datetime, start_inclusive=False))

[[UUID('ac8be800-0de6-11e9-889a-5eec84ba8a7b'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]

Notice how the results are different when using the start_inclusive=False option because the first row has the exact timestamp specified by start_date.

It is also easy to integrate time filters using the filter and predicates parameters described above using special reserved key names to make it appear that the timestamps are part of your metadata. This is useful when integrating with other systems that just want to specify a set of filters (often these are "auto retriever" type systems). The reserved key names are __start_date and __end_date for filters and __uuid_timestamp for predicates. Some examples below:

tpvec.search([1.0, 9.0], limit=4, filter={ "__start_date": specific_datetime, "__end_date": specific_datetime+timedelta(days=1)})

[[UUID('33c52800-ef15-11e7-be03-4f1f9a1bde5a'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]

tpvec.search([1.0, 9.0], limit=4,
             predicates=client.Predicates("__uuid_timestamp", ">", specific_datetime) & client.Predicates("__uuid_timestamp", "<", specific_datetime+timedelta(days=1)))

[[UUID('33c52800-ef15-11e7-be03-4f1f9a1bde5a'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]

Indexing

Indexing speeds up queries over your data. By default, the system creates indexes to query your data by the UUID and the metadata.

To speed up similarity search based on the embeddings, you have to create additional indexes.

Note that if performing a query without an index, you always get an exact result, but the query is slow (it has to read all of the data you store for every query). With an index, your queries are order-of-magnitude faster, but the results are approximate (because there are no known indexing techniques that are exact).

Luckily, TimescaleDB provides 3 excellent approximate indexing algorithms, StreamingDiskANN, HNSW, and ivfflat.

Below are the trade-offs between these algorithms:

Algorithm	Build speed	Query speed	Need to rebuild after updates
StreamingDiskAnn	Fast	Fastest	No
HNSW	Fast	Faster	No
ivfflat	Fastest	Slowest	Yes

You can see benchmarks on the blog.

You should use the StreamingDiskANN index for most use cases. This can be created with:

vec.create_embedding_index(client.TimescaleVectorIndex())

Indexes are created for a particular distance metric type. So it is important that the same distance metric is set on the client during index creation as it is during queries. See the distance type section below.

Each of these indexes has a set of build-time options for controlling the speed/accuracy trade-off when creating the index and an additional query-time option for controlling accuracy during a particular query. The library uses smart defaults for all of these options. The details for how to adjust these options manually are below.

StreamingDiskANN index

The StreamingDiskANN index is a graph-based algorithm that uses the DiskANN algorithm. You can read more about it in the blog announcing its release.

To create this index, run:

vec.create_embedding_index(client.TimescaleVectorIndex())

The above command creates the index using smart defaults. There are a number of parameters you could tune to adjust the accuracy/speed trade-off.

The parameters you can set at index build time are:

Parameter name	Description	Default value
num_neighbors	Sets the maximum number of neighbors per node. Higher values increase accuracy but make the graph traversal slower.	50
search_list_size	This is the S parameter used in the greedy search algorithm used during construction. Higher values improve graph quality at the cost of slower index builds.	100
max_alpha	Is the alpha parameter in the algorithm. Higher values improve graph quality at the cost of slower index builds.	1.0

To set these parameters, you could run:

vec.create_embedding_index(client.TimescaleVectorIndex(num_neighbors=50, search_list_size=100, max_alpha=1.0))

You can also set a parameter to control the accuracy vs. query speed trade-off at query time. The parameter is set in the search() function using the query_params argument. You can set the search_list_size(default: 100). This is the number of additional candidates considered during the graph search at query time. Higher values improve query accuracy while making the query slower.

You can specify this value during search as follows:

vec.search([1.0, 9.0], limit=4, query_params=TimescaleVectorIndexParams(search_list_size=10))

To drop the index, run:

vec.drop_embedding_index()

pgvector HNSW index

Pgvector provides a graph-based indexing algorithm based on the popular HNSW algorithm.

To create this index, run:

vec.create_embedding_index(client.HNSWIndex())

The above command creates the index using smart defaults. There are a number of parameters you could tune to adjust the accuracy/speed trade-off.

The parameters you can set at index build time are:

Parameter name	Description	Default value
m	Represents the maximum number of connections per layer. Think of these connections as edges created for each node during graph construction. Increasing m increases accuracy but also increases index build time and size.	16
ef_construction	Represents the size of the dynamic candidate list for constructing the graph. It influences the trade-off between index quality and construction speed. Increasing ef_construction enables more accurate search results at the expense of lengthier index build times.	64

To set these parameters, you could run:

vec.create_embedding_index(client.HNSWIndex(m=16, ef_construction=64))

You can also set a parameter to control the accuracy vs. query speed trade-off at query time. The parameter is set in the search() function using the query_params argument. You can set the ef_search(default: 40). This parameter specifies the size of the dynamic candidate list used during search. Higher values improve query accuracy while making the query slower.

You can specify this value during search as follows:

vec.search([1.0, 9.0], limit=4, query_params=HNSWIndexParams(ef_search=10))

To drop the index run:

vec.drop_embedding_index()

pgvector ivfflat index

Pgvector provides a clustering-based indexing algorithm. The blog post describes how it works in detail. It provides the fastest index-build speed but the slowest query speeds of any indexing algorithm.

To create this index, run:

vec.create_embedding_index(client.IvfflatIndex())

Note: ivfflat should never be created on empty tables because it needs to cluster data, and that only happens when an index is first created, not when new rows are inserted or modified. Also, if your table undergoes a lot of modifications, you need to rebuild this index occasionally to maintain good accuracy. See the blog post for details.

Pgvector ivfflat has a lists index parameter that is automatically set with a smart default based on the number of rows in your table. If you know that you'll have a different table size, you can specify the number of records to use for calculating the lists parameter as follows:

vec.create_embedding_index(client.IvfflatIndex(num_records=1000000))

You can also set the lists parameter directly:

vec.create_embedding_index(client.IvfflatIndex(num_lists=100))

You can also set a parameter to control the accuracy vs. query speed trade-off at query time. The parameter is set in the search() function using the query_params argument. You can set the probes. This parameter specifies the number of clusters searched during a query. It is recommended to set this parameter to sqrt(lists) where lists is the num_list parameter used above during index creation. Higher values improve query accuracy while making the query slower.

You can specify this value during search as follows:

vec.search([1.0, 9.0], limit=4, query_params=IvfflatIndexParams(probes=10))

To drop the index, run:

vec.drop_embedding_index()

Time partitioning

In many use cases where you have many embeddings, time is an important component associated with the embeddings. For example, when embedding news stories, you often search by time as well as similarity (for example, stories related to Bitcoin in the past week or stories about Clinton in November 2016).

Yet, traditionally, searching by two components "similarity" and "time" is challenging for Approximate Nearest Neighbor (ANN) indexes and makes the similarity-search index less effective.

One approach to solving this is partitioning the data by time and creating ANN indexes on each partition individually. Then, during search, you can:

• Step 1: filter partitions that don't match the time predicate.
• Step 2: perform the similarity search on all matching partitions.
• Step 3: combine all the results from each partition in step 2, re-rank, and filter out results by time.

Step 1 makes the search a lot more efficient by filtering out whole swaths of data in one go.

Timescale-vector supports time partitioning using TimescaleDB's hypertables. To use this feature, simply indicate the length of time for each partition when creating the client:

from datetime import timedelta
from datetime import datetime

vec = client.Async(service_url, "my_data_with_time_partition", 2, time_partition_interval=timedelta(hours=6))
await vec.create_tables()

Then, insert data where the IDs use UUIDs v1 and the time component of the UUIDspecifies the time of the embedding. For example, to create an embedding for the current time, simply do:

id = uuid.uuid1()
await vec.upsert([(id, {"key": "val"}, "the brown fox", [1.0, 1.2])])

To insert data for a specific time in the past, create the UUID using the uuid_from_time function

specific_datetime = datetime(2018, 8, 10, 15, 30, 0)
await vec.upsert([(client.uuid_from_time(specific_datetime), {"key": "val"}, "the brown fox", [1.0, 1.2])])

You can then query the data by specifying a uuid_time_filter in the search call:

rec = await vec.search([1.0, 2.0], limit=4, uuid_time_filter=client.UUIDTimeRange(specific_datetime-timedelta(days=7), specific_datetime+timedelta(days=7)))

Distance metrics

Cosine distance is used by default to measure how similarly an embedding is to a given query. In addition to cosine distance, Euclidean/L2 distance is also supported. The distance type is set when creating the client using the distance_type parameter. For example, to use the Euclidean distance metric, you can create the client with:

vec  = client.Sync(service_url, "my_data", 2, distance_type="euclidean")

Valid values for distance_type are cosine and euclidean.

It is important to note that you should use consistent distance types on clients that create indexes and perform queries. That is because an index is only valid for one particular type of distance measure.

Note that the StreamingDiskANN index only supports cosine distance at this time.

===== PAGE: https://docs.tigerdata.com/ai/langchain-integration-for-pgvector-and-timescale-vector/ =====

LangChain Integration for pgvector, pgvectorscale, and pgai

LangChain is a popular framework for development applications powered by LLMs. pgai on Tiger Cloud has a native LangChain integration, enabling you to use it as a vector store and leverage all its capabilities in your applications built with LangChain.

Here are resources about using pgai on Tiger Cloud with LangChain:

• Getting started with LangChain and pgvectorscale: You'll learn how to use pgai on Tiger Data for (1) semantic search, (2) time-based vector search, (3) self-querying, and (4) how to create indexes to speed up queries.
• Postgres Self Querying: Learn how to use pgai on Tiger Data with self-querying in LangChain.
• Learn more about pgai on Tiger Data and LangChain: A blog post about the unique capabilities that pgai on Tiger Cloud brings to the LangChain ecosystem.

===== PAGE: https://docs.tigerdata.com/ai/llamaindex-integration-for-pgvector-and-timescale-vector/ =====

LlamaIndex Integration for pgvector and Tiger Data Vector

LlamaIndex integration for pgvector and Tiger Data Vector

LlamaIndex is a popular data framework for connecting custom data sources to large language models (LLMs). Tiger Data Vector has a native LlamaIndex integration that supports all the features of pgvector and Tiger Data Vector. It enables you to use Tiger Data Vector as a vector store and leverage all its capabilities in your applications built with LlamaIndex.

Here are resources about using Tiger Data Vector with LlamaIndex:

• Getting started with LlamaIndex and TigerData Vector: You'll learn how to use Tiger Data Vector for (1) similarity search, (2) time-based vector search, (3) faster search with indexes, and (4) retrieval and query engine.
• Time-based retrieval: Learn how to power RAG applications with time-based retrieval.
• Llama Pack: Auto Retrieval with time-based search: This pack demonstrates performing auto-retrieval for hybrid search based on both similarity and time, using the timescale-vector (Postgres) vector store.
• Learn more about TigerData Vector and LlamaIndex : How Tiger Data Vector is a better Postgres for AI applications.

===== PAGE: https://docs.tigerdata.com/ai/pgvectorizer/ =====

Embed your Postgres data with PgVectorizer

Embed Postgres data with PgVectorizer

PgVectorizer enables you to create vector embeddings from any data that you already have stored in Postgres. You can get more background information in the blog post announcing this feature, as well as the "how we built it" post going into the details of the design.

To create vector embeddings, simply attach PgVectorizer to any Postgres table to automatically sync that table's data with a set of embeddings stored in Postgres. For example, say you have a blog table defined in the following way:

import psycopg2
from langchain.docstore.document import Document
from langchain.text_splitter import CharacterTextSplitter
from timescale_vector import client, pgvectorizer
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores.timescalevector import TimescaleVector
from datetime import timedelta

with psycopg2.connect(service_url) as conn:
    with conn.cursor() as cursor:
        cursor.execute('''
        CREATE TABLE IF NOT EXISTS blog (
            id              INT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
            title           TEXT NOT NULL,
            author          TEXT NOT NULL,
            contents        TEXT NOT NULL,
            category        TEXT NOT NULL,
            published_time  TIMESTAMPTZ NULL --NULL if not yet published
        );
        ''')

You can insert some data as follows:

with psycopg2.connect(service_url) as conn:
    with conn.cursor() as cursor:
        cursor.execute('''
            INSERT INTO blog (title, author, contents, category, published_time) VALUES ('First Post', 'Matvey Arye', 'some super interesting content about cats.', 'AI', '2021-01-01');
        ''')

Now, say you want to embed these blogs and store the embeddings in Postgres. First, you need to define an embed_and_write function that takes a set of blog posts, creates the embeddings, and writes them into TigerData Vector. For example, if using LangChain, it could look something like the following.

def get_document(blog):
    text_splitter = CharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
    )
    docs = []
    for chunk in text_splitter.split_text(blog['contents']):
        content = f"Author {blog['author']}, title: {blog['title']}, contents:{chunk}"
        metadata = {
            "id": str(client.uuid_from_time(blog['published_time'])),
            "blog_id": blog['id'],
            "author": blog['author'],
            "category": blog['category'],
            "published_time": blog['published_time'].isoformat(),
        }
        docs.append(Document(page_content=content, metadata=metadata))
    return docs

def embed_and_write(blog_instances, vectorizer):
    embedding = OpenAIEmbeddings()
    vector_store = TimescaleVector(
        collection_name="blog_embedding",
        service_url=service_url,
        embedding=embedding,
        time_partition_interval=timedelta(days=30),
    )

    metadata_for_delete = [{"blog_id": blog['locked_id']} for blog in blog_instances]
    vector_store.delete_by_metadata(metadata_for_delete)

    documents = []
    for blog in blog_instances:
        if blog['published_time'] != None:
            documents.extend(get_document(blog))

    if len(documents) == 0:
        return

    texts = [d.page_content for d in documents]
    metadatas = [d.metadata for d in documents]
    ids = [d.metadata["id"] for d in documents]
    vector_store.add_texts(texts, metadatas, ids)

Then, all you have to do is run the following code in a scheduled job (cron job, Lambda job, etc):

vectorizer = pgvectorizer.Vectorize(service_url, 'blog')
while vectorizer.process(embed_and_write) > 0:
    pass

Every time that job runs, it syncs the table with your embeddings. It syncs all inserts, updates, and deletes to an embeddings table called blog_embedding.

Now, you can simply search the embeddings as follows (again, using LangChain in the example):

embedding = OpenAIEmbeddings()
vector_store = TimescaleVector(
    collection_name="blog_embedding",
    service_url=service_url,
    embedding=embedding,
    time_partition_interval=timedelta(days=30),
)

res = vector_store.similarity_search_with_score("Blogs about cats")
res

[(Document(page_content='Author Matvey Arye, title: First Post, contents:some super interesting content about cats.', metadata={'id': '4a784000-4bc4-11eb-855a-06302dbc8ce7', 'author': 'Matvey Arye', 'blog_id': 1, 'category': 'AI', 'published_time': '2021-01-01T00:00:00+00:00'}),
  0.12595687795193833)]

===== PAGE: https://docs.tigerdata.com/README/ =====

Tiger Cloud is the modern Postgres data platform for all your applications. It enhances Postgres to handle time series, events, real-time analytics, and vector search—all in a single database alongside transactional workloads.

This repository contains the current source for Tiger Data documentation available at https://docs.tigerdata.com/.

We welcome contributions! You can contribute to Tiger Data documentation in the following ways:

• Create an issue in this repository and describe the proposed change. Our doc team takes care of it.
• Update the docs yourself and have your change reviewed and published by our doc team.

Contribute to the Tiger Data docs

To make the contribution yourself:

1. Get the documentation source:

   • No write access? Fork this repository.
   • Already have a write access? Clone this repository.

2. Create a branch from latest, make your changes, and raise a pull request back to latest.

3. Sign a Contributor License Agreement (CLA).

   You have to sign the CLA only the first time you raise a PR. This helps to ensure that the community is free to use your contributions.

4. Review your changes.

   The documentation site is generated in a separate private repository using Gatsby. Once you raise a PR for any branch, GitHub automatically generates a preview for your changes and attaches the link in the comments. Any new commits are visible at the same URL. If you don't see the latest changes, try an incognito browser window. Automated builds are not available for PRs from forked repositories.

See the Contributing guide for style and language guidance.

Learn about Tiger Data

Tiger Data is Postgres made powerful. To learn more about the company and its products, visit tigerdata.com.

===== PAGE: https://docs.tigerdata.com/CONTRIBUTING/ =====

Contribute to Tiger Data documentation

Tiger Data documentation is open for contribution from all community members. The current source is in this repository.

This page explains the structure and language guidelines for contributing to Tiger Data documentation. See the README for how to contribute.

Language

Write in a clear, concise, and actionable manner. Tiger Data documentation uses the Google Developer Documentation Style Guide with the following exceptions:

• Do not capitalize the first word after a colon.
• Use code font (back ticks) for UI elements instead of semi-bold.

Edit individual pages

Each major doc section has a dedicated directory with .md files inside, representing its child pages. This includes an index.md file that serves as a landing page for that doc section by default, unless specifically changed in the navigation tree. To edit a page, modify the corresponding .md file following these recommendations:

• Regular pages should include:

  • A short intro describing the main subject of the page.
  • A visual illustrating the main concept, if relevant.
  • Paragraphs with descriptive headers, organizing the content into logical sections.
  • Procedures to describe the sequence of steps to reach a certain goal. For example, create a Tiger Cloud service.
  • Other visual aids, if necessary.
  • Links to other relevant resources.

• API pages should include:

  • The function name, with empty parentheses if it takes arguments.
  • A brief, specific description of the function, including any possible warnings.
  • One or two samples of the function being used to demonstrate argument syntax.
  • An argument table with Name, Type, Default, Required, Description columns.
  • A return table with Column, Type, and Description columns.

• Troubleshooting pages are not written as whole Markdown files, but are programmatically assembled from individual files in the_troubleshooting folder. Each entry describes a single troubleshooting case and its solution, and contains the following front matter:

  Key	Type	Required	Description
  title	string	✅	The title of the troubleshooting entry, displayed as a heading above it
  section	The literal string troubleshooting	✅	Must be troubleshooting, used to identify troubleshooting entries during site build
  products or topics	array of strings	✅ (can have either or both, but must have at least one)	The products or topics related to the entry. The entry shows up on the troubleshooting pages for the listed products and topics.
  errors	object of form {language: string, message: string}	❌	The error, if any, related to the troubleshooting entry. Displayed as a code block right underneath the title. language is the programming language to use for syntax highlighting.
  keywords	array of strings	❌	These are displayed at the bottom of every troubleshooting page. Each keyword links to a collection of all pages associated with that keyword.
  tags	array of strings	❌	Concepts, actions, or things associated with the troubleshooting entry. These are not displayed in the UI, but they affect the calculation of related pages.

  Beneath the front matter, describe the error and its solution in regular Markdown. You can also use any other components allowed within the docs site.

  The entry shows up on the troubleshooting pages for its associated products and topics. If the page doesn't already exist, add an entry for it in the page index, setting type to placeholder. See Navigation tree.

Edit the navigation hierarchy

The navigation hierarchy of a doc section is governed by page-index/page-index.js within the corresponding directory. For example:

     {
        title: "Tiger Cloud services",
        href: "services",
        excerpt: "About Tiger Cloud services",
        children: [
          {
            title: "Services overview",
            href: "service-overview",
            excerpt: "Tiger Cloud services overview",
          },
          {
            title: "Service explorer",
            href: "service-explorer",
            excerpt: "Tiger Cloud services explorer",
          },
          {
            title: "Troubleshooting Tiger Cloud services",
            href: "troubleshooting",
            type: "placeholder",
          },
        ],
      },

See Use Tiger Cloud section navigation for reference.

To change the structure, add or delete pages in a section, modify the corresponding page-index.js. An entry in a page-index.js includes the following fields:

Key	Type	Required	Description
href	string	✅	The URL segment to use for the page. If there is a corresponding Markdown file, href must match the name of the Markdown file, minus the file extension.
title	string	✅	The title of the page, used as the page name within the TOC on the left. Must be the same as the first header in the corresponding Markdown file.
excerpt	string	✅	The short description of the page, used for the page card if pageComponents is set to featured-cards. Should be up to 100 characters. See pageComponents for details.
type	One of [directory, placeholder, redirect-to-child-page]	❌	If no type is specified, the page is built as a regular webpage. The structure of its children, if present, is defined by children entries and the corresponding structure of subfolders. If the type is directory, the corresponding file becomes a directory. The difference of the directory page is that its child pages sit at the same level as the directory page. They only become children during the site build. If the type is placeholder, the corresponding page is produced programmatically upon site build. If not produced, the link in the navigation tree returns a 404. In particular, this is used for troubleshooting pages. If the type is redirect-to-child-page, no page is built and the link in the navigation tree goes directly to the first child.
children	Array of page entries	❌	Child pages of the current page. For regular pages, the children should be located in a directory with the same name as the parent. The parent is the index.md file in that directory. Fordirectory pages, the children should be located in the same directory as the parent.
pageComponents	One of [['featured-cards'], ['content-list']]	❌	Any page that has child pages can list its children in either card or list style at the bottom of the page. Specify the desired style with this key.
featuredChildren	Array of URLs	❌	Similar to pageComponents, this displays the children of the current page, but only the selected ones.
index	string	❌	If a section landing page needs to be different from the index.md file in that directory, this field specifies the corresponding Markdown file name.

Reuse text in multiple pages

Partials allow you to reuse snippets of content in multiple places. All partials live in the _partials top-level directory. To make a new partial, create a new .md file in this directory. The filename must start with an underscore. Then import it into the target page as an .mdx file and reference in the relevant place. See Formatting examples.

Formatting

In addition to all the regular Markdown formatting, the following elements are available for Tiger Data docs:

• Procedure blocks
• Highlight blocks
• Tabs
• Code blocks without line numbers and the copy button
• Multi-tab code blocks
• Tags

See Formatting examples for how to use them.

Variables

Tiger Data documentation uses variables for its product names, features, and UI elements in Tiger Cloud Console with the following syntax: $VARIABLE_NAME. Variables do not work inside the following:

• Front matter on each page
• HTML tables and tabs

See the full list of available variables.

Links

• Internal page links: internal links do not need to include the domain name https://docs.tigerdata.com. Use the :currentVersion: variable instead of latest in the URL.
• External links: input external links as is.

See Formatting examples for details.

Visuals

When adding screenshots to the docs, aim for a full-screen view to provide better context. Reduce the size of your browser so there is as little wasted space as possible.

Attach the image to your issue or PR, and the doc team uploads and inserts it for you.

SEO optimization

To make a documentation page more visible and clear for Google:

• Include the title and excerpt meta tags at the top of the page. These represent meta title and description required for SEO optimization.

  • title: up to 60 characters, a short description of the page contents. In most cases a variation of the page title.
  • excerpt: under 200 characters, a longer description of the page contents. In most cases a variation of the page intro.

• Summarize the contents of each paragraph in the first sentence of that paragraph.

• Include main page keywords into the meta tags, page title, first header, and intro. These are usually the names of features described in the page. For example, for a page dedicated to creating hypertables, you can use the keyword hypertable in the following way:

  • Title: Create a hypertable in Tiger Cloud
  • Description: Turn a regular Postgres table into a hypertable in a few steps, using Tiger Cloud Console.
  • First header: Create a hypertable

Docs for deprecated products

The previous documentation source is in the deprecated repository called docs.timescale.com-content.

===== PAGE: https://docs.tigerdata.com/mst/index/ =====

Managed Service for TimescaleDB

Managed Service for TimescaleDB (MST) is TimescaleDB hosted on Azure and GCP. MST is offered in partnership with Aiven.

Tiger Cloud is a high-performance developer focused cloud that provides Postgres services enhanced with our blazing fast vector search. You can securely integrate Tiger Cloud with your AWS, GCS or Azure infrastructure. Create a Tiger Cloud service and try for free.

If you need to run TimescaleDB on GCP or Azure, you're in the right place — keep reading.

===== PAGE: https://docs.tigerdata.com/.helper-scripts/README/ =====

README

This directory includes helper scripts for writing and editing docs content. It doesn't include scripts for building content; those are in the web-documentation repo.

Bulk editing for API frontmatter

API frontmatter metadata is stored with the API content it describes. This makes sense in most cases, but sometimes you want to bulk edit metadata or compare phrasing across all API references. There are 2 scripts to help with this. They are currently written to edit the excerpts field, but can be adapted for other fields.

extract_excerpts.sh

This extracts the excerpt from every API reference into a single file named extracted_excerpts.md.

To use:

1. cd into the _scripts/ directory.
2. If you already have an extracted_excerpts.md file from a previous run, delete it.
3. Run ./extract_excerpts.sh.
4. Open extracted_excerpts.md and edit the excerpts directly within the file. Only change the actual excerpts, not the filename or excerpt: label. Otherwise, the next script fails.

insert_excerpts.sh

This takes the edited excerpts from extracted_excerpts.md and updates the original files with the new edits. A backup is created so the data is saved if something goes horribly wrong. (If something goes wrong with the backup, you can always also restore from git.)

To use:

1. Save your edited extracted_excerpts.md.
2. Make sure you are in the _scripts/ directory.
3. Run ./insert_excerpts.sh.
4. Run git diff to double-check that the update worked correctly.
5. Delete the unnecessary backups.

===== PAGE: https://docs.tigerdata.com/navigation/index/ =====

Find a docs page

Looking for information on something specific? There are several ways to find it:

1. For help with the Tiger Cloud Console, try the Tiger Cloud Console index.
2. For help on a specific topic, try browsing by keyword.
3. Or try the full search, which also returns results from the Tiger Data blog and forum.

===== PAGE: https://docs.tigerdata.com/about/index/ =====

About Tiger Data products

===== PAGE: https://docs.tigerdata.com/use-timescale/index/ =====

Use Tiger Data products

This section contains information about using TimescaleDB and Tiger Cloud. If you're not sure how to find the information you need, try the Find a docs page section.

===== PAGE: https://docs.tigerdata.com/use-timescale/OLD-cloud-multi-node/ =====

Multi-node

If you have a larger workload, you might need more than one Timescale instance. Multi-node can give you faster data ingest, and more responsive and efficient queries for many large workloads.

This section shows you how to use multi-node on Timescale. You can also set up multi-node on self-hosted TimescaleDB.

Early access: TimescaleDB v2.18.0

In some cases, your processing speeds could be slower in a multi-node cluster, because distributed hypertables need to push operations down to the various data nodes. It is important that you understand multi-node architecture before you begin, and plan your database according to your specific environment.

Set up multi-node

To create a multi-node cluster, you need an access node that stores metadata for the distributed hypertable and performs query planning across the cluster, and any number of data nodes that store subsets of the distributed hypertable dataset and run queries locally.

Setting up multi-node

1. Log in to your Tiger Cloud account and click Create Service.
2. Click Advanced configuration.
3. Under Choose your architecture, click Multi-node.
4. The customer support team contacts you. When your request is approved, return to the screen for creating a multi-node service.
5. Choose your preferred region, or accept the default region of us-east-1.
6. Accept the default for the data nodes, or click Edit to choose the number of data nodes, and their compute and disk size.
7. Accept the default for the access node, or click Edit to choose the compute and disk size.
8. Click Create service. Take a note of the service information, you need these details to connect to your multi-node cluster. The service takes a few minutes to start up.
9. When the service is ready, you can see the service in the Service Overview page. Click on the name of your new multi-node service to see more information, and to make changes.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_migration_rds_roles/ =====

pg_dumpall -d "source" \
  --quote-all-identifiers \
  --roles-only \
  --no-role-passwords \
  --file=roles.sql

AWS RDS does not permit dumping of roles with passwords, which is why the above command is executed with the --no-role-passwords. However, when the migration of roles to your Tiger Cloud service is complete, you need to manually assign passwords to the necessary roles using the following command:ALTER ROLE name WITH PASSWORD 'password';

Tiger Cloud services do not support roles with superuser access. If your SQL dump includes roles that have such permissions, you'll need to modify the file to be compliant with the security model.

You can use the following sed command to remove unsupported statements and permissions from your roles.sql file:

sed -i -E \
-e '/CREATE ROLE "postgres";/d' \
-e '/ALTER ROLE "postgres"/d' \
-e '/CREATE ROLE "rds/d' \
-e '/ALTER ROLE "rds/d' \
-e '/TO "rds/d' \
-e '/GRANT "rds/d' \
-e 's/(NO)*SUPERUSER//g' \
-e 's/(NO)*REPLICATION//g' \
-e 's/(NO)*BYPASSRLS//g' \
-e 's/GRANTED BY "[^"]*"//g' \
roles.sql

This command works only with the GNU implementation of sed (sometimes referred to as gsed). For the BSD implementation (the default on macOS), you need to add an extra argument to change the -i flag to -i ''.

To check the sed version, you can use the command sed --version. While the GNU version explicitly identifies itself as GNU, the BSD version of sed generally doesn't provide a straightforward --version flag and simply outputs an "illegal option" error.

A brief explanation of this script is:

• CREATE ROLE "postgres"; and ALTER ROLE "postgres": These statements are removed because they require superuser access, which is not supported by Timescale.

• (NO)SUPERUSER | (NO)REPLICATION | (NO)BYPASSRLS: These are permissions that require superuser access.

• CREATE ROLE "rds, ALTER ROLE “rds, TO "rds, GRANT "rds: Any creation or alteration of rds prefixed roles are removed because of their lack of any use in a Tiger Cloud service. Similarly, any grants to or from "rds" prefixed roles are ignored as well.

• GRANTED BY role_specification: The GRANTED BY clause can also have permissions that require superuser access and should therefore be removed. Note: Per the TimescaleDB documentation, the GRANTOR in the GRANTED BY clause must be the current user, and this clause mainly serves the purpose of SQL compatibility. Therefore, it's safe to remove it.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_set_up_align_db_extensions_timescaledb/ =====

1. Ensure that the source and target databases are running the same version of TimescaleDB.

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target service:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

2. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

===== PAGE: https://docs.tigerdata.com/_partials/_beta/ =====

This feature is in beta. Beta features are experimental, and should not be used on production systems. If you have feedback, reach out to your customer success manager, or contact us.

===== PAGE: https://docs.tigerdata.com/_partials/_manage-a-data-exporter/ =====

Attach a data exporter to a Tiger Cloud service

To send telemetry data to an external monitoring tool, you attach a data exporter to your Tiger Cloud service. You can attach only one exporter to a service.

To attach an exporter:

1. In Tiger Cloud Console, choose the service
2. Click Operations > Exporters
3. Select the exporter, then click Attach exporter
4. If you are attaching a first Logs data type exporter, restart the service

Monitor Tiger Cloud service metrics

You can now monitor your service metrics. Use the following metrics to check the service is running correctly:

• timescale.cloud.system.cpu.usage.millicores
• timescale.cloud.system.cpu.total.millicores
• timescale.cloud.system.memory.usage.bytes
• timescale.cloud.system.memory.total.bytes
• timescale.cloud.system.disk.usage.bytes
• timescale.cloud.system.disk.total.bytes

Additionally, use the following tags to filter your results.

Tag	Example variable	Description
host	us-east-1.timescale.cloud
project-id
service-id
region	us-east-1	AWS region
role	replica or primary	For service with replicas
node-id		For multi-node services

Edit a data exporter

To update a data exporter:

1. In Tiger Cloud Console, open Exporters
2. Next to the exporter you want to edit, click the menu > Edit
3. Edit the exporter fields and save your changes

You cannot change fields such as the provider or the AWS region.

Delete a data exporter

To remove a data exporter that you no longer need:

1. Disconnect the data exporter from your Tiger Cloud services

   1. In Tiger Cloud Console, choose the service.
   2. Click Operations > Exporters.
   3. Click the trash can icon.
   4. Repeat for every service attached to the exporter you want to remove.

   The data exporter is now unattached from all services. However, it still exists in your project.

2. Delete the exporter on the project level

   1. In Tiger Cloud Console, open Exporters
   2. Next to the exporter you want to edit, click menu > Delete
   3. Confirm that you want to delete the data exporter.

Reference

When you create the IAM OIDC provider, the URL must match the region you create the exporter in. It must be one of the following:

Region	Zone	Location	URL
ap-southeast-1	Asia Pacific	Singapore	irsa-oidc-discovery-prod-ap-southeast-1.s3.ap-southeast-1.amazonaws.com
ap-southeast-2	Asia Pacific	Sydney	irsa-oidc-discovery-prod-ap-southeast-2.s3.ap-southeast-2.amazonaws.com
ap-northeast-1	Asia Pacific	Tokyo	irsa-oidc-discovery-prod-ap-northeast-1.s3.ap-northeast-1.amazonaws.com
ca-central-1	Canada	Central	irsa-oidc-discovery-prod-ca-central-1.s3.ca-central-1.amazonaws.com
eu-central-1	Europe	Frankfurt	irsa-oidc-discovery-prod-eu-central-1.s3.eu-central-1.amazonaws.com
eu-west-1	Europe	Ireland	irsa-oidc-discovery-prod-eu-west-1.s3.eu-west-1.amazonaws.com
eu-west-2	Europe	London	irsa-oidc-discovery-prod-eu-west-2.s3.eu-west-2.amazonaws.com
sa-east-1	South America	São Paulo	irsa-oidc-discovery-prod-sa-east-1.s3.sa-east-1.amazonaws.com
us-east-1	United States	North Virginia	irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com
us-east-2	United States	Ohio	irsa-oidc-discovery-prod-us-east-2.s3.us-east-2.amazonaws.com
us-west-2	United States	Oregon	irsa-oidc-discovery-prod-us-west-2.s3.us-west-2.amazonaws.com

===== PAGE: https://docs.tigerdata.com/_partials/_early_access_2_18_0/ =====

Early access: TimescaleDB v2.18.0

===== PAGE: https://docs.tigerdata.com/_partials/_multi-node-deprecation/ =====

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_prerequisites/ =====

Best practice is to use an Ubuntu EC2 instance hosted in the same region as your Tiger Cloud service to move data. That is, the machine you run the commands on to move your data from your source database to your target Tiger Cloud service.

Before you move your data:

• Create a target Tiger Cloud service.

  Each Tiger Cloud service has a single Postgres instance that supports the most popular extensions. Tiger Cloud services do not support tablespaces, and there is no superuser associated with a service. Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window.

• To ensure that maintenance does not run while migration is in progress, best practice is to adjust the maintenance window.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_open_support_request/ =====

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-debian-based-end/ =====

1. Update your local repository list

   sudo apt update
   

2. Install TimescaleDB

   sudo apt install timescaledb-2-postgresql-17 postgresql-client-17
   

   To install a specific TimescaleDB release, set the version. For example:

   sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'

   Older versions of TimescaleDB may not support all the OS versions listed on this page.

3. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune
   

   By default, this script is included with the timescaledb-tools package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see Configuration. If you have an issue, run sudo apt install timescaledb-tools.

4. Restart Postgres

   sudo systemctl restart postgresql
   

5. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

6. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_prereqs-cloud-and-self/ =====

To follow the procedure on this page you need to:

• Create a target Tiger Cloud service.

  This procedure also works for self-hosted TimescaleDB.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_setup_environment_awsrds/ =====

Set your connection strings

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Align the extensions on the source and target

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Tune your source database

Updating parameters on a Postgres instance will cause an outage. Choose a time that will cause the least issues to tune this database.

1. Update the DB instance parameter group for your source database

   1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

   2. Click Configuration, scroll down and note the DB instance parameter group, then click Parameter groups

      

   3. Click Create parameter group, fill in the form with the following values, then click Create.

      • Parameter group name - whatever suits your fancy.
      • Description - knock yourself out with this one.
      • Engine type - PostgreSQL
      • Parameter group family - the same as DB instance parameter group in your Configuration.

   4. In Parameter groups, select the parameter group you created, then click Edit.

   5. Update the following parameters, then click Save changes.

      • rds.logical_replication set to 1: record the information needed for logical decoding.
      • wal_sender_timeout set to 0: disable the timeout for the sender process.

   6. In RDS, navigate back to your databases, select the RDS instance to migrate, and click Modify.

   7. Scroll down to Database options, select your new parameter group, and click Continue.

   8. Click Apply immediately or choose a maintenance window, then click Modify DB instance.

      Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.

   9. Verify that the settings are live in your database.

2. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_source_target_note/ =====

In the context of migrations, your existing production database is referred to as the SOURCE database, the Tiger Cloud service that you are migrating your data to is the TARGET.

===== PAGE: https://docs.tigerdata.com/_partials/_not-available-in-free-plan/ =====

This feature is not available under the Free pricing plan.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_migration_docker_subcommand/ =====

Next, download the live-migration docker image:

docker run --rm -it --name live-migration \
    -e PGCOPYDB_SOURCE_PGURI=source \
    -e PGCOPYDB_TARGET_PGURI=target \
    --pid=host \
    -v ~/live-migration:/opt/timescale/ts_cdc \
    timescale/live-migration:latest --help

Live migration moves your PostgreSQL/TimescaleDB to your Tiger Cloud service with minimal downtime.

options:
  -h, --help            Show this help message and exit
  -v, --version         Show the version of live-migration tool

Subcommands:
  {snapshot,clean,migrate}
                        Subcommand help
    snapshot            Create a snapshot
    clean               Clean up resources
    migrate             Start the migration

Live-migration contains 3 subcommands:

1. Snapshot
2. Clean
3. Migrate

On a high-level,

the snapshot subcommand creates a Postgres snapshot connection to the source database along with a replication slot. This is pre-requisite before running the migrate subcommand.

The migrate subcommand carries out the live-migration process by taking help of the snapshot and replication slot created by the snapshot subcommand.

The clean subcommand is designed to remove resources related to live migration. It should be run once the migration has successfully completed or, if you need to restart the migration process from the very start. You should not run clean if you want to resume the last interrupted live migration.

3.a Create a snapshot

Execute this command to establish a snapshot connection; do not interrupt the process. For convenience, consider using a terminal multiplexer such as tmux or screen, which enables the command to run in the background.

docker run --rm -it --name live-migration-snapshot \
    -e PGCOPYDB_SOURCE_PGURI=source \
    -e PGCOPYDB_TARGET_PGURI=target \
    --pid=host \
    -v ~/live-migration:/opt/timescale/ts_cdc \
    timescale/live-migration:latest snapshot

In addition to creating a snapshot, this process also validates prerequisites on the source and target to ensure the database instances are ready for replication.

For example, it checks if all tables on the source have either a PRIMARY KEY or REPLICA IDENTITY set. If not, it displays a warning message listing the tables without REPLICA IDENTITY and waits for user confirmation before proceeding with the snapshot creation.

2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX)
2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB
2024-03-25T12:40:40.884 WARNING:        - public.metrics
Press 'c' and ENTER to continue

3.b Perform live-migration

The migrate subcommand supports following flags

docker run --rm -it --name live-migration-migrate \
    -e PGCOPYDB_SOURCE_PGURI=source \
    -e PGCOPYDB_TARGET_PGURI=target \
    --pid=host \
    -v ~/live-migration:/opt/timescale/ts_cdc \
    timescale/live-migration:latest migrate --help

usage: main.py migrate [-h] [--dir DIR] [--resume] [--skip-roles] [--table-jobs TABLE_JOBS] [--index-jobs INDEX_JOBS]
                       [--skip-extensions [SKIP_EXTENSIONS ...]] [--skip-table-data SKIP_TABLE_DATA [SKIP_TABLE_DATA ...]]

options:
  -h, --help            Show this help message and exit
  --resume              Resume the migration
  --skip-roles          Skip roles migration
  --table-jobs TABLE_JOBS
                        Number of parallel jobs to copy "existing data" from source db to target db (Default: 8)
  --index-jobs INDEX_JOBS
                        Number of parallel jobs to create indexes in target db (Default: 8)
  --skip-extensions [SKIP_EXTENSIONS ...]
                        Skips the given extensions during migration. Empty list skips all extensions.
  --skip-table-data SKIP_TABLE_DATA [SKIP_TABLE_DATA ...]
                        Skips data from the given table during migration. However, the table schema will be migrated. To skip data from a
                        Hypertable, you will need to specify a list of schema qualified chunks belonging to the Hypertable. Currently, this
                        flag does not skip data during live replay from the specified table. Values for this flag must be schema qualified. Eg:
                        --skip-table-data public.exclude_table_1 public.exclude_table_2

Next, we will start the migration process. Open a new terminal and initiate the live migration, and allow it to run uninterrupted.

docker run --rm -it --name live-migration-migrate \
    -e PGCOPYDB_SOURCE_PGURI=source \
    -e PGCOPYDB_TARGET_PGURI=target \
    --pid=host \
    -v ~/live-migration:/opt/timescale/ts_cdc \
    timescale/live-migration:latest migrate

If the migrate command stops for any reason during execution, you can resume the migration from where it left off by adding a --resume flag. This is only possible if the snapshot command is intact and if a volume mount, such as ~/live-migration, is utilized.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_migration_step2/ =====

For the sake of convenience, connection strings to the source and target databases are referred to as source and target throughout this guide.

This can be set in your shell, for example:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://<user>:<password>@<target host>:<target port>/<db_name>"

Do not use a Tiger Cloud connection pooler connection for live migration. There are a number of issues which can arise when using a connection pooler, and no advantage. Very small instances may not have enough connections configured by default, in which case you should modify the value of max_connections, in your instance, as shown on Configure database parameters.

It's important to ensure that the old_snapshot_threshold value is set to the default value of -1 in your source database. This prevents Postgres from treating the data in a snapshot as outdated. If this value is set other than -1, it might affect the existing data migration step.

To check the current value of old_snapshot_threshold, run the command:

psql -X -d source -c 'show old_snapshot_threshold'

If the query returns something other than -1, you must change it.

If you have a superuser on a self-hosted database, run the following command:

psql -X -d source -c 'alter system set old_snapshot_threshold=-1'

Otherwise, if you are using a managed service, use your cloud provider's configuration mechanism to set old_snapshot_threshold to -1.

Next, you should set wal_level to logical so that the write-ahead log (WAL) records information that is needed for logical decoding.

To check the current value of wal_level, run the command:

psql -X -d source -c 'show wal_level'

If the query returns something other than logical, you must change it.

If you have a superuser on a self-hosted database, run the following command:

psql -X -d source -c 'alter system set wal_level=logical'

Otherwise, if you are using a managed service, use your cloud provider's configuration mechanism to set wal_level to logical.

Restart your database for the changes to take effect, and verify that the settings are reflected in your database.

===== PAGE: https://docs.tigerdata.com/_partials/_prometheus-integrate/ =====

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, and a modern alerting approach.

This page shows you how to export your service telemetry to Prometheus:

• For Tiger Cloud, using a dedicated Prometheus exporter in Tiger Cloud Console.
• For self-hosted TimescaleDB, using Postgres Exporter.

Prerequisites

To follow the steps on this page:

• Download and run Prometheus.

• For Tiger Cloud:

  Create a target Tiger Cloud service with the time-series and analytics capability enabled.

• For self-hosted TimescaleDB:

  • Create a target self-hosted TimescaleDB instance. You need your connection details.
  • Install Postgres Exporter. To reduce latency and potential data transfer costs, install Prometheus and Postgres Exporter on a machine in the same AWS region as your Tiger Cloud service.

Export Tiger Cloud service telemetry to Prometheus

To export your data, do the following:

To export metrics from a Tiger Cloud service, you create a dedicated Prometheus exporter in Tiger Cloud Console, attach it to your service, then configure Prometheus to scrape metrics using the exposed URL. The Prometheus exporter exposes the metrics related to the Tiger Cloud service like CPU, memory, and storage. To scrape other metrics, use Postgres Exporter as described for self-hosted TimescaleDB. The Prometheus exporter is available for Scale and Enterprise pricing plans.

1. Create a Prometheus exporter

   1. In Tiger Cloud Console, click Exporters > + New exporter.

   2. Select Metrics for data type and Prometheus for provider.

      

   3. Choose the region for the exporter. Only services in the same project and region can be attached to this exporter.

   4. Name your exporter.

   5. Change the auto-generated Prometheus credentials, if needed. See official documentation on basic authentication in Prometheus.

2. Attach the exporter to a service

   1. Select a service, then click Operations > Exporters.

   2. Select the exporter in the drop-down, then click Attach exporter.

      

   The exporter is now attached to your service. To unattach it, click the trash icon in the exporter list.

   

3. Configure the Prometheus scrape target

   1. Select your service, then click Operations > Exporters and click the information icon next to the exporter. You see the exporter details.

      

   2. Copy the exporter URL.

   3. In your Prometheus installation, update prometheus.yml to point to the exporter URL as a scrape target:

      scrape_configs:
       - job_name: "timescaledb-exporter"
         scheme: https
         static_configs:
           - targets: ["my-exporter-url"]
         basic_auth:
           username: "user"
           password: "pass"
      

      See the Prometheus documentation for details on configuring scrape targets.

      You can now monitor your service metrics. Use the following metrics to check the service is running correctly:

      • timescale.cloud.system.cpu.usage.millicores
      • timescale.cloud.system.cpu.total.millicores
      • timescale.cloud.system.memory.usage.bytes
      • timescale.cloud.system.memory.total.bytes
      • timescale.cloud.system.disk.usage.bytes
      • timescale.cloud.system.disk.total.bytes

      Additionally, use the following tags to filter your results.

      Tag	Example variable	Description
      host	us-east-1.timescale.cloud
      project-id
      service-id
      region	us-east-1	AWS region
      role	replica or primary	For service with replicas

To export metrics from self-hosted TimescaleDB, you import telemetry data about your database to Postgres Exporter, then configure Prometheus to scrape metrics from it. Postgres Exporter exposes metrics that you define, excluding the system metrics.

1. Create a user to access telemetry data about your database

   1. Connect to your database in psql using your connection details.

   2. Create a user named monitoring with a secure password:

      CREATE USER monitoring WITH PASSWORD '<password>';
      

   3. Grant the pg_read_all_stats permission to the monitoring user:

      GRANT pg_read_all_stats to monitoring;
      

2. Import telemetry data about your database to Postgres Exporter

   1. Connect Postgres Exporter to your database:

      Use your connection details to import telemetry data about your database. You connect as the monitoring user:

      • Local installation:

        export DATA_SOURCE_NAME="postgres://<user>:<password>@<host>:<port>/<database>?sslmode=<sslmode>"
        ./postgres_exporter
        

      • Docker:

        docker run -d \
           -e DATA_SOURCE_NAME="postgres://<user>:<password>@<host>:<port>/<database>?sslmode=<sslmode>" \
           -p 9187:9187 \
           prometheuscommunity/postgres-exporter
        

   2. Check the metrics for your database in the Prometheus format:

      • Browser:

        Navigate to http://<exporter-host>:9187/metrics.

      • Command line:

        curl http://<exporter-host>:9187/metrics
        

3. Configure Prometheus to scrape metrics

   1. In your Prometheus installation, update prometheus.yml to point to your Postgres Exporter instance as a scrape target. In the following example, you replace <exporter-host> with the hostname or IP address of the PostgreSQL Exporter.

      global:
        scrape_interval: 15s
      
      scrape_configs:
      - job_name: 'postgresql'
        static_configs:
         - targets: ['<exporter-host>:9187']
      

      If prometheus.yml has not been created during installation, create it manually. If you are using Docker, you can find the IPAddress in Inspect > Networks for the container running Postgres Exporter.

   2. Restart Prometheus.

   3. Check the Prometheus UI at http://<prometheus-host>:9090/targets and http://<prometheus-host>:9090/tsdb-status.

      You see the Postgres Exporter target and the metrics scraped from it.

You can further visualize your data with Grafana. Use the Grafana Postgres dashboard or create a custom dashboard that suits your needs.

===== PAGE: https://docs.tigerdata.com/_partials/_early_access_11_25/ =====

Early access: October 2025

===== PAGE: https://docs.tigerdata.com/_partials/_devops-cli-service-forks/ =====

To manage development forks:

1. Install Tiger CLI

   Use the terminal to install the CLI:

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   brew install --cask timescale/tap/tiger-cli
   

   curl -fsSL https://cli.tigerdata.com | sh
   

2. Set up API credentials

   1. Log Tiger CLI into your Tiger Data account:

      tiger auth login
      

      Tiger CLI opens Console in your browser. Log in, then click Authorize.

      You can have a maximum of 10 active client credentials. If you get an error, open credentials and delete an unused credential.

   2. Select a Tiger Cloud project:

      Auth URL is: https://console.cloud.timescale.com/oauth/authorize?client_id=lotsOfURLstuff
      Opening browser for authentication...
      Select a project:
      
      > 1. Tiger Project (tgrproject)
      2. YourCompany (Company wide project) (cpnproject)
      3. YourCompany Department (dptproject)
      
      Use ↑/↓ arrows or number keys to navigate, enter to select, q to quit
      

      If only one project is associated with your account, this step is not shown.

      Where possible, Tiger CLI stores your authentication information in the system keychain/credential manager. If that fails, the credentials are stored in ~/.config/tiger/credentials with restricted file permissions (600). By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml.

3. Test your authenticated connection to Tiger Cloud by listing services

   tiger service list
   

   This call returns something like:

   • No services:

     🏜️  No services found! Your project is looking a bit empty.
     🚀 Ready to get started? Create your first service with: tiger service create
     

   • One or more services:

     ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐
     │ SERVICE ID │        NAME         │ STATUS │    TYPE     │    REGION    │     CREATED      │
     ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤
     │ tgrservice │ tiger-agent-service │ READY  │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │
     └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘
     

4. Fork the service

    tiger service fork tgrservice --now --no-wait --name bob
   

   By default a fork matches the resource of the parent Tiger Cloud services. For paid plans specify --cpu and/or --memory for dedicated resources.

   You see something like:

   🍴 Forking service 'tgrservice' to create 'bob' at current state...
   ✅ Fork request accepted!
   📋 New Service ID: <service_id>
   🔐 Password saved to system keyring for automatic authentication
   🎯 Set service '<service_id>' as default service.
   ⏳ Service is being forked. Use 'tiger service list' to check status.
   ┌───────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
   │     PROPERTY      │                                              VALUE                                               │
   ├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ Service ID        │ <service_id>                                                                                       │
   │ Name              │ bob                                                                                              │
   │ Status            │                                                                                                  │
   │ Type              │ TIMESCALEDB                                                                                      │
   │ Region            │ eu-central-1                                                                                     │
   │ CPU               │ 0.5 cores (500m)                                                                                 │
   │ Memory            │ 2 GB                                                                                             │
   │ Direct Endpoint   │ <service-id>.<project-id>.tsdb.cloud.timescale.com:<port>                                             │
   │ Created           │ 2025-10-08 13:58:07 UTC                                                                          │
   │ Connection String │ postgresql://tsdbadmin@<service-id>.<project-id>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require │
   └───────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
   

5. When you are done, delete your forked service

   1. Use the CLI to request service delete:

      tiger service delete <service_id>
      

   2. Validate the service delete:

      Are you sure you want to delete service '<service_id>'? This operation cannot be undone.
      Type the service ID '<service_id>' to confirm:
      <service_id>
      

      You see something like:

      🗑️  Delete request accepted for service '<service_id>'.
      ✅ Service '<service_id>' has been successfully deleted.
      

===== PAGE: https://docs.tigerdata.com/_partials/_cloud-intro/ =====

Tiger Cloud is the modern Postgres data platform for all your applications. It enhances Postgres to handle time series, events, real-time analytics, and vector search—all in a single database alongside transactional workloads.

You get one system that handles live data ingestion, late and out-of-order updates, and low latency queries, with the performance, reliability, and scalability your app needs. Ideal for IoT, crypto, finance, SaaS, and a myriad other domains, Tiger Cloud allows you to build data-heavy, mission-critical apps while retaining the familiarity and reliability of Postgres.

===== PAGE: https://docs.tigerdata.com/_partials/_add-timescaledb-to-a-database/ =====

1. Connect to a database on your Postgres instance

   In Postgres, the default user and database are both postgres. To use a different database, set <database-name> to the name of that database:

   psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>"
   

2. Add TimescaleDB to the database

   CREATE EXTENSION IF NOT EXISTS timescaledb;
   

3. Check that TimescaleDB is installed

   \dx
   

   You see the list of installed extensions:

   List of installed extensions
   Name     | Version |   Schema   |                                      Description
   -------------+---------+------------+---------------------------------------------------------------------------------------
   plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
   timescaledb | 2.17.2  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
   

   Press q to exit the list of extensions.

===== PAGE: https://docs.tigerdata.com/_partials/_cloudtrial_unused/ =====

• Get started at the click of a button
• Get access to advanced cloud features like transparent bottomless object storage
• Don't waste time running high performance, highly available TimescaleDB and Postgres in the cloud

===== PAGE: https://docs.tigerdata.com/_partials/_integration-debezium-self-hosted-config-database/ =====

1. Configure your self-hosted Postgres deployment

   1. Open postgresql.conf.

      The Postgres configuration files are usually located in:

      • Docker: /home/postgres/pgdata/data/
      • Linux: /etc/postgresql/<version>/main/ or /var/lib/pgsql/<version>/data/
      • MacOS: /opt/homebrew/var/postgresql@<version>/
      • Windows: C:\Program Files\PostgreSQL\<version>\data\

   2. Enable logical replication.

      Modify the following settings in postgresql.conf:

      wal_level = logical
      max_replication_slots = 10
      max_wal_senders = 10
      

   3. Open pg_hba.conf and enable host replication.

      To allow replication connections, add the following:

      local replication debezium                         trust
      

      This permission is for the debezium Postgres user running on a local or Docker deployment. For more about replication permissions, see Configuring Postgres to allow replication with the Debezium connector host.

   4. Restart Postgres.

2. Connect to your self-hosted TimescaleDB instance

   Use psql.

3. Create a Debezium user in Postgres

   Create a user with the LOGIN and REPLICATION permissions:

   CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD '<debeziumpassword>';
   

4. Enable a replication spot for Debezium

   1. Create a table for Debezium to listen to:

      CREATE TABLE accounts (created_at TIMESTAMPTZ DEFAULT NOW(),
       name TEXT,
       city TEXT);
      

   2. Turn the table into a hypertable:

      SELECT create_hypertable('accounts', 'created_at');
      

      Debezium also works with continuous aggregates.

   3. Create a publication and enable a replication slot:

      CREATE PUBLICATION dbz_publication FOR ALL TABLES WITH (publish = 'insert, update');
      

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_self_postgres_check_versions/ =====

To see the versions of Postgres and TimescaleDB running in a self-hosted database instance:

1. Set your connection string

   This variable holds the connection information for the database to upgrade:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   

2. Retrieve the version of Postgres that you are running

   psql -X -d source -c "SELECT version();"
   

   Postgres returns something like:

   -----------------------------------------------------------------------------------------------------------------------------------------
   PostgreSQL 17.2 (Ubuntu 17.2-1.pgdg22.04+1) on aarch64-unknown-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
   (1 row)
   

3. Retrieve the version of TimescaleDB that you are running

   psql -X -d source -c "\dx timescaledb;"
   

   Postgres returns something like:

       Name     | Version |   Schema   |                             Description
   -------------+---------+------------+---------------------------------------------------------------------
   timescaledb | 2.17.2   | public     | Enables scalable inserts and complex queries for time-series data
   (1 row)
   

===== PAGE: https://docs.tigerdata.com/_partials/_create-hypertable-energy/ =====

Optimize time-series data in hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. To create a hypertable to store the energy consumption data, call CREATE TABLE.

   CREATE TABLE "metrics"(
       created timestamp with time zone default now() not null,
       type_id integer                                not null,
       value   double precision                       not null
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

===== PAGE: https://docs.tigerdata.com/_partials/_livesync-limitations/ =====

• This works for Postgres databases only as source. TimescaleDB is not yet supported.

• The source must be running Postgres 13 or later.

• Schema changes must be co-ordinated.

  Make compatible changes to the schema in your Tiger Cloud service first, then make the same changes to the source Postgres instance.

• Ensure that the source Postgres instance and the target Tiger Cloud service have the same extensions installed.

  The source Postgres connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table.

• There is WAL volume growth on the source Postgres instance during large table copy.

• Continuous aggregate invalidation

  The connector uses session_replication_role=replica during data replication, which prevents table triggers from firing. This includes the internal triggers that mark continuous aggregates as invalid when underlying data changes.

  If you have continuous aggregates on your target database, they do not automatically refresh for data inserted during the migration. This limitation only applies to data below the continuous aggregate's materialization watermark. For example, backfilled data. New rows synced above the continuous aggregate watermark are used correctly when refreshing.

  This can lead to:

  • Missing data in continuous aggregates for the migration period.
  • Stale aggregate data.
  • Queries returning incomplete results.

  If the continuous aggregate exists in the source database, best practice is to add it to the Postgres connector publication. If it only exists on the target database, manually refresh the continuous aggregate using the force option of refresh_continuous_aggregate.

===== PAGE: https://docs.tigerdata.com/_partials/_financial-industry-data-analysis/ =====

The financial industry is extremely data-heavy and relies on real-time and historical data for decision-making, risk assessment, fraud detection, and market analysis. Tiger Data simplifies management of these large volumes of data, while also providing you with meaningful analytical insights and optimizing storage costs.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_setup_environment_postgres/ =====

Set your connection strings

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Align the extensions on the source and target

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Tune your source database

You need admin rights to to update the configuration on your source database. If you are using a managed service, follow the instructions in the From AWS RDS/Aurora tab on this page.

1. Install the wal2json extension on your source database

   Install wal2json on your source database.

2. Prevent Postgres from treating the data in a snapshot as outdated

   psql -X -d source -c 'alter system set old_snapshot_threshold=-1'
   

   This is not applicable if the source database is Postgres 17 or later.

3. Set the write-Ahead Log (WAL) to record the information needed for logical decoding

   psql -X -d source -c 'alter system set wal_level=logical'
   

4. Restart the source database

   Your configuration changes are now active. However, verify that the settings are live in your database.

5. Enable live-migration to replicate DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_using_postgres_copy/ =====

Restoring data into a Tiger Cloud service with COPY

1. Connect to your Tiger Cloud service:

   psql "postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

2. Restore the data to your Tiger Cloud service:

   \copy  FROM '.csv' WITH (FORMAT CSV);
   

   Repeat for each table and hypertable you want to migrate.

===== PAGE: https://docs.tigerdata.com/_partials/_services-intro/ =====

A Tiger Cloud service is a single optimised Postgres instance extended with innovations in the database engine and cloud infrastructure to deliver speed without sacrifice. A Tiger Cloud service is 10-1000x faster at scale! It is ideal for applications requiring strong data consistency, complex relationships, and advanced querying capabilities. Get ACID compliance, extensive SQL support, JSON handling, and extensibility through custom functions, data types, and extensions.

Each service is associated with a project in Tiger Cloud. Each project can have multiple services. Each user is a member of one or more projects.

You create free and standard services in Tiger Cloud Console, depending on your pricing plan. A free service comes at zero cost and gives you limited resources to get to know Tiger Cloud. Once you are ready to try out more advanced features, you can switch to a paid plan and convert your free service to a standard one.

The Free pricing plan and services are currently in beta.

To the Postgres you know and love, Tiger Cloud adds the following capabilities:

• Standard services:

  • Real-time analytics: store and query time-series data at scale for real-time analytics and other use cases. Get faster time-based queries with hypertables, continuous aggregates, and columnar storage. Save money by compressing data into the columnstore, moving cold data to low-cost bottomless storage in Amazon S3, and deleting old data with automated policies.
  • AI-focused: build AI applications from start to scale. Get fast and accurate similarity search with the pgvector and pgvectorscale extensions.
  • Hybrid applications: get a full set of tools to develop applications that combine time-based data and AI.

  All standard Tiger Cloud services include the tooling you expect for production and developer environments: live migration, automatic backups and PITR, high availability, read replicas, data forking, connection pooling, tiered storage, usage-based storage, secure in-Tiger Cloud Console SQL editing, service metrics and insights, streamlined maintenance, and much more. Tiger Cloud continuously monitors your services and prevents common Postgres out-of-memory crashes.

• Free services:

  Postgres with TimescaleDB and vector extensions

  Free services offer limited resources and a basic feature scope, perfect to get to know Tiger Cloud in a development environment.

===== PAGE: https://docs.tigerdata.com/_partials/_mst-intro/ =====

Managed Service for TimescaleDB (MST) is TimescaleDB hosted on Azure and GCP. MST is offered in partnership with Aiven.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_migrate_data/ =====

Migrate your data, then start downtime

1. Pull the live-migration docker image to you migration machine

   sudo docker pull timescale/live-migration:latest
   

   To list the available commands, run:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest --help
   

   To see the available flags for each command, run --help for that command. For example:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest migrate --help
   

2. Create a snapshot image of your source database in your Tiger Cloud service

   This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine:

   docker run --rm -it --name live-migration-snapshot \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest snapshot
   

   Live-migration supplies information about updates you need to make to the source database and target service. For example:

   2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX)
   2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB
   2024-03-25T12:40:40.884 WARNING:        - public.metrics
   

   If you have warnings, stop live-migration, make the suggested changes and start again.

3. Synchronize data between your source database and your Tiger Cloud service

   This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target.

   docker run --rm -it --name live-migration-migrate \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest migrate
   

   If the source Postgres version is 17 or later, you need to pass additional flag -e PGVERSION=17 to the migrate command.

   After migrating the schema, live-migration prompts you to create hypertables for tables that contain time-series data in your Tiger Cloud service. Run create_hypertable() to convert these table. For more information, see the Hypertable docs.

   During this process, you see the migration process:

   Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB)
   

   If migrate stops add --resume to start from where it left off.

   Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message:

   Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB)
       To stop replication, hit 'c' and then ENTER
   

   Wait until replay_lag is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished.

4. Start app downtime

   1. Stop your app writing to the source database, then let the the remaining transactions finish to fully sync with the target. You can use tools like the pg_top CLI or pg_stat_activity to view the current transaction on the source database.

   2. Stop Live-migration.

      hit 'c' and then ENTER
      

      Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message:

      Migration successfully completed
      

===== PAGE: https://docs.tigerdata.com/_partials/_hypershift-intro/ =====

You can use hypershift to migrate existing Postgres databases in one step, and enable compression and create hypertables instantly.

Use Hypershift to migrate your data to a Tiger Cloud service from these sources:

• Standard Postgres databases
• Amazon RDS databases
• Other Tiger Data databases, including Managed Service for TimescaleDB and self-hosted TimescaleDB

===== PAGE: https://docs.tigerdata.com/_partials/_import-data-nyc-taxis/ =====

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Import time-series data into a hypertable

   1. Unzip nyc_data.tar.gz to a <local folder>.

      This test dataset contains historical data from New York's yellow taxi network.

      To import up to 100GB of data directly from your current Postgres-based database, migrate with downtime using native Postgres tooling. To seamlessly import 100GB-10TB+ of data, use the live migration tooling supplied by Tiger Data. To add data from non-Postgres data sources, see Import and ingest data.

   2. In Terminal, navigate to <local folder> and update the following string with your connection details to connect to your service.

      psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>?sslmode=require"
      

   3. Create an optimized hypertable for your time-series data:

      1. Create a hypertable with hypercore enabled by default for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data.

         In your sql client, run the following command:

         CREATE TABLE "rides"(
           vendor_id TEXT,
           pickup_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
           dropoff_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
           passenger_count NUMERIC,
           trip_distance NUMERIC,
           pickup_longitude  NUMERIC,
           pickup_latitude   NUMERIC,
           rate_code         INTEGER,
           dropoff_longitude NUMERIC,
           dropoff_latitude  NUMERIC,
           payment_type INTEGER,
           fare_amount NUMERIC,
           extra NUMERIC,
           mta_tax NUMERIC,
           tip_amount NUMERIC,
           tolls_amount NUMERIC,
           improvement_surcharge NUMERIC,
           total_amount NUMERIC
         ) WITH (
           tsdb.hypertable,
           tsdb.partition_column='pickup_datetime',
           tsdb.create_default_indexes=false,
           tsdb.segmentby='vendor_id',
           tsdb.orderby='pickup_datetime DESC'
         );
         

         If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

      2. Add another dimension to partition your hypertable more efficiently:

         SELECT add_dimension('rides', by_hash('payment_type', 2));
         

      3. Create an index to support efficient queries by vendor, rate code, and passenger count:

         CREATE INDEX ON rides (vendor_id, pickup_datetime DESC);
         CREATE INDEX ON rides (rate_code, pickup_datetime DESC);
         CREATE INDEX ON rides (passenger_count, pickup_datetime DESC);
         

   4. Create Postgres tables for relational data:

      1. Add a table to store the payment types data:

         CREATE TABLE IF NOT EXISTS "payment_types"(
           payment_type INTEGER,
           description TEXT
         );
         INSERT INTO payment_types(payment_type, description) VALUES
           (1, 'credit card'),
           (2, 'cash'),
           (3, 'no charge'),
           (4, 'dispute'),
           (5, 'unknown'),
           (6, 'voided trip');
         

      2. Add a table to store the rates data:

         CREATE TABLE IF NOT EXISTS "rates"(
          rate_code   INTEGER,
          description TEXT
         );
         INSERT INTO rates(rate_code, description) VALUES
          (1, 'standard rate'),
          (2, 'JFK'),
          (3, 'Newark'),
          (4, 'Nassau or Westchester'),
          (5, 'negotiated fare'),
          (6, 'group ride');
         

   5. Upload the dataset to your service

      \COPY rides FROM nyc_data_rides.csv CSV;
      

2. Have a quick look at your data

   You query hypertables in exactly the same way as you would a relational Postgres table. Use one of the following SQL editors to run a query and see the data you uploaded:

   • Data mode: write queries, visualize data, and share your results in Tiger Cloud Console for all your Tiger Cloud services.
   • SQL editor: write, fix, and organize SQL faster and more accurately in Tiger Cloud Console for a Tiger Cloud service.
   • psql: easily run queries on your Tiger Cloud services or self-hosted TimescaleDB deployment from Terminal.

   For example:

   • Display the number of rides for each fare type:

     SELECT rate_code, COUNT(vendor_id) AS num_trips
     FROM rides
     WHERE pickup_datetime < '2016-01-08'
     GROUP BY rate_code
     ORDER BY rate_code;
     

     This simple query runs in 3 seconds. You see something like:

     rate_code	num_trips
     1	2266401
     2	54832
     3	4126
     4	967
     5	7193
     6	17
     99	42

   • To select all rides taken in the first week of January 2016, and return the total number of trips taken for each rate code:

     SELECT rates.description, COUNT(vendor_id) AS num_trips
     FROM rides
     JOIN rates ON rides.rate_code = rates.rate_code
     WHERE pickup_datetime < '2016-01-08'
     GROUP BY rates.description
     ORDER BY LOWER(rates.description);
     

     On this large amount of data, this analytical query on data in the rowstore takes about 59 seconds. You see something like:

     description	num_trips
     group ride	17
     JFK	54832
     Nassau or Westchester	967
     negotiated fare	7193
     Newark	4126
     standard rate	2266401

===== PAGE: https://docs.tigerdata.com/_partials/_create-hypertable-twelvedata-stocks/ =====

Optimize time-series data in hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Create a hypertable to store the real-time stock data

   CREATE TABLE stocks_real_time (
     time TIMESTAMPTZ NOT NULL,
     symbol TEXT NOT NULL,
     price DOUBLE PRECISION NULL,
     day_volume INT NULL
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

3. Create an index to support efficient queries

   Index on the symbol and time columns:

   CREATE INDEX ix_symbol_time ON stocks_real_time (symbol, time DESC);
   

Create standard Postgres tables for relational data

When you have other relational data that enhances your time-series data, you can create standard Postgres tables just as you would normally. For this dataset, there is one other table of data called company.

1. Add a table to store the company data

   CREATE TABLE company (
     symbol TEXT NOT NULL,
     name TEXT NOT NULL
   );
   

You now have two tables in your Tiger Cloud service. One hypertable named stocks_real_time, and one regular Postgres table named company.

===== PAGE: https://docs.tigerdata.com/_partials/_tiered-storage-billing/ =====

For low-cost storage, Tiger Data charges only for the size of your data in S3 in the Apache Parquet format, regardless of whether it was compressed in Tiger Cloud before tiering. There are no additional expenses, such as data transfer or compute.

===== PAGE: https://docs.tigerdata.com/_partials/_create-hypertable-blockchain/ =====

Optimize time-series data using hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed. You can also connect to your service using psql.

2. Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data:

   CREATE TABLE transactions (
      time TIMESTAMPTZ NOT NULL,
      block_id INT,
      hash TEXT,
      size INT,
      weight INT,
      is_coinbase BOOLEAN,
      output_total BIGINT,
      output_total_usd DOUBLE PRECISION,
      fee BIGINT,
      fee_usd DOUBLE PRECISION,
      details JSONB
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time',
      tsdb.segmentby='block_id',
      tsdb.orderby='time DESC'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

3. Create an index on the hash column to make queries for individual transactions faster:

   CREATE INDEX hash_idx ON public.transactions USING HASH (hash);
   

4. Create an index on the block_id column to make block-level queries faster:

When you create a hypertable, it is partitioned on the time column. TimescaleDB automatically creates an index on the time column. However, you'll often filter your time-series data on other columns as well. You use indexes to improve query performance.

```sql
CREATE INDEX block_idx ON public.transactions (block_id);
```

1. Create a unique index on the time and hash columns to make sure you don't accidentally insert duplicate records:

   CREATE UNIQUE INDEX time_hash_idx ON public.transactions (time, hash);
   

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_run_cleanup/ =====

1. Validate the migrated data

   The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app.

2. Stop app downtime

   Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service.

3. Cleanup resources associated with live-migration from your migration machine

   This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration.

   docker run --rm -it --name live-migration-clean \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest clean --prune
   

===== PAGE: https://docs.tigerdata.com/_partials/_timescale-cloud-services/ =====

Tiger Cloud services run optimized Tiger Data extensions on latest Postgres, in a highly secure cloud environment. Each service is a specialized database instance tuned for your workload. Available capabilities are:

<thead>
    <tr>
        <th>Capability</th>
        <th>Extensions</th>
    </tr>
</thead>
<tbody>
    <tr>
        <td><strong>Real-time analytics</strong> <p>Lightning-fast ingest and querying of time-based and event data.</p></td>
        <td><ul><li>TimescaleDB</li><li>TimescaleDB Toolkit</li></ul>   </td>
    </tr>
    <tr>
        <td ><strong>AI and vector </strong><p>Seamlessly build RAG, search, and AI agents.</p></td>
        <td><ul><li>TimescaleDB</li><li>pgvector</li><li>pgvectorscale</li><li>pgai</li></ul></td>
    </tr>
    <tr>
        <td ><strong>Hybrid</strong><p>Everything for real-time analytics and AI workloads, combined.</p></td>
        <td><ul><li>TimescaleDB</li><li>TimescaleDB Toolkit</li><li>pgvector</li><li>pgvectorscale</li><li>pgai</li></ul></td>
    </tr>
    <tr>
        <td ><strong>Support</strong></td>
        <td><ul><li>24/7 support no matter where you are.</li><li> Continuous incremental backup/recovery. </li><li>Point-in-time forking/branching.</li><li>Zero-downtime upgrades. </li><li>Multi-AZ high availability. </li><li>An experienced global ops and support team that can build and manage Postgres at scale.</li></ul></td>
    </tr>
</tbody>

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_set_up_source_and_target/ =====

For the sake of convenience, connection strings to the source and target databases are referred to as source and target throughout this guide.

This can be set in your shell, for example:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://<user>:<password>@<target host>:<target port>/<db_name>"

===== PAGE: https://docs.tigerdata.com/_partials/_start-coding-ruby/ =====

"Quick Start: Ruby and TimescaleDB"

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install Rails.

Connect a Rails app to your service

Every Tiger Cloud service is a 100% Postgres database hosted in Tiger Cloud with Tiger Data extensions such as TimescaleDB. You connect to your Tiger Cloud service from a standard Rails app configured for Postgres.

1. Create a new Rails app configured for Postgres

   Rails creates and bundles your app, then installs the standard Postgres Gems.

   rails new my_app -d=postgresql
   cd my_app
   

2. Install the TimescaleDB gem

   1. Open Gemfile, add the following line, then save your changes:

      gem 'timescaledb'
      

   2. In Terminal, run the following command:

      bundle install
      

3. Connect your app to your Tiger Cloud service

   1. In <my_app_home>/config/database.yml update the configuration to read securely connect to your Tiger Cloud service by adding url: <%= ENV['DATABASE_URL'] %> to the default configuration:

      default: &default
        adapter: postgresql
        encoding: unicode
        pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
        url: <%= ENV['DATABASE_URL'] %>
      

   2. Set the environment variable for DATABASE_URL to the value of Service URL from your connection details

      export DATABASE_URL="value of Service URL"
      

   3. Create the database:

      • Tiger Cloud: nothing to do. The database is part of your Tiger Cloud service.

      • Self-hosted TimescaleDB, create the database for the project:

        rails db:create
        

   4. Run migrations:

      rails db:migrate
      

   5. Verify the connection from your app to your Tiger Cloud service:

      echo "\dx" | rails dbconsole
      

      The result shows the list of extensions in your Tiger Cloud service

   Name	Version	Schema	Description
   pg_buffercache	1.5	public	examine the shared buffer cache
   pg_stat_statements	1.11	public	track planning and execution statistics of all SQL statements executed
   plpgsql	1.0	pg_catalog	PL/pgSQL procedural language
   postgres_fdw	1.1	public	foreign-data wrapper for remote Postgres servers
   timescaledb	2.18.1	public	Enables scalable inserts and complex queries for time-series data (Community Edition)
   timescaledb_toolkit	1.19.0	public	Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities

Optimize time-series data in hypertables

Hypertables are Postgres tables designed to simplify and accelerate data analysis. Anything you can do with regular Postgres tables, you can do with hypertables - but much faster and more conveniently.

In this section, you use the helpers in the TimescaleDB gem to create and manage a hypertable.

1. Generate a migration to create the page loads table

   rails generate migration create_page_loads
   

This creates the <my_app_home>/db/migrate/<migration-datetime>_create_page_loads.rb migration file.

1. Add hypertable options

   Replace the contents of <my_app_home>/db/migrate/<migration-datetime>_create_page_loads.rb with the following:

   class CreatePageLoads < ActiveRecord::Migration[8.0]
     def change
       hypertable_options = {
         time_column: 'created_at',
         chunk_time_interval: '1 day',
         compress_segmentby: 'path',
         compress_orderby: 'created_at',
         compress_after: '7 days',
         drop_after: '30 days'
       }
   
       create_table :page_loads, id: false, primary_key: [:created_at, :user_agent, :path], hypertable: hypertable_options do |t|
         t.timestamptz :created_at, null: false
         t.string :user_agent
         t.string :path
         t.float :performance
       end
     end
   end
   

   The id column is not included in the table. This is because TimescaleDB requires that any UNIQUE or PRIMARY KEY indexes on the table include all partitioning columns. In this case, this is the time column. A new Rails model includes a PRIMARY KEY index for id by default: either remove the column or make sure that the index includes time as part of a "composite key."

   For more information, check the Roby docs around composite primary keys.

2. Create a PageLoad model

   Create a new file called <my_app_home>/app/models/page_load.rb and add the following code:

   class PageLoad < ApplicationRecord
     extend Timescaledb::ActsAsHypertable
     include Timescaledb::ContinuousAggregatesHelper
   
     acts_as_hypertable time_column: "created_at",
       segment_by: "path",
       value_column: "performance"
   
     scope :chrome_users, -> { where("user_agent LIKE ?", "%Chrome%") }
     scope :firefox_users, -> { where("user_agent LIKE ?", "%Firefox%") }
     scope :safari_users, -> { where("user_agent LIKE ?", "%Safari%") }
   
     scope :performance_stats, -> {
       select("stats_agg(#{value_column}) as stats_agg")
     }
   
     scope :slow_requests, -> { where("performance > ?", 1.0) }
     scope :fast_requests, -> { where("performance < ?", 0.1) }
   
     continuous_aggregates scopes: [:performance_stats],
       timeframes: [:minute, :hour, :day],
       refresh_policy: {
         minute: {
           start_offset: '3 minute',
           end_offset: '1 minute',
           schedule_interval: '1 minute'
         },
         hour: {
           start_offset: '3 hours',
           end_offset: '1 hour',
           schedule_interval: '1 minute'
         },
         day: {
           start_offset: '3 day',
           end_offset: '1 day',
           schedule_interval: '1 minute'
         }
       }
   end
   

3. Run the migration

   rails db:migrate
   

Insert data your service

The TimescaleDB gem provides efficient ways to insert data into hypertables. This section shows you how to ingest test data into your hypertable.

1. Create a controller to handle page loads

   Create a new file called <my_app_home>/app/controllers/application_controller.rb and add the following code:

   class ApplicationController < ActionController::Base
     around_action :track_page_load
   
     private
   
     def track_page_load
       start_time = Time.current
       yield
       end_time = Time.current
   
       PageLoad.create(
         path: request.path,
         user_agent: request.user_agent,
         performance: (end_time - start_time)
       )
     end
   end
   

2. Generate some test data

   Use bin/console to join a Rails console session and run the following code to define some random page load access data:

   def generate_sample_page_loads(total: 1000)
     time = 1.month.ago
     paths = %w[/ /about /contact /products /blog]
     browsers = [
       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0",
       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15"
     ]
   
     total.times.map do
       time = time + rand(60).seconds
       {
         path: paths.sample,
         user_agent: browsers.sample,
         performance: rand(0.1..2.0),
         created_at: time,
         updated_at: time
       }
     end
   end
   

3. Insert the generated data into your Tiger Cloud service

   PageLoad.insert_all(generate_sample_page_loads, returning: false)
   

4. Validate the test data in your Tiger Cloud service

PageLoad.count
PageLoad.first

Reference

This section lists the most common tasks you might perform with the TimescaleDB gem.

Query scopes

The TimescaleDB gem provides several convenient scopes for querying your time-series data.

• Built-in time-based scopes:

  PageLoad.last_hour.count
  PageLoad.today.count
  PageLoad.this_week.count
  PageLoad.this_month.count
  

• Browser-specific scopes:

  PageLoad.chrome_users.last_hour.count
  PageLoad.firefox_users.last_hour.count
  PageLoad.safari_users.last_hour.count
  
  PageLoad.slow_requests.last_hour.count
  PageLoad.fast_requests.last_hour.count
  

• Query continuous aggregates:

  This query fetches the average and standard deviation from the performance stats for the /products path over the last day.

  PageLoad::PerformanceStatsPerMinute.last_hour
  PageLoad::PerformanceStatsPerHour.last_day
  PageLoad::PerformanceStatsPerDay.last_month
  
  stats = PageLoad::PerformanceStatsPerHour.last_day.where(path: '/products').select("average(stats_agg) as average, stddev(stats_agg) as stddev").first
  puts "Average: #{stats.average}"
  puts "Standard Deviation: #{stats.stddev}"
  

TimescaleDB features

The TimescaleDB gem provides utility methods to access hypertable and chunk information. Every model that uses the acts_as_hypertable method has access to these methods.

Access hypertable and chunk information

• View chunk or hypertable information:

  PageLoad.chunks.count
  PageLoad.hypertable.detailed_size
  

• Compress/Decompress chunks:

  PageLoad.chunks.uncompressed.first.compress!  # Compress the first uncompressed chunk
  PageLoad.chunks.compressed.first.decompress!  # Decompress the oldest chunk
  PageLoad.hypertable.compression_stats # View compression stats
  
  

Access hypertable stats

You collect hypertable stats using methods that provide insights into your hypertable's structure, size, and compression status:

• Get basic hypertable information:

  hypertable = PageLoad.hypertable
  hypertable.hypertable_name  # The name of your hypertable
  hypertable.schema_name      # The schema where the hypertable is located
  

• Get detailed size information:

  hypertable.detailed_size # Get detailed size information for the hypertable
  hypertable.compression_stats # Get compression statistics
  hypertable.chunks_detailed_size # Get chunk information
  hypertable.approximate_row_count # Get approximate row count
  hypertable.dimensions.map(&:column_name) # Get dimension information
  hypertable.continuous_aggregates.map(&:view_name) # Get continuous aggregate view names
  

Continuous aggregates

The continuous_aggregates method generates a class for each continuous aggregate.

• Get all the continuous aggregate classes:

  PageLoad.descendants # Get all continuous aggregate classes
  

• Manually refresh a continuous aggregate:

  PageLoad.refresh_aggregates
  

• Create or drop a continuous aggregate:

  Create or drop all the continuous aggregates in the proper order to build them hierarchically. See more about how it works in this blog post.

  PageLoad.create_continuous_aggregates
  PageLoad.drop_continuous_aggregates
  

Next steps

Now that you have integrated the ruby gem into your app:

• Learn more about the TimescaleDB gem.
• Check out the official docs.
• Follow the LTTB, Open AI long-term storage, and candlesticks tutorials.

===== PAGE: https://docs.tigerdata.com/_partials/_add-data-energy/ =====

Load energy consumption data

When you have your database set up, you can load the energy consumption data into the metrics hypertable.

This is a large dataset, so it might take a long time, depending on your network connection.

1. Download the dataset:

metrics.csv.gz

1. Use your file manager to decompress the downloaded dataset, and take a note of the path to the metrics.csv file.

2. At the psql prompt, copy the data from the metrics.csv file into your hypertable. Make sure you point to the correct path, if it is not in your current working directory:

   \COPY metrics FROM metrics.csv CSV;
   

3. You can check that the data has been copied successfully with this command:

   SELECT * FROM metrics LIMIT 5;
   

   You should get five records that look like this:

            created            | type_id | value
   -------------------------------+---------+-------
    2023-05-31 23:59:59.043264+00 |      13 |  1.78
    2023-05-31 23:59:59.042673+00 |       2 |   126
    2023-05-31 23:59:59.042667+00 |      11 |  1.79
    2023-05-31 23:59:59.042623+00 |      23 | 0.408
    2023-05-31 23:59:59.042603+00 |      12 |  0.96
   

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_dump_database_roles/ =====

pg_dumpall -d "source" \
  -l database name \
  --quote-all-identifiers \
  --roles-only \
  --file=roles.sql

Tiger Cloud services do not support roles with superuser access. If your SQL dump includes roles that have such permissions, you'll need to modify the file to be compliant with the security model.

You can use the following sed command to remove unsupported statements and permissions from your roles.sql file:

sed -i -E \
-e '/CREATE ROLE "postgres";/d' \
-e '/ALTER ROLE "postgres"/d' \
-e '/CREATE ROLE "tsdbadmin";/d' \
-e '/ALTER ROLE "tsdbadmin"/d' \
-e 's/(NO)*SUPERUSER//g' \
-e 's/(NO)*REPLICATION//g' \
-e 's/(NO)*BYPASSRLS//g' \
-e 's/GRANTED BY "[^"]*"//g' \
roles.sql

This command works only with the GNU implementation of sed (sometimes referred to as gsed). For the BSD implementation (the default on macOS), you need to add an extra argument to change the -i flag to -i ''.

To check the sed version, you can use the command sed --version. While the GNU version explicitly identifies itself as GNU, the BSD version of sed generally doesn't provide a straightforward --version flag and simply outputs an "illegal option" error.

A brief explanation of this script is:

• CREATE ROLE "postgres"; and ALTER ROLE "postgres": These statements are removed because they require superuser access, which is not supported by Timescale.

• (NO)SUPERUSER | (NO)REPLICATION | (NO)BYPASSRLS: These are permissions that require superuser access.

• GRANTED BY role_specification: The GRANTED BY clause can also have permissions that require superuser access and should therefore be removed. Note: according to the TimescaleDB documentation, the GRANTOR in the GRANTED BY clause must be the current user, and this clause mainly serves the purpose of SQL compatibility. Therefore, it's safe to remove it.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-debian-based-start/ =====

1. Install the latest Postgres packages

   sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget
   

2. Run the Postgres package setup script

   sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
   

===== PAGE: https://docs.tigerdata.com/_partials/_free-plan-beta/ =====

The Free pricing plan and services are currently in beta.

===== PAGE: https://docs.tigerdata.com/_partials/_livesync-configure-source-database/ =====

1. Tune the Write Ahead Log (WAL) on the Postgres source database

   psql source <<EOF
   ALTER SYSTEM SET wal_level='logical';
   ALTER SYSTEM SET max_wal_senders=10;
   ALTER SYSTEM SET wal_sender_timeout=0;
   EOF
   

   • GUC “wal_level” as “logical”
   • GUC “max_wal_senders” as 10
   • GUC “wal_sender_timeout” as 0

   This will require a restart of the Postgres source database.

2. Create a user for the connector and assign permissions

   1. Create <pg connector username>:

      psql source -c "CREATE USER <pg connector username> PASSWORD '<password>'"
      

      You can use an existing user. However, you must ensure that the user has the following permissions.

   2. Grant permissions to create a replication slot:

      psql source -c "ALTER ROLE <pg connector username> REPLICATION"
      

   3. Grant permissions to create a publication:

      psql source -c "GRANT CREATE ON DATABASE <database name> TO <pg connector username>"
      

   4. Assign the user permissions on the source database:

      psql source <<EOF
      GRANT USAGE ON SCHEMA "public" TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

      If the tables you are syncing are not in the public schema, grant the user permissions for each schema you are syncing:

      psql source <<EOF
      GRANT USAGE ON SCHEMA <schema> TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA <schema> GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

   5. On each table you want to sync, make <pg connector username> the owner:

      psql source -c 'ALTER TABLE  OWNER TO <pg connector username>;'
      

      You can skip this step if the replicating user is already the owner of the tables.

3. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_datadog-data-exporter/ =====

1. In Tiger Cloud Console, open Exporters

2. Click New exporter

3. Select Metrics for Data type and Datadog for provider

   

4. Choose your AWS region and provide the API key

   The AWS region must be the same for your Tiger Cloud exporter and the Datadog provider.

5. Set Site to your Datadog region, then click Create exporter

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_6e_turn_on_compression_policies/ =====

6e. Enable policies that compress data in the target hypertable

In the following command, replace <hypertable> with the fully qualified table name of the target hypertable, for example public.metrics:

psql -d target -f -v hypertable=<hypertable> - <<'EOF'
SELECT public.alter_job(j.id, scheduled=>true)
FROM _timescaledb_config.bgw_job j
JOIN _timescaledb_catalog.hypertable h ON h.id = j.hypertable_id
WHERE j.proc_schema IN ('_timescaledb_internal', '_timescaledb_functions')
  AND j.proc_name = 'policy_compression'
  AND j.id >= 1000
  AND format('%I.%I', h.schema_name, h.table_name)::text::regclass = :'hypertable'::text::regclass;
EOF

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-redhat-rocky/ =====

1. Install TimescaleDB

   To avoid errors, do not install TimescaleDB Apache 2 Edition and TimescaleDB Community Edition at the same time.

   sudo dnf install -y postgresql16-server postgresql16-contrib timescaledb-2-postgresql-16
   

2. Initialize the Postgres instance

```bash
sudo /usr/pgsql-16/bin/postgresql-16-setup initdb
```

1. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune --pg-config=/usr/pgsql-16/bin/pg_config
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

2. Enable and start Postgres

   sudo systemctl enable postgresql-16
   sudo systemctl start postgresql-16
   

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are now in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_cloud-mst-restart-workers/ =====

On Tiger Cloud and Managed Service for TimescaleDB, restart background workers by doing one of the following:

• Run SELECT timescaledb_pre_restore(), followed by SELECT timescaledb_post_restore().
• Power the service off and on again. This might cause a downtime of a few minutes while the service restores from backup and replays the write-ahead log.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_setup_enable_replication/ =====

Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_timescale-cloud-platforms/ =====

You use Tiger Data's open-source products to create your best app from the comfort of your own developer environment.

See the available services and supported systems.

Available services

Tiger Data offers the following services for your self-hosted installations:

<thead>
    <tr>
        <th>Service type</th>
        <th>Description</th>
    </tr>
</thead>
<tbody>
    <tr>
        <td><strong>Self-hosted support</strong></td>
        <td><ul><li>24/7 support no matter where you are.</li><li>An experienced global ops and support team that
        can build and manage Postgres at scale.</li></ul>
        Want to try it out? <a href="https://www.tigerdata.com/self-managed-support">See how we can help</a>.
        </td>
    </tr>
</tbody>

Postgres, TimescaleDB support matrix

TimescaleDB and TimescaleDB Toolkit run on Postgres v10, v11, v12, v13, v14, v15, v16, and v17. Currently Postgres 15 and higher are supported.

TimescaleDB version	Postgres 17	Postgres 16	Postgres 15	Postgres 14	Postgres 13	Postgres 12	Postgres 11	Postgres 10
2.22.x	✅	✅	✅	❌	❌	❌	❌	❌
2.21.x	✅	✅	✅	❌	❌	❌	❌	❌
2.20.x	✅	✅	✅	❌	❌	❌	❌	❌
2.17 - 2.19	✅	✅	✅	✅	❌	❌	❌	❌
2.16.x	❌	✅	✅	✅	❌	❌	❌	❌
2.13 - 2.15	❌	✅	✅	✅	✅	❌	❌	❌
2.12.x	❌	❌	✅	✅	✅	❌	❌	❌
2.10.x	❌	❌	✅	✅	✅	✅	❌	❌
2.5 - 2.9	❌	❌	❌	✅	✅	✅	❌	❌
2.4	❌	❌	❌	❌	✅	✅	❌	❌
2.1 - 2.3	❌	❌	❌	❌	✅	✅	✅	❌
2.0	❌	❌	❌	❌	❌	✅	✅	❌
1.7	❌	❌	❌	❌	❌	✅	✅	✅

We recommend not using TimescaleDB with Postgres 17.1, 16.5, 15.9, 14.14, 13.17, 12.21. These minor versions introduced a breaking binary interface change that, once identified, was reverted in subsequent minor Postgres versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with Postgres 17.2, 16.6, etc and higher. Users of Tiger Cloud and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected.

Supported operating system

You can deploy TimescaleDB and TimescaleDB Toolkit on the following systems:

Operation system	Version
Debian	13 Trixe, 12 Bookworm, 11 Bullseye
Ubuntu	24.04 Noble Numbat, 22.04 LTS Jammy Jellyfish
Red Hat Enterprise	Linux 9, Linux 8
Fedora	Fedora 35, Fedora 34, Fedora 33
Rocky Linux	Rocky Linux 9 (x86_64), Rocky Linux 8
ArchLinux (community-supported)	Check the available packages

Operation system	Version
Microsoft Windows	10, 11
Microsoft Windows Server	2019, 2020

Operation system	Version
macOS	From 10.15 Catalina to 14 Sonoma

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_install_psql_ec2_instance/ =====

Install the psql client tools on the intermediary instance

1. Connect to your intermediary EC2 instance. For example:

   ssh -i "<key-pair>.pem" ubuntu@<EC2 instance's Public IPv4>
   

2. On your intermediary EC2 instance, install the Postgres client.

   sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
   wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo tee /etc/apt/trusted.gpg.d/pgdg.asc &>/dev/null
   sudo apt update
   sudo apt install postgresql-client-16 -y # "postgresql-client-16" if your source DB is using PG 16.
   psql --version && pg_dump --version
   

Keep this terminal open, you need it to connect to the RDS instance for migration.

Setup secure connectivity between your RDS and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

2. Scroll down to Security group rules (1) and select the EC2 Security Group - Inbound group. The Security Groups (1) window opens. Click the Security group ID, then click Edit inbound rules

   

3. On your intermediary EC2 instance, get your local IP address:

   ec2metadata --local-ipv4
   

   Bear with me on this one, you need this IP address to enable access to your RDS instance,

4. In Edit inbound rules, click Add rule, then create a PostgreSQL, TCP rule granting access to the local IP address for your EC2 instance (told you :-)). Then click Save rules.

   

Test the connection between your RDS and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

2. On your intermediary EC2 instance, use the values of Endpoint, Port, Master username, and DB name to create the postgres connectivity string to the SOURCE variable.

   

   export SOURCE="postgres://<Master username>:<Master password>@<Endpoint>:<Port>/<DB name>"
   

   The value of Master password was supplied when this Postgres RDS instance was created.

3. Test your connection:

   psql -d source
   

   You are connected to your RDS instance from your intermediary EC2 instance.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_setup_connection_strings/ =====

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

===== PAGE: https://docs.tigerdata.com/_partials/_psql-installation-windows/ =====

Install psql on Windows

The psql tool is installed by default on Windows systems when you install Postgres, and this is the most effective way to install the tool. These instructions use the interactive installer provided by Postgres and EnterpriseDB.

Installing psql on Windows

1. Download and run the Postgres installer from www.enterprisedb.com.
2. In the Select Components dialog, check Command Line Tools, along with any other components you want to install, and click Next.
3. Complete the installation wizard to install the package.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_run_live_migration/ =====

1. Pull the live-migration docker image to you migration machine

   sudo docker pull timescale/live-migration:latest
   

   To list the available commands, run:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest --help
   

   To see the available flags for each command, run --help for that command. For example:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest migrate --help
   

2. Create a snapshot image of your source database in your Tiger Cloud service

   This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine:

   docker run --rm -it --name live-migration-snapshot \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest snapshot
   

   Live-migration supplies information about updates you need to make to the source database and target service. For example:

   2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX)
   2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB
   2024-03-25T12:40:40.884 WARNING:        - public.metrics
   

   If you have warnings, stop live-migration, make the suggested changes and start again.

3. Synchronize data between your source database and your Tiger Cloud service

   This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target.

   docker run --rm -it --name live-migration-migrate \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest migrate
   

   If the source Postgres version is 17 or later, you need to pass additional flag -e PGVERSION=17 to the migrate command.

   After migrating the schema, live-migration prompts you to create hypertables for tables that contain time-series data in your Tiger Cloud service. Run create_hypertable() to convert these table. For more information, see the Hypertable docs.

   During this process, you see the migration process:

   Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB)
   

   If migrate stops add --resume to start from where it left off.

   Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message:

   Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB)
       To stop replication, hit 'c' and then ENTER
   

   Wait until replay_lag is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished.

4. Start app downtime

   1. Stop your app writing to the source database, then let the the remaining transactions finish to fully sync with the target. You can use tools like the pg_top CLI or pg_stat_activity to view the current transaction on the source database.

   2. Stop Live-migration.

      hit 'c' and then ENTER
      

      Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message:

      Migration successfully completed
      

===== PAGE: https://docs.tigerdata.com/_partials/_experimental/ =====

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

===== PAGE: https://docs.tigerdata.com/_partials/_compression-intro/ =====

Compressing your time-series data allows you to reduce your chunk size by more than 90%. This saves on storage costs, and keeps your queries operating at lightning speed.

When you enable compression, the data in your hypertable is compressed chunk by chunk. When the chunk is compressed, multiple records are grouped into a single row. The columns of this row hold an array-like structure that stores all the data. This means that instead of using lots of rows to store the data, it stores the same data in a single row. Because a single row takes up less disk space than many rows, it decreases the amount of disk space required, and can also speed up your queries.

For example, if you had a table with data that looked a bit like this:

Timestamp	Device ID	Device Type	CPU	Disk IO
12:00:01	A	SSD	70.11	13.4
12:00:01	B	HDD	69.70	20.5
12:00:02	A	SSD	70.12	13.2
12:00:02	B	HDD	69.69	23.4
12:00:03	A	SSD	70.14	13.0
12:00:03	B	HDD	69.70	25.2

You can convert this to a single row in array form, like this:

Timestamp	Device ID	Device Type	CPU	Disk IO
[12:00:01, 12:00:01, 12:00:02, 12:00:02, 12:00:03, 12:00:03]	[A, B, A, B, A, B]	[SSD, HDD, SSD, HDD, SSD, HDD]	[70.11, 69.70, 70.12, 69.69, 70.14, 69.70]	[13.4, 20.5, 13.2, 23.4, 13.0, 25.2]

===== PAGE: https://docs.tigerdata.com/_partials/_prereqs-cloud-only/ =====

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

  You need your connection details.

===== PAGE: https://docs.tigerdata.com/_partials/_hypercore_manual_workflow/ =====

1. Stop the jobs that are automatically adding chunks to the columnstore

   Retrieve the list of jobs from the timescaledb_information.jobs view to find the job you need to alter_job.

   SELECT alter_job(JOB_ID, scheduled => false);
   

2. Convert a chunk to update back to the rowstore

   CALL convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk');
   

3. Update the data in the chunk you added to the rowstore

   Best practice is to structure your INSERT statement to include appropriate partition key values, such as the timestamp. TimescaleDB adds the data to the correct chunk:

   INSERT INTO metrics (time, value)
   VALUES ('2025-01-01T00:00:00', 42);
   

4. Convert the updated chunks back to the columnstore

   CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk');
   

5. Restart the jobs that are automatically converting chunks to the columnstore

   SELECT alter_job(JOB_ID, scheduled => true);
   

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dump_roles_schema_data_mst/ =====

1. Dump the roles from your source database

   Export your role-based security hierarchy. <db_name> has the same value as <db_name> in source. I know, it confuses me as well.

   pg_dumpall -d "source" \
     -l <db_name>  \
     --quote-all-identifiers \
     --roles-only \
     --no-role-passwords \
     --file=roles.sql
   

   MST does not allow you to export passwords with roles. You assign passwords to these roles when you have uploaded them to your Tiger Cloud service.

2. Remove roles with superuser access

   Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   

-e '/DROP ROLE IF EXISTS "postgres";/d'
-e '/DROP ROLE IF EXISTS "tsdbadmin";/d'
-e '/CREATE ROLE "postgres";/d'
-e '/ALTER ROLE "postgres"/d'
-e '/CREATE ROLE "rds/d'
-e '/ALTER ROLE "rds/d'
-e '/TO "rds/d'
-e '/GRANT "rds/d'
-e '/GRANT "pg_read_all_stats" TO "tsdbadmin"/d'
-e 's/(NO)SUPERUSER//g'
-e 's/(NO)REPLICATION//g'
-e 's/(NO)BYPASSRLS//g'
-e 's/GRANTED BY "[^"]"//g'
-e '/CREATE ROLE "tsdbadmin";/d'
-e '/ALTER ROLE "tsdbadmin"/d'
-e 's/WITH ADMIN OPTION,/WITH /g'
-e 's/WITH ADMIN OPTION//g'
-e 's/GRANTED BY "."//g'
-e '/GRANT "pg_." TO/d'
-e '/CREATE ROLE "_aiven";/d'
-e '/ALTER ROLE "_aiven"/d'
-e '/GRANT SET ON PARAMETER "pgaudit.[^"]+" TO "_tsdbadmin_auditing"/d'
-e '/GRANT SET ON PARAMETER "anon.[^"]+" TO "tsdbadmin_group"/d'
roles.sql

1. **Dump the source database schema and data**

The `pg_dump` flags remove superuser access and tablespaces from your data. When you run
`pgdump`, check the run time, [a long-running `pg_dump` can cause issues][long-running-pgdump].

```bash
pg_dump -d "source" \
--format=plain \
--quote-all-identifiers \
--no-tablespaces \
--no-owner \
--no-privileges \
--file=dump.sql

To dramatically reduce the time taken to dump the source database, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_migrate_data_timescaledb/ =====

Migrate your data, then start downtime

2. Pull the live-migration docker image to you migration machine

   sudo docker pull timescale/live-migration:latest
   

   To list the available commands, run:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest --help
   

   To see the available flags for each command, run --help for that command. For example:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest migrate --help
   

3. Create a snapshot image of your source database in your Tiger Cloud service

   This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine:

   docker run --rm -it --name live-migration-snapshot \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest snapshot
   

   Live-migration supplies information about updates you need to make to the source database and target service. For example:

   2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX)
   2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB
   2024-03-25T12:40:40.884 WARNING:        - public.metrics
   

   If you have warnings, stop live-migration, make the suggested changes and start again.

4. Synchronize data between your source database and your Tiger Cloud service

   This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target.

   docker run --rm -it --name live-migration-migrate \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest migrate
   

   If the source Postgres version is 17 or later, you need to pass additional flag -e PGVERSION=17 to the migrate command.

   During this process, you see the migration process:

   Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB)
   

   If migrate stops add --resume to start from where it left off.

   Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message:

   Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB)
       To stop replication, hit 'c' and then ENTER
   

   Wait until replay_lag is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished.

5. Start app downtime

   1. Stop your app writing to the source database, then let the the remaining transactions finish to fully sync with the target. You can use tools like the pg_top CLI or pg_stat_activity to view the current transaction on the source database.

   2. Stop Live-migration.

      hit 'c' and then ENTER
      

      Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message:

      Migration successfully completed
      

===== PAGE: https://docs.tigerdata.com/_partials/_prereqs-cloud-account-only/ =====

To follow the steps on this page:

• Create a target Tiger Data account.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_set_up_database_first_steps/ =====

1. Take the applications that connect to the source database offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss.

2. Set your connection strings

   These variables hold the connection information for the source database and target Tiger Cloud service:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

   You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-redhat/ =====

1. Install the latest Postgres packages

   sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-$(rpm -E %{rhel})-x86_64/pgdg-redhat-repo-latest.noarch.rpm
   

2. Add the TimescaleDB repository

   sudo tee /etc/yum.repos.d/timescale_timescaledb.repo <<EOL
   [timescale_timescaledb]
   name=timescale_timescaledb
   baseurl=https://packagecloud.io/timescale/timescaledb/el/$(rpm -E %{rhel})/\$basearch
   repo_gpgcheck=1
   gpgcheck=0
   enabled=1
   gpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey
   sslverify=1
   sslcacert=/etc/pki/tls/certs/ca-bundle.crt
   metadata_expire=300
   EOL
   

3. Update your local repository list

   sudo yum update
   

4. Install TimescaleDB

   To avoid errors, do not install TimescaleDB Apache 2 Edition and TimescaleDB Community Edition at the same time.

   sudo yum install timescaledb-2-postgresql-17 postgresql17
   

   On Red Hat Enterprise Linux 8 and later, disable the built-in Postgres module:

   sudo dnf -qy module disable postgresql

5. Initialize the Postgres instance

```bash
sudo /usr/pgsql-17/bin/postgresql-17-setup initdb
```

1. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune --pg-config=/usr/pgsql-17/bin/pg_config
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

2. Enable and start Postgres

   sudo systemctl enable postgresql-17
   sudo systemctl start postgresql-17
   

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are now in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_chunk-interval/ =====

Postgres builds the index on the fly during ingestion. That means that to build a new entry on the index, a significant portion of the index needs to be traversed during every row insertion. When the index does not fit into memory, it is constantly flushed to disk and read back. This wastes IO resources which would otherwise be used for writing the heap/WAL data to disk.

The default chunk interval is 7 days. However, best practice is to set chunk_interval so that prior to processing, the indexes for chunks currently being ingested into fit within 25% of main memory. For example, on a system with 64 GB of memory, if index growth is approximately 2 GB per day, a 1-week chunk interval is appropriate. If index growth is around 10 GB per day, use a 1-day interval.

You set chunk_interval when you create a hypertable, or by calling set_chunk_time_interval on an existing hypertable.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_tune_source_database_mst/ =====

1. Enable live-migration to replicate DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_tutorials_hypertable_intro/ =====

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

===== PAGE: https://docs.tigerdata.com/_partials/_hypertable-intro/ =====

Tiger Cloud supercharges your real-time analytics by letting you run complex queries continuously, with near-zero latency. Under the hood, this is achieved by using hypertables—Postgres tables that automatically partition your time-series data by time and optionally by other dimensions. When you run a query, Tiger Cloud identifies the correct partition, called chunk, and runs the query on it, instead of going through the entire table.

Hypertables offer the following benefits:

• Efficient data management with automated partitioning by time: Tiger Cloud splits your data into chunks that hold data from a specific time range. For example, one day or one week. You can configure this range to better suit your needs.

• Better performance with strategic indexing: an index on time in the descending order is automatically created when you create a hypertable. More indexes are created on the chunk level, to optimize performance. You can create additional indexes, including unique indexes, on the columns you need.

• Faster queries with chunk skipping: Tiger Cloud skips the chunks that are irrelevant in the context of your query, dramatically reducing the time and resources needed to fetch results. Even more—you can enable chunk skipping on non-partitioning columns.

• Advanced data analysis with hyperfunctions: Tiger Cloud enables you to efficiently process, aggregate, and analyze significant volumes of data while maintaining high performance.

To top it all, there is no added complexity—you interact with hypertables in the same way as you would with regular Postgres tables. All the optimization magic happens behind the scenes.

Inheritance is not supported for hypertables and may lead to unexpected behavior.

===== PAGE: https://docs.tigerdata.com/_partials/_kubernetes-install-self-hosted/ =====

Running TimescaleDB on Kubernetes is similar to running Postgres. This procedure outlines the steps for a non-distributed system.

To connect your Kubernetes cluster to self-hosted TimescaleDB running in the cluster:

1. Create a default namespace for Tiger Data components

   1. Create the Tiger Data namespace:

      kubectl create namespace timescale
      

   2. Set this namespace as the default for your session:

      kubectl config set-context --current --namespace=timescale
      

   For more information, see Kubernetes Namespaces.

2. Set up a persistent volume claim (PVC) storage

   To manually set up a persistent volume and claim for self-hosted Kubernetes, run the following command:

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: timescale-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 10Gi
   EOF
   

3. Deploy TimescaleDB as a StatefulSet

   By default, the TimescaleDB Docker image you are installing on Kubernetes uses the default Postgres database, user and password. To deploy TimescaleDB on Kubernetes, run the following command:

   kubectl apply -f - <<EOF
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: timescaledb
   spec:
     serviceName: timescaledb
     replicas: 1
     selector:
       matchLabels:
         app: timescaledb
     template:
       metadata:
         labels:
           app: timescaledb
       spec:
         containers:
           - name: timescaledb
             image: 'timescale/timescaledb:latest-pg17'
             env:
               - name: POSTGRES_USER
                 value: postgres
               - name: POSTGRES_PASSWORD
                 value: postgres
               - name: POSTGRES_DB
                 value: postgres
               - name: PGDATA
                 value: /var/lib/postgresql/data/pgdata
             ports:
               - containerPort: 5432
             volumeMounts:
               - mountPath: /var/lib/postgresql/data
                 name: timescale-storage
         volumes:
           - name: timescale-storage
             persistentVolumeClaim:
               claimName: timescale-pvc
   EOF
   

4. Allow applications to connect by exposing TimescaleDB within Kubernetes

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: timescaledb
spec:
  selector:
    app: timescaledb
  ports:
    - protocol: TCP
      port: 5432
      targetPort: 5432
  type: ClusterIP
EOF

1. Create a Kubernetes secret to store the database credentials

   kubectl create secret generic timescale-secret \
   --from-literal=PGHOST=timescaledb \
   --from-literal=PGPORT=5432 \
   --from-literal=PGDATABASE=postgres \
   --from-literal=PGUSER=postgres \
   --from-literal=PGPASSWORD=postgres
   

2. Deploy an application that connects to TimescaleDB

   kubectl apply -f - <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: timescale-app
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: timescale-app
     template:
       metadata:
         labels:
           app: timescale-app
       spec:
         containers:
         - name: timescale-container
           image: postgres:latest
           envFrom:
             - secretRef:
                 name: timescale-secret
   EOF
   

3. Test the database connection

   1. Create and run a pod to verify database connectivity using your connection details saved in timescale-secret:

      kubectl run test-pod --image=postgres --restart=Never \
      --env="PGHOST=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGHOST}' | base64 --decode)" \
      --env="PGPORT=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGPORT}' | base64 --decode)" \
      --env="PGDATABASE=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGDATABASE}' | base64 --decode)" \
      --env="PGUSER=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGUSER}' | base64 --decode)" \
      --env="PGPASSWORD=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGPASSWORD}' | base64 --decode)" \
      -- sleep infinity
      

   2. Launch the Postgres interactive shell within the created test-pod:

      kubectl exec -it test-pod -- bash -c "psql -h \$PGHOST -U \$PGUSER -d \$PGDATABASE"
      

   You see the Postgres interactive terminal.

===== PAGE: https://docs.tigerdata.com/_partials/_caggs-migrate-permissions/ =====

You might get a permissions error when migrating a continuous aggregate from old to new format using cagg_migrate. The user performing the migration must have the following permissions:

• Select, insert, and update permissions on the tables _timescale_catalog.continuous_agg_migrate_plan and _timescale_catalog.continuous_agg_migrate_plan_step
• Usage permissions on the sequence _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq

To solve the problem, change to a user capable of granting permissions, and grant the following permissions to the user performing the migration:

GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan TO <USER>;
GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan_step TO <USER>;
GRANT USAGE ON SEQUENCE _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq TO <USER>;

===== PAGE: https://docs.tigerdata.com/_partials/_candlestick_intro/ =====

The financial sector regularly uses candlestick charts to visualize the price change of an asset. Each candlestick represents a time period, such as one minute or one hour, and shows how the asset's price changed during that time.

Candlestick charts are generated from the open, high, low, close, and volume data for each financial asset during the time period. This is often abbreviated as OHLCV:

• Open: opening price
• High: highest price
• Low: lowest price
• Close: closing price
• Volume: volume of transactions

===== PAGE: https://docs.tigerdata.com/_partials/_start-coding-java/ =====

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install the Java Development Kit (JDK).

• Install the PostgreSQL JDBC driver.

All code in this quick start is for Java 16 and later. If you are working with older JDK versions, use legacy coding techniques.

Connect to your Tiger Cloud service

In this section, you create a connection to your service using an application in a single file. You can use any of your favorite build tools, including gradle or maven.

1. Create a directory containing a text file called Main.java, with this content:

   package com.timescale.java;
   
   public class Main {
   
       public static void main(String... args) {
           System.out.println("Hello, World!");
       }
   }
   

2. From the command line in the current directory, run the application:

   java Main.java
   

   If the command is successful, Hello, World! line output is printed to your console.

3. Import the PostgreSQL JDBC driver. If you are using a dependency manager, include the PostgreSQL JDBC Driver as a dependency.

4. Download the JAR artifact of the JDBC Driver and save it with the Main.java file.

5. Import the JDBC Driver into the Java application and display a list of available drivers for the check:

   package com.timescale.java;
   
   import java.sql.DriverManager;
   
   public class Main {
   
       public static void main(String... args) {
           DriverManager.drivers().forEach(System.out::println);
       }
   }
   

6. Run all the examples:

   java -cp *.jar Main.java
   

If the command is successful, a string similar to org.postgresql.Driver@7f77e91b is printed to your console. This means that you are ready to connect to TimescaleDB from Java.

1. Locate your TimescaleDB credentials and use them to compose a connection string for JDBC.

   You'll need:

   • password
   • username
   • host URL
   • port
   • database name

2. Compose your connection string variable, using this format:

   var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
   

   For more information about creating connection strings, see the JDBC documentation.

   This method of composing a connection string is for test or development purposes only. For production, use environment variables for sensitive details like your password, hostname, and port number.

   package com.timescale.java;
   
   import java.sql.DriverManager;
   import java.sql.SQLException;
   
   public class Main {
   
       public static void main(String... args) throws SQLException {
           var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
           var conn = DriverManager.getConnection(connUrl);
           System.out.println(conn.getClientInfo());
       }
   }
   

3. Run the code:

   java -cp *.jar Main.java
   

   If the command is successful, a string similar to {ApplicationName=PostgreSQL JDBC Driver} is printed to your console.

Create a relational table

In this section, you create a table called sensors which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called sensor_data which holds the measurements of those sensors. The measurements contain the time, sensor_id, temperature reading, and CPU percentage of the sensors.

1. Compose a string which contains the SQL statement to create a relational table. This example creates a table called sensors, with columns id, type and location:

   CREATE TABLE sensors (
       id SERIAL PRIMARY KEY,
       type TEXT NOT NULL,
       location TEXT NOT NULL
   );
   

2. Create a statement, execute the query you created in the previous step, and check that the table was created successfully:

   package com.timescale.java;
   
   import java.sql.DriverManager;
   import java.sql.SQLException;
   
   public class Main {
   
       public static void main(String... args) throws SQLException {
           var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
           var conn = DriverManager.getConnection(connUrl);
   
           var createSensorTableQuery = """
                   CREATE TABLE sensors (
                       id SERIAL PRIMARY KEY,
                       type TEXT NOT NULL,
                       location TEXT NOT NULL
                   )
                   """;
           try (var stmt = conn.createStatement()) {
               stmt.execute(createSensorTableQuery);
           }
   
           var showAllTablesQuery = "SELECT tablename FROM pg_catalog.pg_tables WHERE schemaname = 'public'";
           try (var stmt = conn.createStatement();
                var rs = stmt.executeQuery(showAllTablesQuery)) {
               System.out.println("Tables in the current database: ");
               while (rs.next()) {
                   System.out.println(rs.getString("tablename"));
               }
           }
       }
   }
   

Create a hypertable

When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable.

1. Create a CREATE TABLE SQL statement for your hypertable. Notice how the hypertable has the compulsory time column:

   CREATE TABLE sensor_data (
       time TIMESTAMPTZ NOT NULL,
       sensor_id INTEGER REFERENCES sensors (id),
       value DOUBLE PRECISION
   );
   

2. Create a statement, execute the query you created in the previous step:

   SELECT create_hypertable('sensor_data', by_range('time'));
   

   The by_range and by_hash dimension builder is an addition to TimescaleDB 2.13.

3. Execute the two statements you created, and commit your changes to the database:

   package com.timescale.java;
   
   import java.sql.Connection;
   import java.sql.DriverManager;
   import java.sql.SQLException;
   import java.util.List;
   
   public class Main {
   
       public static void main(String... args) {
           final var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
           try (var conn = DriverManager.getConnection(connUrl)) {
               createSchema(conn);
               insertData(conn);
           } catch (SQLException ex) {
               System.err.println(ex.getMessage());
           }
       }
   
       private static void createSchema(final Connection conn) throws SQLException {
           try (var stmt = conn.createStatement()) {
               stmt.execute("""
                       CREATE TABLE sensors (
                           id SERIAL PRIMARY KEY,
                           type TEXT NOT NULL,
                           location TEXT NOT NULL
                       )
                       """);
           }
   
           try (var stmt = conn.createStatement()) {
               stmt.execute("""
                       CREATE TABLE sensor_data (
                           time TIMESTAMPTZ NOT NULL,
                           sensor_id INTEGER REFERENCES sensors (id),
                           value DOUBLE PRECISION
                       )
                       """);
           }
   
           try (var stmt = conn.createStatement()) {
               stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))");
           }
       }
   }
   

Insert data

You can insert data into your hypertables in several different ways. In this section, you can insert single rows, or insert by batches of rows.

1. Open a connection to the database, use prepared statements to formulate the INSERT SQL statement, then execute the statement:

   final List<Sensor> sensors = List.of(
           new Sensor("temperature", "bedroom"),
           new Sensor("temperature", "living room"),
           new Sensor("temperature", "outside"),
           new Sensor("humidity", "kitchen"),
           new Sensor("humidity", "outside"));
   for (final var sensor : sensors) {
       try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) {
           stmt.setString(1, sensor.type());
           stmt.setString(2, sensor.location());
           stmt.executeUpdate();
       }
   }
   

If you want to insert a batch of rows by using a batching mechanism. In this example, you generate some sample time-series data to insert into the sensor_data hypertable:

1. Insert batches of rows:

   final var sensorDataCount = 100;
   final var insertBatchSize = 10;
   try (var stmt = conn.prepareStatement("""
           INSERT INTO sensor_data (time, sensor_id, value)
           VALUES (
               generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'),
               floor(random() * 4 + 1)::INTEGER,
               random()
           )
           """)) {
       for (int i = 0; i < sensorDataCount; i++) {
           stmt.addBatch();
   
           if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) {
               stmt.executeBatch();
           }
       }
   }
   

Execute a query

This section covers how to execute queries against your database.

Execute queries on TimescaleDB

1. Define the SQL query you'd like to run on the database. This example combines time-series and relational data. It returns the average values for every 15 minute interval for sensors with specific type and location.

   SELECT time_bucket('15 minutes', time) AS bucket, avg(value)
   FROM sensor_data
   JOIN sensors ON sensors.id = sensor_data.sensor_id
   WHERE sensors.type = ? AND sensors.location = ?
   GROUP BY bucket
   ORDER BY bucket DESC;
   

2. Execute the query with the prepared statement and read out the result set for all a-type sensors located on the floor:

   try (var stmt = conn.prepareStatement("""
           SELECT time_bucket('15 minutes', time) AS bucket, avg(value)
           FROM sensor_data
           JOIN sensors ON sensors.id = sensor_data.sensor_id
           WHERE sensors.type = ? AND sensors.location = ?
           GROUP BY bucket
           ORDER BY bucket DESC
           """)) {
       stmt.setString(1, "temperature");
       stmt.setString(2, "living room");
   
       try (var rs = stmt.executeQuery()) {
           while (rs.next()) {
               System.out.printf("%s: %f%n", rs.getTimestamp(1), rs.getDouble(2));
           }
       }
   }
   

   If the command is successful, you'll see output like this:

   2021-05-12 23:30:00.0: 0,508649
   2021-05-12 23:15:00.0: 0,477852
   2021-05-12 23:00:00.0: 0,462298
   2021-05-12 22:45:00.0: 0,457006
   2021-05-12 22:30:00.0: 0,568744
   ...
   

Next steps

Now that you're able to connect, read, and write to a TimescaleDB instance from your Java application, and generate the scaffolding necessary to build a new application from an existing TimescaleDB instance, be sure to check out these advanced TimescaleDB tutorials:

• Continuous Aggregates
• Migrate Your own Data

Complete code samples

This section contains complete code samples.

Complete code sample

package com.timescale.java;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.List;

public class Main {

    public static void main(String... args) {
        final var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
        try (var conn = DriverManager.getConnection(connUrl)) {
            createSchema(conn);
            insertData(conn);
        } catch (SQLException ex) {
            System.err.println(ex.getMessage());
        }
    }

    private static void createSchema(final Connection conn) throws SQLException {
        try (var stmt = conn.createStatement()) {
            stmt.execute("""
                    CREATE TABLE sensors (
                        id SERIAL PRIMARY KEY,
                        type TEXT NOT NULL,
                        location TEXT NOT NULL
                    )
                    """);
        }

        try (var stmt = conn.createStatement()) {
            stmt.execute("""
                    CREATE TABLE sensor_data (
                        time TIMESTAMPTZ NOT NULL,
                        sensor_id INTEGER REFERENCES sensors (id),
                        value DOUBLE PRECISION
                    )
                    """);
        }

        try (var stmt = conn.createStatement()) {
            stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))");
        }
    }

    private static void insertData(final Connection conn) throws SQLException {
        final List<Sensor> sensors = List.of(
                new Sensor("temperature", "bedroom"),
                new Sensor("temperature", "living room"),
                new Sensor("temperature", "outside"),
                new Sensor("humidity", "kitchen"),
                new Sensor("humidity", "outside"));
        for (final var sensor : sensors) {
            try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) {
                stmt.setString(1, sensor.type());
                stmt.setString(2, sensor.location());
                stmt.executeUpdate();
            }
        }

        final var sensorDataCount = 100;
        final var insertBatchSize = 10;
        try (var stmt = conn.prepareStatement("""
                INSERT INTO sensor_data (time, sensor_id, value)
                VALUES (
                    generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'),
                    floor(random() * 4 + 1)::INTEGER,
                    random()
                )
                """)) {
            for (int i = 0; i < sensorDataCount; i++) {
                stmt.addBatch();

                if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) {
                    stmt.executeBatch();
                }
            }
        }
    }

    private record Sensor(String type, String location) {
    }
}

Execute more complex queries

package com.timescale.java;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.List;

public class Main {

    public static void main(String... args) {
        final var connUrl = "jdbc:postgresql://<HOSTNAME>:<PORT>/<DATABASE_NAME>?user=<USERNAME>&password=<PASSWORD>";
        try (var conn = DriverManager.getConnection(connUrl)) {
            createSchema(conn);
            insertData(conn);
            executeQueries(conn);
        } catch (SQLException ex) {
            System.err.println(ex.getMessage());
        }
    }

    private static void createSchema(final Connection conn) throws SQLException {
        try (var stmt = conn.createStatement()) {
            stmt.execute("""
                    CREATE TABLE sensors (
                        id SERIAL PRIMARY KEY,
                        type TEXT NOT NULL,
                        location TEXT NOT NULL
                    )
                    """);
        }

        try (var stmt = conn.createStatement()) {
            stmt.execute("""
                    CREATE TABLE sensor_data (
                        time TIMESTAMPTZ NOT NULL,
                        sensor_id INTEGER REFERENCES sensors (id),
                        value DOUBLE PRECISION
                    )
                    """);
        }

        try (var stmt = conn.createStatement()) {
            stmt.execute("SELECT create_hypertable('sensor_data', by_range('time'))");
        }
    }

    private static void insertData(final Connection conn) throws SQLException {
        final List<Sensor> sensors = List.of(
                new Sensor("temperature", "bedroom"),
                new Sensor("temperature", "living room"),
                new Sensor("temperature", "outside"),
                new Sensor("humidity", "kitchen"),
                new Sensor("humidity", "outside"));
        for (final var sensor : sensors) {
            try (var stmt = conn.prepareStatement("INSERT INTO sensors (type, location) VALUES (?, ?)")) {
                stmt.setString(1, sensor.type());
                stmt.setString(2, sensor.location());
                stmt.executeUpdate();
            }
        }

        final var sensorDataCount = 100;
        final var insertBatchSize = 10;
        try (var stmt = conn.prepareStatement("""
                INSERT INTO sensor_data (time, sensor_id, value)
                VALUES (
                    generate_series(now() - INTERVAL '24 hours', now(), INTERVAL '5 minutes'),
                    floor(random() * 4 + 1)::INTEGER,
                    random()
                )
                """)) {
            for (int i = 0; i < sensorDataCount; i++) {
                stmt.addBatch();

                if ((i > 0 && i % insertBatchSize == 0) || i == sensorDataCount - 1) {
                    stmt.executeBatch();
                }
            }
        }
    }

    private static void executeQueries(final Connection conn) throws SQLException {
        try (var stmt = conn.prepareStatement("""
                SELECT time_bucket('15 minutes', time) AS bucket, avg(value)
                FROM sensor_data
                JOIN sensors ON sensors.id = sensor_data.sensor_id
                WHERE sensors.type = ? AND sensors.location = ?
                GROUP BY bucket
                ORDER BY bucket DESC
                """)) {
            stmt.setString(1, "temperature");
            stmt.setString(2, "living room");

            try (var rs = stmt.executeQuery()) {
                while (rs.next()) {
                    System.out.printf("%s: %f%n", rs.getTimestamp(1), rs.getDouble(2));
                }
            }
        }
    }

    private record Sensor(String type, String location) {
    }
}

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_self_postgres_implement_migration_path/ =====

You cannot upgrade TimescaleDB and Postgres at the same time. You upgrade each product in the following steps:

1. Upgrade TimescaleDB

   psql -X -d source -c "ALTER EXTENSION timescaledb UPDATE TO '<version number>';"
   

2. If your migration path dictates it, upgrade Postgres

   Follow the procedure in Upgrade Postgres. The version of TimescaleDB installed in your Postgres deployment must be the same before and after the Postgres upgrade.

3. If your migration path dictates it, upgrade TimescaleDB again

   psql -X -d source -c "ALTER EXTENSION timescaledb UPDATE TO '<version number>';"
   

4. Check that you have upgraded to the correct version of TimescaleDB

   psql -X -d source -c "\dx timescaledb;"
   

   Postgres returns something like:

   Name     | Version | Schema |                                      Description
   -------------+---------+--------+---------------------------------------------------------------------------------------
   timescaledb | 2.17.2  | public | Enables scalable inserts and complex queries for time-series data (Community Edition)
   

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_validate_production_load/ =====

Now that dual-writes have been in place for a while, the target database should be holding up to production write traffic. Now would be the right time to determine if the target database can serve all production traffic (both reads and writes). How exactly this is done is application-specific and up to you to determine.

===== PAGE: https://docs.tigerdata.com/_partials/_prereqs-cloud-no-connection/ =====

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_import_prerequisites/ =====

Best practice is to use an Ubuntu EC2 instance hosted in the same region as your Tiger Cloud service as a migration machine. That is, the machine you run the commands on to move your data from your source database to your target Tiger Cloud service.

Before you migrate your data:

• Create a target Tiger Cloud service.

  Each Tiger Cloud service has a single database that supports the most popular extensions. Tiger Cloud services do not support tablespaces, and there is no superuser associated with a service. Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window.

• To ensure that maintenance does not run during the process, adjust the maintenance window.

===== PAGE: https://docs.tigerdata.com/_partials/_hypercore-intro-short/ =====

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

===== PAGE: https://docs.tigerdata.com/_partials/_caggs-intro/ =====

In modern applications, data usually grows very quickly. This means that aggregating it into useful summaries can become very slow. If you are collecting data very frequently, you might want to aggregate your data into minutes or hours instead. For example, if an IoT device takes temperature readings every second, you might want to find the average temperature for each hour. Every time you run this query, the database needs to scan the entire table and recalculate the average. TimescaleDB makes aggregating data lightning fast, accurate, and easy with continuous aggregates.

Continuous aggregates in TimescaleDB are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background.

Continuous aggregates have a much lower maintenance burden than regular Postgres materialized views, because the whole view is not created from scratch on each refresh. This means that you can get on with working your data instead of maintaining your database.

Because continuous aggregates are based on hypertables, you can query them in exactly the same way as your other tables. This includes continuous aggregates in the rowstore, compressed into the columnstore, or tiered to object storage. You can even create continuous aggregates on top of your continuous aggregates, for an even more fine-tuned aggregation.

Real-time aggregation enables you to combine pre-aggregated data from the materialized view with the most recent raw data. This gives you up-to-date results on every query. In TimescaleDB v2.13 and later, real-time aggregates are DISABLED by default. In earlier versions, real-time aggregates are ENABLED by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data.

===== PAGE: https://docs.tigerdata.com/_partials/_kubernetes-prereqs/ =====

• Install self-managed Kubernetes or sign up for a Kubernetes Turnkey Cloud Solution.
• Install kubectl for command-line interaction with your cluster.

===== PAGE: https://docs.tigerdata.com/_partials/_high-availability-setup/ =====

1. In Tiger Cloud Console, select the service to enable replication for.

2. Click Operations, then select High availability.

3. Choose your replication strategy, then click Change configuration.

   

4. In Change high availability configuration, click Change config.

===== PAGE: https://docs.tigerdata.com/_partials/_vpc-limitations/ =====

• You can attach:

  • Up to 50 Customer VPCs to a Peering VPC.
  • A Tiger Cloud service to a single Peering VPC at a time. The service and the Peering VPC must be in the same AWS region. However, you can peer a Customer VPC and a Peering VPC that are in different regions.
  • Multiple Tiger Cloud services to the same Peering VPC.

• You cannot attach a Tiger Cloud service to multiple Peering VPCs at the same time.

  The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your pricing plan in Tiger Cloud Console.

===== PAGE: https://docs.tigerdata.com/_partials/_integration-apache-kafka-install/ =====

1. Extract the Kafka binaries to a local folder

   curl https://dlcdn.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz | tar -xzf -
   cd kafka_2.13-3.9.0
   

   From now on, the folder where you extracted the Kafka binaries is called <KAFKA_HOME>.

2. Configure and run Apache Kafka

   KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
   ./bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/kraft/reconfig-server.properties
   ./bin/kafka-server-start.sh config/kraft/reconfig-server.properties
   

   Use the -daemon flag to run this process in the background.

3. Create Kafka topics

   In another Terminal window, navigate to <KAFKA_HOME>, then call kafka-topics.sh and create the following topics:

   • accounts: publishes JSON messages that are consumed by the timescale-sink connector and inserted into your Tiger Cloud service.
   • deadletter: stores messages that cause errors and that Kafka Connect workers cannot process.

   ./bin/kafka-topics.sh \
        --create \
        --topic accounts \
        --bootstrap-server localhost:9092 \
        --partitions 10
   
   ./bin/kafka-topics.sh \
        --create \
        --topic deadletter \
        --bootstrap-server localhost:9092 \
        --partitions 10
   

4. Test that your topics are working correctly

   1. Run kafka-console-producer to send messages to the accounts topic:

      bin/kafka-console-producer.sh --topic accounts --bootstrap-server localhost:9092
      

   2. Send some events. For example, type the following:

      >Tiger
      >How Cool
      

   3. In another Terminal window, navigate to <KAFKA_HOME>, then run kafka-console-consumer to consume the events you just sent:

      bin/kafka-console-consumer.sh --topic accounts --from-beginning --bootstrap-server localhost:9092
      

      You see

      Tiger
      How Cool
      

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_tune_source_database_awsrds/ =====

Updating parameters on a Postgres instance will cause an outage. Choose a time that will cause the least issues to tune this database.

1. Update the DB instance parameter group for your source database

   1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

   2. Click Configuration, scroll down and note the DB instance parameter group, then click Parameter groups

      

   3. Click Create parameter group, fill in the form with the following values, then click Create.

      • Parameter group name - whatever suits your fancy.
      • Description - knock yourself out with this one.
      • Engine type - PostgreSQL
      • Parameter group family - the same as DB instance parameter group in your Configuration.

   4. In Parameter groups, select the parameter group you created, then click Edit.

   5. Update the following parameters, then click Save changes.

      • rds.logical_replication set to 1: record the information needed for logical decoding.
      • wal_sender_timeout set to 0: disable the timeout for the sender process.

   6. In RDS, navigate back to your databases, select the RDS instance to migrate, and click Modify.

   7. Scroll down to Database options, select your new parameter group, and click Continue.

   8. Click Apply immediately or choose a maintenance window, then click Modify DB instance.

      Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.

   9. Verify that the settings are live in your database.

2. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_foreign-data-wrappers/ =====

You use Postgres foreign data wrappers (FDWs) to query external data sources from a Tiger Cloud service. These external data sources can be one of the following:

• Other Tiger Cloud services
• Postgres databases outside of Tiger Cloud

If you are using VPC peering, you can create FDWs in your Customer VPC to query a service in your Tiger Cloud project. However, you can't create FDWs in your Tiger Cloud services to query a data source in your Customer VPC. This is because Tiger Cloud VPC peering uses AWS PrivateLink for increased security. See VPC peering documentation for additional details.

Postgres FDWs are particularly useful if you manage multiple Tiger Cloud services with different capabilities, and need to seamlessly access and merge regular and time-series data.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Query another data source

To query another data source:

You create Postgres FDWs with the postgres_fdw extension, which is enabled by default in Tiger Cloud.

1. Connect to your service

   See how to connect.

2. Create a server

   Run the following command using your connection details:

   CREATE SERVER myserver
   FOREIGN DATA WRAPPER postgres_fdw
   OPTIONS (host '<host>', dbname 'tsdb', port '<port>');
   

3. Create user mapping

   Run the following command using your connection details:

   CREATE USER MAPPING FOR tsdbadmin
   SERVER myserver
   OPTIONS (user 'tsdbadmin', password '<password>');
   

4. Import a foreign schema (recommended) or create a foreign table

   • Import the whole schema:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     FROM SERVER myserver
     INTO foreign_stuff ;
     

   • Alternatively, import a limited number of tables:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     LIMIT TO (table1, table2)
     FROM SERVER myserver
     INTO foreign_stuff;
     

   • Create a foreign table. Skip if you are importing a schema:

     CREATE FOREIGN TABLE films (
         code        char(5) NOT NULL,
         title       varchar(40) NOT NULL,
         did         integer NOT NULL,
         date_prod   date,
         kind        varchar(10),
         len         interval hour to minute
     )
     SERVER film_server;
     

A user with the tsdbadmin role assigned already has the required USAGE permission to create Postgres FDWs. You can enable another user, without the tsdbadmin role assigned, to query foreign data. To do so, explicitly grant the permission. For example, for a new grafana user:

CREATE USER grafana;

GRANT grafana TO tsdbadmin;

CREATE SCHEMA fdw AUTHORIZATION grafana;

CREATE SERVER db1 FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host '<host>', dbname 'tsdb', port '<port>');

CREATE USER MAPPING FOR grafana SERVER db1
OPTIONS (user 'tsdbadmin', password '<password>');

GRANT USAGE ON FOREIGN SERVER db1 TO grafana;

SET ROLE grafana;

IMPORT FOREIGN SCHEMA public
       FROM SERVER db1
       INTO fdw;

You create Postgres FDWs with the postgres_fdw extension. See documenation on how to enable it.

1. Connect to your database

   Use psql to connect to your database.

2. Create a server

   Run the following command using your connection details:

   CREATE SERVER myserver
   FOREIGN DATA WRAPPER postgres_fdw
   OPTIONS (host '<host>', dbname '<database_name>', port '<port>');
   

3. Create user mapping

   Run the following command using your connection details:

   CREATE USER MAPPING FOR postgres
   SERVER myserver
   OPTIONS (user 'postgres', password '<password>');
   

4. Import a foreign schema (recommended) or create a foreign table

   • Import the whole schema:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     FROM SERVER myserver
     INTO foreign_stuff ;
     

   • Alternatively, import a limited number of tables:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     LIMIT TO (table1, table2)
     FROM SERVER myserver
     INTO foreign_stuff;
     

   • Create a foreign table. Skip if you are importing a schema:

     CREATE FOREIGN TABLE films (
         code        char(5) NOT NULL,
         title       varchar(40) NOT NULL,
         did         integer NOT NULL,
         date_prod   date,
         kind        varchar(10),
         len         interval hour to minute
     )
     SERVER film_server;
     

===== PAGE: https://docs.tigerdata.com/_partials/_cookbook-iot/ =====

IoT recipes

This section contains recipes for IoT issues:

Work with columnar IoT data

Narrow and medium width tables are a great way to store IoT data. A lot of reasons are outlined in Designing Your Database Schema: Wide vs. Narrow Postgres Tables.

One of the key advantages of narrow tables is that the schema does not have to change when you add new sensors. Another big advantage is that each sensor can sample at different rates and times. This helps support things like hysteresis, where new values are written infrequently unless the value changes by a certain amount.

Narrow table format example

Working with narrow table data structures presents a few challenges. In the IoT world one concern is that many data analysis approaches - including machine learning as well as more traditional data analysis - require that your data is resampled and synchronized to a common time basis. Fortunately, TimescaleDB provides you with hyperfunctions and other tools to help you work with this data.

An example of a narrow table format is:

ts	sensor_id	value
2024-10-31 11:17:30.000	1007	23.45

Typically you would couple this with a sensor table:

sensor_id	sensor_name	units
1007	temperature	degreesC
1012	heat_mode	on/off
1013	cooling_mode	on/off
1041	occupancy	number of people in room

A medium table retains the generic structure but adds columns of various types so that you can use the same table to store float, int, bool, or even JSON (jsonb) data:

ts	sensor_id	d	i	b	t	j
2024-10-31 11:17:30.000	1007	23.45	null	null	null	null
2024-10-31 11:17:47.000	1012	null	null	TRUE	null	null
2024-10-31 11:18:01.000	1041	null	4	null	null	null

To remove all-null entries, use an optional constraint such as:

    CONSTRAINT at_least_one_not_null
        CHECK ((d IS NOT NULL) OR (i IS NOT NULL) OR (b IS NOT NULL) OR (j IS NOT NULL) OR (t IS NOT NULL))

Get the last value of every sensor

There are several ways to get the latest value of every sensor. The following examples use the structure defined in Narrow table format example as a reference:

• SELECT DISTINCT ON
• JOIN LATERAL

SELECT DISTINCT ON

If you have a list of sensors, the easy way to get the latest value of every sensor is to use SELECT DISTINCT ON:

WITH latest_data AS (
    SELECT DISTINCT ON (sensor_id) ts, sensor_id, d
    FROM iot_data
    WHERE d is not null
      AND ts > CURRENT_TIMESTAMP - INTERVAL '1 week'  -- important
    ORDER BY sensor_id, ts DESC
)
SELECT
    sensor_id, sensors.name, ts, d
FROM latest_data
LEFT OUTER JOIN sensors ON latest_data.sensor_id = sensors.id
WHERE latest_data.d is not null
ORDER BY sensor_id, ts; -- Optional, for displaying results ordered by sensor_id

The common table expression (CTE) used above is not strictly necessary. However, it is an elegant way to join to the sensor list to get a sensor name in the output. If this is not something you care about, you can leave it out:

SELECT DISTINCT ON (sensor_id) ts, sensor_id, d
    FROM iot_data
    WHERE d is not null
      AND ts > CURRENT_TIMESTAMP - INTERVAL '1 week'  -- important
    ORDER BY sensor_id, ts DESC

It is important to take care when down-selecting this data. In the previous examples, the time that the query would scan back was limited. However, if there any sensors that have either not reported in a long time or in the worst case, never reported, this query devolves to a full table scan. In a database with 1000+ sensors and 41 million rows, an unconstrained query takes over an hour.

JOIN LATERAL

An alternative to SELECT DISTINCT ON is to use a JOIN LATERAL. By selecting your entire sensor list from the sensors table rather than pulling the IDs out using SELECT DISTINCT, JOIN LATERAL can offer some improvements in performance:

SELECT sensor_list.id, latest_data.ts, latest_data.d
FROM sensors sensor_list
    -- Add a WHERE clause here to downselect the sensor list, if you wish
LEFT JOIN LATERAL (
    SELECT ts, d
    FROM iot_data raw_data
    WHERE sensor_id = sensor_list.id
    ORDER BY ts DESC
    LIMIT 1
) latest_data ON true
WHERE latest_data.d is not null -- only pulling out float values ("d" column) in this example
  AND latest_data.ts > CURRENT_TIMESTAMP - interval '1 week' -- important
ORDER BY sensor_list.id, latest_data.ts;

Limiting the time range is important, especially if you have a lot of data. Best practice is to use these kinds of queries for dashboards and quick status checks. To query over a much larger time range, encapsulate the previous example into a materialized query that refreshes infrequently, perhaps once a day.

Shoutout to Christopher Piggott for this recipe.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_from_timescaledb_version/ =====

It is very important that the version of the TimescaleDB extension is the same in the source and target databases. This requires upgrading the TimescaleDB extension in the source database before migrating.

You can determine the version of TimescaleDB in the target database with the following command:

psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"

To update the TimescaleDB extension in your source database, first ensure that the desired version is installed from your package repository. Then you can upgrade the extension with the following query:

psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"

For more information and guidance, consult the Upgrade TimescaleDB page.

===== PAGE: https://docs.tigerdata.com/_partials/_since_2_18_0/ =====

Since TimescaleDB v2.18.0

===== PAGE: https://docs.tigerdata.com/_partials/_add-data-nyctaxis/ =====

Load trip data

When you have your database set up, you can load the taxi trip data into the rides hypertable.

This is a large dataset, so it might take a long time, depending on your network connection.

1. Download the dataset:

nyc_data.tar.gz

1. Use your file manager to decompress the downloaded dataset, and take a note of the path to the nyc_data_rides.csv file.

2. At the psql prompt, copy the data from the nyc_data_rides.csv file into your hypertable. Make sure you point to the correct path, if it is not in your current working directory:

   \COPY rides FROM nyc_data_rides.csv CSV;
   

You can check that the data has been copied successfully with this command:

SELECT * FROM rides LIMIT 5;

You should get five records that look like this:

-[ RECORD 1 ]---------+--------------------
vendor_id             | 1
pickup_datetime       | 2016-01-01 00:00:01
dropoff_datetime      | 2016-01-01 00:11:55
passenger_count       | 1
trip_distance         | 1.20
pickup_longitude      | -73.979423522949219
pickup_latitude       | 40.744613647460938
rate_code             | 1
dropoff_longitude     | -73.992034912109375
dropoff_latitude      | 40.753944396972656
payment_type          | 2
fare_amount           | 9
extra                 | 0.5
mta_tax               | 0.5
tip_amount            | 0
tolls_amount          | 0
improvement_surcharge | 0.3
total_amount          | 10.3

===== PAGE: https://docs.tigerdata.com/_partials/_cloud-create-service/ =====

Create a Tiger Cloud service

1. Sign in to the{" "} Tiger Cloud Console and click Create service.

2. Choose if you want a Time-series or Dynamic Postgres service.

{props.demoData && (

3. Click Get started to create your service with demo data, and launch the Allmilk Factory interactive demo. You can exit the demo at any time, and revisit it from the same point later on. You can also re-run the demo after you have completed it.

   
)}

4. Click Download the cheatsheet to download an SQL file that contains the login details for your new service. You can also copy the details directly from this page. When you have copied your password, click I stored my password, go to service overview at the bottom of the page.

5. When your service is ready to use, is shows a green Running label in the Service Overview. You also receive an email confirming that your service is ready to use.

===== PAGE: https://docs.tigerdata.com/_partials/_caggs-real-time-historical-data-refreshes/ =====

Real-time aggregates automatically add the most recent data when you query your continuous aggregate. In other words, they include data more recent than your last materialized bucket.

If you add new historical data to an already-materialized bucket, it won't be reflected in a real-time aggregate. You should wait for the next scheduled refresh, or manually refresh by calling refresh_continuous_aggregate. You can think of real-time aggregates as being eventually consistent for historical data.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_awsrds_connect_intermediary/ =====

Create an intermediary EC2 Ubuntu instance

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.
2. Click Actions > Set up EC2 connection. Press Create EC2 instance and use the following settings:
   • AMI: Ubuntu Server.
   • Key pair: use an existing pair or create a new one that you will use to access the intermediary machine.
   • VPC: by default, this is the same as the database instance.
   • Configure Storage: adjust the volume to at least the size of RDS/Aurora Postgres instance you are migrating from. You can reduce the space used by your data on Tiger Cloud using Hypercore.
3. Click Lauch instance. AWS creates your EC2 instance, then click Connect to instance > SSH client. Follow the instructions to create the connection to your intermediary EC2 instance.

Install the psql client tools on the intermediary instance

1. Connect to your intermediary EC2 instance. For example:

   ssh -i "<key-pair>.pem" ubuntu@<EC2 instance's Public IPv4>
   

2. On your intermediary EC2 instance, install the Postgres client.

   sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
   wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo tee /etc/apt/trusted.gpg.d/pgdg.asc &>/dev/null
   sudo apt update
   sudo apt install postgresql-client-16 -y # "postgresql-client-16" if your source DB is using PG 16.
   psql --version && pg_dump --version
   

Keep this terminal open, you need it to connect to the RDS/Aurora Postgres instance for migration.

Set up secure connectivity between your RDS/Aurora Postgres and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.

2. Scroll down to Security group rules (1) and select the EC2 Security Group - Inbound group. The Security Groups (1) window opens. Click the Security group ID, then click Edit inbound rules

   

3. On your intermediary EC2 instance, get your local IP address:

   ec2metadata --local-ipv4
   

   Bear with me on this one, you need this IP address to enable access to your RDS/Aurora Postgres instance.

4. In Edit inbound rules, click Add rule, then create a PostgreSQL, TCP rule granting access to the local IP address for your EC2 instance (told you :-)). Then click Save rules.

   

Test the connection between your RDS/Aurora Postgres and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.

2. On your intermediary EC2 instance, use the values of Endpoint, Port, Master username, and DB name to create the postgres connectivity string to the SOURCE variable.

   

   export SOURCE="postgres://<Master username>:<Master password>@<Endpoint>:<Port>/<DB name>"
   

   The value of Master password was supplied when this RDS/Aurora Postgres instance was created.

3. Test your connection:

   psql -d source
   

   You are connected to your RDS/Aurora Postgres instance from your intermediary EC2 instance.

===== PAGE: https://docs.tigerdata.com/_partials/_transit-gateway/ =====

1. Create a Peering VPC in Tiger Cloud Console

   1. In Security > VPC, click Create a VPC:

      

   2. Choose your region and IP range, name your VPC, then click Create VPC:

      

      Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your plan in Tiger Cloud Console.

   3. Add a peering connection:

      1. In the VPC Peering column, click Add.
      2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID.

      

      1. Click Add connection.

2. Accept and configure peering connection in your AWS account

   Once your peering connection appears as Processing, you can accept and configure it in AWS:

   1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as Connected in Tiger Cloud Console.

   2. Configure at least the following in your AWS account networking:

      • Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs.
      • Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs.
      • Security groups to allow outbound TCP 5432.

3. Attach a Tiger Cloud service to the Peering VPC In Tiger Cloud Console

   1. Select the service you want to connect to the Peering VPC.
   2. Click Operations > Security > VPC.
   3. Select the VPC, then click Attach VPC.

   You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time.

===== PAGE: https://docs.tigerdata.com/_partials/_cloud-intro-short/ =====

A Tiger Cloud service is a single optimised Postgres instance extended with innovations in the database engine such as TimescaleDB, in a cloud infrastructure that delivers speed without sacrifice.

A Tiger Cloud service is a radically faster Postgres database for transactional, analytical, and agentic workloads at scale.

It’s not a fork. It’s not a wrapper. It is Postgres—extended with innovations in the database engine and cloud infrastructure to deliver speed (10-1000x faster at scale) without sacrifice. A Tiger Cloud service brings together the familiarity and reliability of Postgres with the performance of purpose-built engines.

Tiger Cloud is the fastest Postgres cloud. It includes everything you need to run Postgres in a production-reliable, scalable, observable environment.

===== PAGE: https://docs.tigerdata.com/_partials/_since_2_22_0/ =====

Since TimescaleDB v2.22.0

===== PAGE: https://docs.tigerdata.com/_partials/_integration-prereqs/ =====

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

===== PAGE: https://docs.tigerdata.com/_partials/_cloud_self_configuration/ =====

Please refer to the Grand Unified Configuration (GUC) parameters for a complete list.

Policies

timescaledb.max_background_workers (int)

Max background worker processes allocated to TimescaleDB. Set to at least 1 + the number of databases loaded with the TimescaleDB extension in a Postgres instance. Default value is 16.

Tiger Cloud service tuning

timescaledb.disable_load (bool)

Disable the loading of the actual extension

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_step2/ =====

2. Modify the application to write to the target database

How exactly to do this is dependent on the language that your application is written in, and on how exactly your ingestion and application function. In the simplest case, you simply execute two inserts in parallel. In the general case, you must think about how to handle the failure to write to either the source or target database, and what mechanism you want to or can build to recover from such a failure.

Should your time-series data have foreign-key references into a plain table, you must ensure that your application correctly maintains the foreign key relations. If the referenced column is a *SERIAL type, the same row inserted into the source and target may not obtain the same autogenerated id. If this happens, the data backfilled from the source to the target is internally inconsistent. In the best case it causes a foreign key violation, in the worst case, the foreign key constraint is maintained, but the data references the wrong foreign key. To avoid these issues, best practice is to follow live migration.

You may also want to execute the same read queries on the source and target database to evaluate the correctness and performance of the results which the queries deliver. Bear in mind that the target database spends a certain amount of time without all data being present, so you should expect that the results are not the same for some period (potentially a number of days).

===== PAGE: https://docs.tigerdata.com/_partials/_timescaledb_supported_linux/ =====

Operation system	Version
Debian	13 Trixe, 12 Bookworm, 11 Bullseye
Ubuntu	24.04 Noble Numbat, 22.04 LTS Jammy Jellyfish
Red Hat Enterprise	Linux 9, Linux 8
Fedora	Fedora 35, Fedora 34, Fedora 33
Rocky Linux	Rocky Linux 9 (x86_64), Rocky Linux 8
ArchLinux (community-supported)	Check the available packages

===== PAGE: https://docs.tigerdata.com/_partials/_add-data-twelvedata-stocks/ =====

Load financial data

This tutorial uses real-time stock trade data, also known as tick data, from Twelve Data. A direct download link is provided below.

To ingest data into the tables that you created, you need to download the dataset and copy the data to your database.

1. Download the real_time_stock_data.zip file. The file contains two .csv files; one with company information, and one with real-time stock trades for the past month. Download:

   real_time_stock_data.zip

2. In a new terminal window, run this command to unzip the .csv files:

   unzip real_time_stock_data.zip
   

3. At the psql prompt, use the COPY command to transfer data into your Tiger Cloud service. If the .csv files aren't in your current directory, specify the file paths in these commands:

   \COPY stocks_real_time from './tutorial_sample_tick.csv' DELIMITER ',' CSV HEADER;
   

   \COPY company from './tutorial_sample_company.csv' DELIMITER ',' CSV HEADER;
   

   Because there are millions of rows of data, the COPY process could take a few minutes depending on your internet connection and local client resources.

===== PAGE: https://docs.tigerdata.com/_partials/_hypercore_policy_workflow/ =====

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Enable columnstore on a hypertable

   Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data. For example:

   • Use CREATE TABLE for a hypertable

     CREATE TABLE crypto_ticks (
        "time" TIMESTAMPTZ,
        symbol TEXT,
        price DOUBLE PRECISION,
        day_volume NUMERIC
     ) WITH (
       tsdb.hypertable,
       tsdb.partition_column='time',
       tsdb.segmentby='symbol',
       tsdb.orderby='time DESC'
     );
     

     If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

   • Use ALTER MATERIALIZED VIEW for a continuous aggregate

     ALTER MATERIALIZED VIEW assets_candlestick_daily set (
        timescaledb.enable_columnstore = true,
        timescaledb.segmentby = 'symbol' );
     

     Before you say huh, a continuous aggregate is a specialized hypertable.

3. Add a policy to convert chunks to the columnstore at a specific time interval

   Create a columnstore_policy that automatically converts chunks in a hypertable to the columnstore at a specific time interval. For example, convert yesterday's crypto trading data to the columnstore:

   CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '1d');
   

   TimescaleDB is optimized for fast updates on compressed data in the columnstore. To modify data in the columnstore, use standard SQL.

4. Check the columnstore policy

   1. View your data space saving:

      When you convert data to the columnstore, as well as being optimized for analytics, it is compressed by more than 90%. This helps you save on storage costs and keeps your queries operating at lightning speed. To see the amount of space saved:

      SELECT
        pg_size_pretty(before_compression_total_bytes) as before,
        pg_size_pretty(after_compression_total_bytes) as after
      FROM hypertable_columnstore_stats('crypto_ticks');
      

      You see something like:

      before	after
      194 MB	24 MB

   2. View the policies that you set or the policies that already exist:

      SELECT * FROM timescaledb_information.jobs
      WHERE proc_name='policy_compression';
      

      See timescaledb_information.jobs.

5. Pause a columnstore policy

   SELECT * FROM timescaledb_information.jobs where
      proc_name = 'policy_compression' AND relname = 'crypto_ticks'
   
   -- Select the JOB_ID from the results
   
   SELECT alter_job(JOB_ID, scheduled => false);
   

   See alter_job.

6. Restart a columnstore policy

   SELECT alter_job(JOB_ID, scheduled => true);
   

   See alter_job.

7. Remove a columnstore policy

   CALL remove_columnstore_policy('crypto_ticks');
   

   See remove_columnstore_policy.

8. Disable columnstore

   If your table has chunks in the columnstore, you have to convert the chunks back to the rowstore before you disable the columnstore.

   ALTER TABLE crypto_ticks SET (timescaledb.enable_columnstore = false);
   

   See alter_table_hypercore.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_switch_production_workload/ =====

Once you've validated that all the data is present, and that the target database can handle the production workload, the final step is to switch to the target database as your primary. You may want to continue writing to the source database for a period, until you are certain that the target database is holding up to all production traffic.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dump_roles_schema_data_multi_node/ =====

1. Dump the roles from your source database

   Export your role-based security hierarchy. If you only use the default postgres role, this step is not necessary.

   pg_dumpall -d "source" \
     --quote-all-identifiers \
     --roles-only \
     --no-role-passwords \
     --file=roles.sql
   

   MST does not allow you to export passwords with roles. You assign passwords to these roles when you have uploaded them to your Tiger Cloud service.

2. Remove roles with superuser access

   Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

===== PAGE: https://docs.tigerdata.com/_partials/_cloud-create-connect-tutorials/ =====

A service in Tiger Cloud is a cloud instance which contains your database. Each service contains a single database, named tsdb. You can connect to a service from your local system using the psql command-line utility. If you've used Postgres before, you might already have psql installed. If not, check out the installing psql section.

1. In the Tiger Cloud Console, click Create service.

2. Click Download the cheatsheet to download an SQL file that contains the login details for your new service. You can also copy the details directly from this page. When you have copied your password, click I stored my password, go to service overview at the bottom of the page.

   When your service is ready to use, is shows a green Running label in the Service Overview. You also receive an email confirming that your service is ready to use.

3. On your local system, at the command prompt, connect to the service using the Service URL from the SQL file that you downloaded. When you are prompted, enter the password:

   psql -x "<SERVICE_URL>"
   Password for user tsdbadmin:
   

   If your connection is successful, you'll see a message like this, followed by the psql prompt:

   psql (13.3, server 12.8 (Ubuntu 12.8-1.pgdg21.04+1))
   SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
   Type "help" for help.
   tsdb=>
   

===== PAGE: https://docs.tigerdata.com/_partials/_integration-prereqs-cloud-only/ =====

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details.

===== PAGE: https://docs.tigerdata.com/_partials/_grafana-connect/ =====

Connect Grafana to Tiger Cloud

To visualize the results of your queries, enable Grafana to read the data in your service:

1. Log in to Grafana

   In your browser, log in to either:

   • Self-hosted Grafana: at http://localhost:3000/. The default credentials are admin, admin.
   • Grafana Cloud: use the URL and credentials you set when you created your account.

2. Add your service as a data source

   1. Open Connections > Data sources, then click Add new data source.

   2. Select PostgreSQL from the list.

   3. Configure the connection:

      • Host URL, Database name, Username, and Password

        Configure using your connection details. Host URL is in the format <host>:<port>.

      • TLS/SSL Mode: select require.

      • PostgreSQL options: enable TimescaleDB.

      • Leave the default setting for all other fields.

   4. Click Save & test.

   Grafana checks that your details are set correctly.

===== PAGE: https://docs.tigerdata.com/_partials/_prereqs-cloud-project-and-self/ =====

To follow the procedure on this page you need to:

• Create a Tiger Data account.

  This procedure also works for self-hosted TimescaleDB.

===== PAGE: https://docs.tigerdata.com/_partials/_caggs-function-support/ =====

The following table summarizes the aggregate functions supported in continuous aggregates:

Function, clause, or feature	TimescaleDB 2.6 and earlier	TimescaleDB 2.7, 2.8, and 2.9	TimescaleDB 2.10 and later
Parallelizable aggregate functions	✅	✅	✅
Non-parallelizable SQL aggregates	❌	✅	✅
ORDER BY	❌	✅	✅
Ordered-set aggregates	❌	✅	✅
Hypothetical-set aggregates	❌	✅	✅
DISTINCT in aggregate functions	❌	✅	✅
FILTER in aggregate functions	❌	✅	✅
FROM clause supports JOINS	❌	❌	✅

DISTINCT works in aggregate functions, not in the query definition. For example, for the table:

CREATE TABLE public.candle(
symbol_id uuid                     NOT NULL,
symbol    text                     NOT NULL,
"time"    timestamp with time zone NOT NULL,
open      double precision         NOT NULL,
high      double precision         NOT NULL,
low       double precision         NOT NULL,
close     double precision         NOT NULL,
volume    double precision         NOT NULL
);

• The following works:

  CREATE MATERIALIZED VIEW candles_start_end
  WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 hour', "time"), COUNT(DISTINCT symbol), first(time, time) as first_candle, last(time, time) as last_candle
  FROM candle
  GROUP BY 1;
  

• This does not:

  CREATE MATERIALIZED VIEW candles_start_end
  WITH (timescaledb.continuous) AS
  SELECT DISTINCT ON (symbol)
  symbol,symbol_id, first(time, time) as first_candle, last(time, time) as last_candle
  FROM candle
  GROUP BY symbol_id;
  

===== PAGE: https://docs.tigerdata.com/_partials/_psql-installation-macports/ =====

Installing psql using MacPorts

1. Install the latest version of libpqxx:

   sudo port install libpqxx
   

2. View the files that were installed by libpqxx:

   port contents libpqxx
   

===== PAGE: https://docs.tigerdata.com/_partials/_toolkit-install-update-redhat-base/ =====

Prerequisites

To follow this procedure:

• Install TimescaleDB.
• Create a TimescaleDB repository in your yum repo.d directory.

Install TimescaleDB Toolkit

These instructions use the yum package manager.

1. Set up the repository:

   curl -s https://packagecloud.io/install/repositories/timescale/timescaledb/script.deb.sh | sudo bash
   

2. Update your local repository list:

   yum update
   

3. Install TimescaleDB Toolkit:

   yum install timescaledb-toolkit-postgresql-17
   

4. Connect to the database where you want to use Toolkit.

5. Create the Toolkit extension in the database:

   CREATE EXTENSION timescaledb_toolkit;
   

Update TimescaleDB Toolkit

Update Toolkit by installing the latest version and running ALTER EXTENSION.

1. Update your local repository list:

   yum update
   

2. Install the latest version of TimescaleDB Toolkit:

   yum install timescaledb-toolkit-postgresql-17
   

3. Connect to the database where you want to use the new version of Toolkit.

4. Update the Toolkit extension in the database:

   ALTER EXTENSION timescaledb_toolkit UPDATE;
   

   For some Toolkit versions, you might need to disconnect and reconnect active sessions.

===== PAGE: https://docs.tigerdata.com/_partials/_cookbook-hypertables/ =====

Hypertable recipes

This section contains recipes about hypertables.

Remove duplicates from an existing hypertable

Looking to remove duplicates from an existing hypertable? One method is to run a PARTITION BY query to get ROW_NUMBER() and then the ctid of rows where row_number>1. You then delete these rows. However, you need to check tableoid and ctid. This is because ctid is not unique and might be duplicated in different chunks. The following code example took 17 hours to process a table with 40 million rows:

CREATE OR REPLACE FUNCTION deduplicate_chunks(ht_name TEXT, partition_columns TEXT, bot_id INT DEFAULT NULL)
    RETURNS TABLE
            (
                chunk_schema  name,
                chunk_name    name,
                deleted_count INT
            )
AS
$$
DECLARE
    chunk         RECORD;
    where_clause  TEXT := '';
    deleted_count INT;
BEGIN
    IF bot_id IS NOT NULL THEN
        where_clause := FORMAT('WHERE bot_id = %s', bot_id);
    END IF;

    FOR chunk IN
        SELECT c.chunk_schema, c.chunk_name
        FROM timescaledb_information.chunks c
        WHERE c.hypertable_name = ht_name
        LOOP
            EXECUTE FORMAT('
            WITH cte AS (
                SELECT ctid,
                       ROW_NUMBER() OVER (PARTITION BY %s ORDER BY %s ASC) AS row_num,
                       *
                FROM %I.%I
                %s
            )
            DELETE FROM %I.%I
            WHERE ctid IN (
                SELECT ctid
                FROM cte
                WHERE row_num > 1
            )
            RETURNING 1;
        ', partition_columns, partition_columns, chunk.chunk_schema, chunk.chunk_name, where_clause, chunk.chunk_schema,
                           chunk.chunk_name)
                INTO deleted_count;

            RETURN QUERY SELECT chunk.chunk_schema, chunk.chunk_name, COALESCE(deleted_count, 0);
        END LOOP;
END
$$ LANGUAGE plpgsql;


SELECT *
FROM deduplicate_chunks('nudge_events', 'bot_id, session_id, nudge_id, time', 2540);

Shoutout to Mathias Ose and Christopher Piggott for this recipe.

Get faster JOIN queries with Common Table Expressions

Imagine there is a query that joins a hypertable to another table on a shared key:

    SELECT timestamp,
      FROM hypertable as h
      JOIN related_table as rt
        ON rt.id = h.related_table_id
     WHERE h.timestamp BETWEEN '2024-10-10 00:00:00' AND '2024-10-17 00:00:00'

If you run EXPLAIN on this query, you see that the query planner performs a NestedJoin between these two tables, which means querying the hypertable multiple times. Even if the hypertable is well indexed, if it is also large, the query will be slow. How do you force a once-only lookup? Use materialized Common Table Expressions (CTEs).

If you split the query into two parts using CTEs, you can materialize the hypertable lookup and force Postgres to perform it only once.

WITH cached_query AS materialized (
  SELECT *
    FROM hypertable
   WHERE BETWEEN '2024-10-10 00:00:00' AND '2024-10-17 00:00:00'
)
  SELECT *
    FROM cached_query as c
    JOIN related_table as rt
      ON rt.id = h.related_table_id

Now if you run EXPLAIN once again, you see that this query performs only one lookup. Depending on the size of your hypertable, this could result in a multi-hour query taking mere seconds.

Shoutout to Rowan Molony for this recipe.

===== PAGE: https://docs.tigerdata.com/_partials/_experimental-private-beta/ =====

This feature is experimental and offered as part of a private beta. Do not use this feature in production.

===== PAGE: https://docs.tigerdata.com/_partials/_hypershift-alternatively/ =====

Alternatively, if you have data in an existing database, you can migrate it directly into your new Tiger Cloud service using hypershift. For more information about hypershift, including instructions for how to migrate your data, see the Migrate and sync data to Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/_partials/_timescaledb_supported_windows/ =====

Operation system	Version
Microsoft Windows	10, 11
Microsoft Windows Server	2019, 2020

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_post_data_dump_source_schema/ =====

pg_dump -d "source" \
  --format=plain \
  --quote-all-identifiers \
  --no-tablespaces \
  --no-owner \
  --no-privileges \
  --section=post-data \
  --file=post-data-dump.sql \
  --snapshot=$(cat /tmp/pgcopydb/snapshot)

• --section=post-data is used to dump post-data items include definitions of indexes, triggers, rules, and constraints other than validated check constraints.

• --snapshot is used to specified the synchronized snapshot when making a dump of the database.

• --no-tablespaces is required because Tiger Cloud does not support tablespaces other than the default. This is a known limitation.

• --no-owner is required because Tiger Cloud's tsdbadmin user is not a superuser and cannot assign ownership in all cases. This flag means that everything is owned by the user used to connect to the target, regardless of ownership in the source. This is a known limitation.

• --no-privileges is required because the tsdbadmin user for your Tiger Cloud service is not a superuser and cannot assign privileges in all cases. This flag means that privileges assigned to other users must be reassigned in the target database as a manual clean-up task. This is a known limitation.

===== PAGE: https://docs.tigerdata.com/_partials/_create-hypertable/ =====

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

To create a hypertable:

1. Connect to your service

   In Tiger Cloud Console, click Data, then select a service.

2. Create a Postgres table

   Copy the following into your query, then click Run:

   CREATE TABLE stocks_real_time (
     time TIMESTAMPTZ NOT NULL,
     symbol TEXT NOT NULL,
     price DOUBLE PRECISION NULL,
     day_volume INT NULL
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

   You see the result immediately:

   

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_pre_data_dump_source_schema/ =====

pg_dump -d "source" \
  --format=plain \
  --quote-all-identifiers \
  --no-tablespaces \
  --no-owner \
  --no-privileges \
  --section=pre-data \
  --file=pre-data-dump.sql \
  --snapshot=$(cat /tmp/pgcopydb/snapshot)

• --section=pre-data is used to dump only the definition of tables, sequences, check constraints and inheritance hierarchy. It excludes indexes, foreign key constraints, triggers and rules.

• --snapshot is used to specified the synchronized snapshot when making a dump of the database.

• --no-tablespaces is required because Tiger Cloud does not support tablespaces other than the default. This is a known limitation.

• --no-owner is required because Tiger Cloud's tsdbadmin user is not a superuser and cannot assign ownership in all cases. This flag means that everything is owned by the user used to connect to the target, regardless of ownership in the source. This is a known limitation.

• --no-privileges is required because the tsdbadmin user for your Tiger Cloud service is not a superuser and cannot assign privileges in all cases. This flag means that privileges assigned to other users must be reassigned in the target database as a manual clean-up task. This is a known limitation.

===== PAGE: https://docs.tigerdata.com/_partials/_hypertable-detailed-size-api/ =====

hypertable_detailed_size()

Get detailed information about disk space used by a hypertable or continuous aggregate, returning size information for the table itself, any indexes on the table, any toast tables, and the total size of all. All sizes are reported in bytes. If the function is executed on a distributed hypertable, it returns size information as a separate row per node, including the access node.

When a continuous aggregate name is provided, the function transparently looks up the backing hypertable and returns its statistics instead.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get the size information for a hypertable.

-- disttable is a distributed hypertable --
SELECT * FROM hypertable_detailed_size('disttable') ORDER BY node_name;

 table_bytes | index_bytes | toast_bytes | total_bytes |  node_name
-------------+-------------+-------------+-------------+-------------
       16384 |       40960 |           0 |       57344 | data_node_1
        8192 |       24576 |           0 |       32768 | data_node_2
           0 |        8192 |           0 |        8192 |

The access node is listed without a user-given node name. Normally, the access node holds no data, but still maintains, for example, index information that occupies a small amount of disk space.

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable or continuous aggregate to show detailed size of.

Returns

Column	Type	Description
table_bytes	BIGINT	Disk space used by main_table (like pg_relation_size(main_table))
index_bytes	BIGINT	Disk space used by indexes
toast_bytes	BIGINT	Disk space of toast tables
total_bytes	BIGINT	Total disk space used by the specified table, including all indexes and TOAST data
node_name	TEXT	For distributed hypertables, this is the user-given name of the node for which the size is reported. NULL is returned for the access node and non-distributed hypertables.

If executed on a relation that is not a hypertable, the function returns NULL.

===== PAGE: https://docs.tigerdata.com/_partials/_billing-for-inactive-services/ =====

You are charged for all active services in your account, even if you are not actively using them. To reduce costs, pause or delete your unused services.

===== PAGE: https://docs.tigerdata.com/_partials/_devops-cli-install/ =====

1. Install Tiger CLI

   Use the terminal to install the CLI:

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   brew install --cask timescale/tap/tiger-cli
   

   curl -fsSL https://cli.tigerdata.com | sh
   

2. Set up API credentials

   1. Log Tiger CLI into your Tiger Data account:

      tiger auth login
      

      Tiger CLI opens Console in your browser. Log in, then click Authorize.

      You can have a maximum of 10 active client credentials. If you get an error, open credentials and delete an unused credential.

   2. Select a Tiger Cloud project:

      Auth URL is: https://console.cloud.timescale.com/oauth/authorize?client_id=lotsOfURLstuff
      Opening browser for authentication...
      Select a project:
      
      > 1. Tiger Project (tgrproject)
      2. YourCompany (Company wide project) (cpnproject)
      3. YourCompany Department (dptproject)
      
      Use ↑/↓ arrows or number keys to navigate, enter to select, q to quit
      

      If only one project is associated with your account, this step is not shown.

      Where possible, Tiger CLI stores your authentication information in the system keychain/credential manager. If that fails, the credentials are stored in ~/.config/tiger/credentials with restricted file permissions (600). By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml.

3. Test your authenticated connection to Tiger Cloud by listing services

   tiger service list
   

   This call returns something like:

   • No services:

     🏜️  No services found! Your project is looking a bit empty.
     🚀 Ready to get started? Create your first service with: tiger service create
     

   • One or more services:

     ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐
     │ SERVICE ID │        NAME         │ STATUS │    TYPE     │    REGION    │     CREATED      │
     ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤
     │ tgrservice │ tiger-agent-service │ READY  │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │
     └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘
     

===== PAGE: https://docs.tigerdata.com/_partials/_graphing-ohlcv-data/ =====

Graph OHLCV data

When you have extracted the raw OHLCV data, you can use it to graph the result in a candlestick chart, using Grafana. To do this, you need to have Grafana set up to connect to your self-hosted TimescaleDB instance.

Graphing OHLCV data

1. Ensure you have Grafana installed, and you are using the TimescaleDB database that contains the Twelve Data dataset set up as a data source.

2. In Grafana, from the Dashboards menu, click New Dashboard. In the New Dashboard page, click Add a new panel.

3. In the Visualizations menu in the top right corner, select Candlestick from the list. Ensure you have set the Twelve Data dataset as your data source.

4. Click Edit SQL and paste in the query you used to get the OHLCV values.

5. In the Format as section, select Table.

6. Adjust elements of the table as required, and click Apply to save your graph to the dashboard.

   

===== PAGE: https://docs.tigerdata.com/_partials/_create-hypertable-nyctaxis/ =====

Optimize time-series data in hypertables

Time-series data represents how a system, process, or behavior changes over time. Hypertables are Postgres tables that help you improve insert and query performance by automatically partitioning your data by time. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range.

Hypertables exist alongside regular Postgres tables. You interact with hypertables and regular Postgres tables in the same way. You use regular Postgres tables for relational data.

1. Create a hypertable to store the taxi trip data

   CREATE TABLE "rides"(
       vendor_id TEXT,
       pickup_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
       dropoff_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
       passenger_count NUMERIC,
       trip_distance NUMERIC,
       pickup_longitude  NUMERIC,
       pickup_latitude   NUMERIC,
       rate_code         INTEGER,
       dropoff_longitude NUMERIC,
       dropoff_latitude  NUMERIC,
       payment_type INTEGER,
       fare_amount NUMERIC,
       extra NUMERIC,
       mta_tax NUMERIC,
       tip_amount NUMERIC,
       tolls_amount NUMERIC,
       improvement_surcharge NUMERIC,
       total_amount NUMERIC
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='pickup_datetime',
      tsdb.create_default_indexes=false
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

2. Add another dimension to partition your hypertable more efficiently

   SELECT add_dimension('rides', by_hash('payment_type', 2));
   

3. Create an index to support efficient queries

   Index by vendor, rate code, and passenger count:

   CREATE INDEX ON rides (vendor_id, pickup_datetime DESC);
   CREATE INDEX ON rides (rate_code, pickup_datetime DESC);
   CREATE INDEX ON rides (passenger_count, pickup_datetime DESC);
   

Create standard Postgres tables for relational data

When you have other relational data that enhances your time-series data, you can create standard Postgres tables just as you would normally. For this dataset, there are two other tables of data, called payment_types and rates.

1. Add a relational table to store the payment types data

   CREATE TABLE IF NOT EXISTS "payment_types"(
       payment_type INTEGER,
       description TEXT
   );
   INSERT INTO payment_types(payment_type, description) VALUES
   (1, 'credit card'),
   (2, 'cash'),
   (3, 'no charge'),
   (4, 'dispute'),
   (5, 'unknown'),
   (6, 'voided trip');
   

2. Add a relational table to store the rates data

   CREATE TABLE IF NOT EXISTS "rates"(
       rate_code   INTEGER,
       description TEXT
   );
   INSERT INTO rates(rate_code, description) VALUES
   (1, 'standard rate'),
   (2, 'JFK'),
   (3, 'Newark'),
   (4, 'Nassau or Westchester'),
   (5, 'negotiated fare'),
   (6, 'group ride');
   

You can confirm that the scripts were successful by running the \dt command in the psql command line. You should see this:

           List of relations
 Schema |     Name      | Type  |  Owner
--------+---------------+-------+----------
 public | payment_types | table | tsdbadmin
 public | rates         | table | tsdbadmin
 public | rides         | table | tsdbadmin
(3 rows)

===== PAGE: https://docs.tigerdata.com/_partials/_integration-debezium-docker/ =====

1. Run Zookeeper in Docker

   In another Terminal window, run the following command:

   docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 quay.io/debezium/zookeeper:3.0
   

   Check the output log to see that zookeeper is running.

2. Run Kafka in Docker

   In another Terminal window, run the following command:

   docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper quay.io/debezium/kafka:3.0
   

   Check the output log to see that Kafka is running.

3. Run Kafka Connect in Docker

   In another Terminal window, run the following command:

   docker run -it --rm --name connect \
   -p 8083:8083 \
   -e GROUP_ID=1 \
   -e CONFIG_STORAGE_TOPIC=accounts \
   -e OFFSET_STORAGE_TOPIC=offsets \
   -e STATUS_STORAGE_TOPIC=storage \
   --link kafka:kafka \
   --link timescaledb:timescaledb \
   quay.io/debezium/connect:3.0
   

   Check the output log to see that Kafka Connect is running.

4. Register the Debezium Postgres source connector

   Update the <properties> for the <debezium-user> you created in your self-hosted TimescaleDB instance in the following command. Then run the command in another Terminal window:

   curl -X POST http://localhost:8083/connectors \
   -H "Content-Type: application/json" \
   -d '{
      "name": "timescaledb-connector",
      "config": {
         "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
         "database.hostname": "timescaledb",
         "database.port": "5432",
         "database.user": "<debezium-user>",
         "database.password": "<debezium-password>",
         "database.dbname" : "postgres",
         "topic.prefix": "accounts",
         "plugin.name": "pgoutput",
         "schema.include.list": "public,_timescaledb_internal",
         "transforms": "timescaledb",
         "transforms.timescaledb.type": "io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb",
         "transforms.timescaledb.database.hostname": "timescaledb",
         "transforms.timescaledb.database.port": "5432",
         "transforms.timescaledb.database.user": "<debezium-user>",
         "transforms.timescaledb.database.password": "<debezium-password>",
         "transforms.timescaledb.database.dbname": "postgres"
      }
   }'
   

5. Verify timescaledb-source-connector is included in the connector list

   1. Check the tasks associated with timescaledb-connector:

      curl -i -X GET -H "Accept:application/json" localhost:8083/connectors/timescaledb-connector
      

      You see something like:

      {"name":"timescaledb-connector","config":
      { "connector.class":"io.debezium.connector.postgresql.PostgresConnector",
      "transforms.timescaledb.database.hostname":"timescaledb",
      "transforms.timescaledb.database.password":"debeziumpassword","database.user":"debezium",
      "database.dbname":"postgres","transforms.timescaledb.database.dbname":"postgres",
      "transforms.timescaledb.database.user":"debezium",
      "transforms.timescaledb.type":"io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb",
      "transforms.timescaledb.database.port":"5432","transforms":"timescaledb",
      "schema.include.list":"public,_timescaledb_internal","database.port":"5432","plugin.name":"pgoutput",
      "topic.prefix":"accounts","database.hostname":"timescaledb","database.password":"debeziumpassword",
      "name":"timescaledb-connector"},"tasks":[{"connector":"timescaledb-connector","task":0}],"type":"source"}
      

6. Verify timescaledb-connector is running

   1. Open the Terminal window running Kafka Connect. When the connector is active, you see something like the following:

      2025-04-30 10:40:15,168 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal._hyper_1_1_chunk' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,168 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal.bgw_job_stat' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,175 INFO   Postgres|accounts|streaming  SignalProcessor started. Scheduling it every 5000ms   [io.debezium.pipeline.signal.SignalProcessor]
      2025-04-30 10:40:15,175 INFO   Postgres|accounts|streaming  Creating thread debezium-postgresconnector-accounts-SignalProcessor   [io.debezium.util.Threads]
      2025-04-30 10:40:15,175 INFO   Postgres|accounts|streaming  Starting streaming   [io.debezium.pipeline.ChangeEventSourceCoordinator]
      2025-04-30 10:40:15,176 INFO   Postgres|accounts|streaming  Retrieved latest position from stored offset 'LSN{0/1FCE570}'   [io.debezium.connector.postgresql.PostgresStreamingChangeEventSource]
      2025-04-30 10:40:15,176 INFO   Postgres|accounts|streaming  Looking for WAL restart position for last commit LSN 'null' and last change LSN 'LSN{0/1FCE570}'   [io.debezium.connector.postgresql.connection.WalPositionLocator]
      2025-04-30 10:40:15,176 INFO   Postgres|accounts|streaming  Initializing PgOutput logical decoder publication   [io.debezium.connector.postgresql.connection.PostgresReplicationConnection]
      2025-04-30 10:40:15,189 INFO   Postgres|accounts|streaming  Obtained valid replication slot ReplicationSlot [active=false, latestFlushedLsn=LSN{0/1FCCFF0}, catalogXmin=884]   [io.debezium.connector.postgresql.connection.PostgresConnection]
      2025-04-30 10:40:15,189 INFO   Postgres|accounts|streaming  Connection gracefully closed   [io.debezium.jdbc.JdbcConnection]
      2025-04-30 10:40:15,204 INFO   Postgres|accounts|streaming  Requested thread factory for component PostgresConnector, id = accounts named = keep-alive   [io.debezium.util.Threads]
      2025-04-30 10:40:15,204 INFO   Postgres|accounts|streaming  Creating thread debezium-postgresconnector-accounts-keep-alive   [io.debezium.util.Threads]
      2025-04-30 10:40:15,216 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal.bgw_policy_chunk_stats' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,216 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for 'public.accounts' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,217 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal.bgw_job_stat_history' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,217 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal._hyper_1_1_chunk' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,217 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal.bgw_job_stat' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,219 INFO   Postgres|accounts|streaming  Processing messages   [io.debezium.connector.postgresql.PostgresStreamingChangeEventSource]
      

   2. Watch the events in the accounts topic on your self-hosted TimescaleDB instance.

      In another Terminal instance, run the following command:

      docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka quay.io/debezium/kafka:3.0 watch-topic -a -k accounts
      

      You see the topics being streamed. For example:

      status-task-timescaledb-connector-0	{"state":"RUNNING","trace":null,"worker_id":"172.17.0.5:8083","generation":31}
      status-topic-timescaledb.public.accounts:connector-timescaledb-connector	{"topic":{"name":"timescaledb.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009337985}}
      status-topic-accounts._timescaledb_internal.bgw_job_stat:connector-timescaledb-connector	{"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338118}}
      status-topic-accounts._timescaledb_internal.bgw_job_stat:connector-timescaledb-connector	{"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338120}}
      status-topic-accounts._timescaledb_internal.bgw_job_stat_history:connector-timescaledb-connector	{"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat_history","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338243}}
      status-topic-accounts._timescaledb_internal.bgw_job_stat_history:connector-timescaledb-connector	{"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat_history","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338245}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338250}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}}
      ["timescaledb-connector",{"server":"accounts"}]	{"last_snapshot_record":true,"lsn":33351024,"txId":893,"ts_usec":1746009337290783,"snapshot":"INITIAL","snapshot_completed":true}
      status-connector-timescaledb-connector	{"state":"UNASSIGNED","trace":null,"worker_id":"172.17.0.5:8083","generation":31}
      status-task-timescaledb-connector-0	{"state":"UNASSIGNED","trace":null,"worker_id":"172.17.0.5:8083","generation":31}
      status-connector-timescaledb-connector	{"state":"RUNNING","trace":null,"worker_id":"172.17.0.5:8083","generation":33}
      status-task-timescaledb-connector-0	{"state":"RUNNING","trace":null,"worker_id":"172.17.0.5:8083","generation":33}
      

===== PAGE: https://docs.tigerdata.com/_partials/_integration-debezium-cloud-config-service/ =====

1. Connect to your Tiger Cloud service

   For Tiger Cloud, open an SQL editor in Tiger Cloud Console. For self-hosted, use psql.

2. Enable logical replication for your Tiger Cloud service

   1. Run the following command to enable logical replication:

      ALTER SYSTEM SET wal_level = logical;
      SELECT pg_reload_conf();
      

   2. Restart your service.

3. Create a table

   Create a table to test the integration. For example:

   CREATE TABLE sensor_data (
   id SERIAL PRIMARY KEY,
   device_id TEXT NOT NULL,
   temperature FLOAT NOT NULL,
   recorded_at TIMESTAMPTZ DEFAULT now()
   );
   

===== PAGE: https://docs.tigerdata.com/_partials/_hypercore-direct-compress/ =====

When you set timescaledb.enable_direct_compress_copy your data gets compressed in memory during ingestion with COPY statements. By writing the compressed batches immediately in the columnstore, the IO footprint is significantly lower. Also, the columnstore policy you set is less important, INSERT already produces compressed chunks.

Please note that this feature is a tech preview and not production-ready. Using this feature could lead to regressed query performance and/or storage ratio, if the ingested batches are not correctly ordered or are of too high cardinality.

To enable in-memory data compression during ingestion:

SET timescaledb.enable_direct_compress_copy=on;

Important facts

• High cardinality use cases do not produce good batches and lead to degreaded query performance.
• The columnstore is optimized to store 1000 records per batch, which is the optimal format for ingestion per segment by.
• WAL records are written for the compressed batches rather than the individual tuples.
• Currently only COPY is support, INSERT will eventually follow.
• Best results are achieved for batch ingestion with 1000 records or more, upper boundary is 10.000 records.
• Continous Aggregates are not supported at the moment.

===== PAGE: https://docs.tigerdata.com/_partials/_hypertable-size-api/ =====

hypertable_size()

Get the total disk space used by a hypertable or continuous aggregate, that is, the sum of the size for the table itself including chunks, any indexes on the table, and any toast tables. The size is reported in bytes. This is equivalent to computing the sum of total_bytes column from the output of hypertable_detailed_size function.

When a continuous aggregate name is provided, the function transparently looks up the backing hypertable and returns its statistics instead.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get the size information for a hypertable.

SELECT hypertable_size('devices');

 hypertable_size
-----------------
           73728

Get the size information for all hypertables.

SELECT hypertable_name, hypertable_size(format('%I.%I', hypertable_schema, hypertable_name)::regclass)
  FROM timescaledb_information.hypertables;

Get the size information for a continuous aggregate.

SELECT hypertable_size('device_stats_15m');

 hypertable_size
-----------------
           73728

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable or continuous aggregate to show size of.

Returns

Name	Type	Description
hypertable_size	BIGINT	Total disk space used by the specified hypertable, including all indexes and TOAST data

NULL is returned if the function is executed on a non-hypertable relation.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_explain_pg_dump_flags/ =====

• --no-tablespaces is required because Tiger Cloud does not support tablespaces other than the default. This is a known limitation.

• --no-owner is required because Tiger Cloud's tsdbadmin user is not a superuser and cannot assign ownership in all cases. This flag means that everything is owned by the user used to connect to the target, regardless of ownership in the source. This is a known limitation.

• --no-privileges is required because the tsdbadmin user for your Tiger Cloud service is not a superuser and cannot assign privileges in all cases. This flag means that privileges assigned to other users must be reassigned in the target database as a manual clean-up task. This is a known limitation.

===== PAGE: https://docs.tigerdata.com/_partials/_livesync-configure-source-database-awsrds/ =====

Updating parameters on a Postgres instance will cause an outage. Choose a time that will cause the least issues to tune this database.

1. Tune the Write Ahead Log (WAL) on the RDS/Aurora Postgres source database

   1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

   2. Click Configuration, scroll down and note the DB instance parameter group, then click Parameter Groups

      

   3. Click Create parameter group, fill in the form with the following values, then click Create.

      • Parameter group name - whatever suits your fancy.
      • Description - knock yourself out with this one.
      • Engine type - PostgreSQL
      • Parameter group family - the same as DB instance parameter group in your Configuration.

   4. In Parameter groups, select the parameter group you created, then click Edit.

   5. Update the following parameters, then click Save changes.

      • rds.logical_replication set to 1: record the information needed for logical decoding.
      • wal_sender_timeout set to 0: disable the timeout for the sender process.

   6. In RDS, navigate back to your databases, select the RDS instance to migrate, and click Modify.

   7. Scroll down to Database options, select your new parameter group, and click Continue.

   8. Click Apply immediately or choose a maintenance window, then click Modify DB instance.

      Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.

   9. Verify that the settings are live in your database.

2. Create a user for the source Postgres connector and assign permissions

   1. Create <pg connector username>:

      psql source -c "CREATE USER <pg connector username> PASSWORD '<password>'"
      

      You can use an existing user. However, you must ensure that the user has the following permissions.

   2. Grant permissions to create a replication slot:

      psql source -c "GRANT rds_replication TO <pg connector username>"
      

   3. Grant permissions to create a publication:

      psql source -c "GRANT CREATE ON DATABASE <database name> TO <pg connector username>"
      

   4. Assign the user permissions on the source database:

      psql source <<EOF
      GRANT USAGE ON SCHEMA "public" TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

      If the tables you are syncing are not in the public schema, grant the user permissions for each schema you are syncing:

      psql source <<EOF
      GRANT USAGE ON SCHEMA <schema> TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA <schema> GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

   5. On each table you want to sync, make <pg connector username> the owner:

      psql source -c 'ALTER TABLE  OWNER TO <pg connector username>;'
      

      You can skip this step if the replicating user is already the owner of the tables.

3. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_step1/ =====

1. Set up a target database instance in Tiger Cloud

Create a Tiger Cloud service.

If you intend on migrating more than 400 GB, open a support request to ensure that enough disk is pre-provisioned on your Tiger Cloud service.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

===== PAGE: https://docs.tigerdata.com/_partials/_quickstart-intro/ =====

Easily integrate your app with Tiger Cloud. Use your favorite programming language to connect to your Tiger Cloud service, create and manage hypertables, then ingest and query data.

===== PAGE: https://docs.tigerdata.com/_partials/_start-coding-node/ =====

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install Node.js.

• Install the Node.js package manager npm.

Connect to TimescaleDB

In this section, you create a connection to TimescaleDB with a common Node.js ORM (object relational mapper) called Sequelize.

1. At the command prompt, initialize a new Node.js app:

   npm init -y
   

   This creates a package.json file in your directory, which contains all of the dependencies for your project. It looks something like this:

   {
     "name": "node-sample",
     "version": "1.0.0",
     "description": "",
     "main": "index.js",
     "scripts": {
       "test": "echo \"Error: no test specified\" && exit 1"
     },
     "keywords": [],
     "author": "",
     "license": "ISC"
   }
   

2. Install Express.js:

   npm install express
   

3. Create a simple web page to check the connection. Create a new file called index.js, with this content:

   const express = require('express')
   const app = express()
   const port = 3000;
   
   app.use(express.json());
   app.get('/', (req, res) => res.send('Hello World!'))
   app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`))
   

4. Test your connection by starting the application:

   node index.js
   

In your web browser, navigate to http://localhost:3000. If the connection is successful, it shows "Hello World!"

1. Add Sequelize to your project:

   npm install sequelize sequelize-cli pg pg-hstore
   

2. Locate your TimescaleDB credentials and use them to compose a connection string for Sequelize.

   You'll need:

   • password
   • username
   • host URL
   • port
   • database name

3. Compose your connection string variable, using this format:

   'postgres://<user>:<password>@<host>:<port>/<dbname>'
   

4. Open the index.js file you created. Require Sequelize in the application, and declare the connection string:

   const Sequelize = require('sequelize')
   const sequelize = new Sequelize('postgres://<user>:<password>@<host>:<port>/<dbname>',
       {
           dialect: 'postgres',
           protocol: 'postgres',
           dialectOptions: {
               ssl: {
                   require: true,
                   rejectUnauthorized: false
               }
           }
       })
   

   Make sure you add the SSL settings in the dialectOptions sections. You can't connect to TimescaleDB using SSL without them.

5. You can test the connection by adding these lines to index.js after the app.get statement:

   sequelize.authenticate().then(() => {
       console.log('Connection has been established successfully.');
   }).catch(err => {
       console.error('Unable to connect to the database:', err);
   });
   

   Start the application on the command line:

   node index.js
   

   If the connection is successful, you'll get output like this:

   Example app listening at http://localhost:3000
   Executing (default): SELECT 1+1 AS result
   Connection has been established successfully.
   

Create a relational table

In this section, you create a relational table called page_loads.

1. Use the Sequelize command line tool to create a table and model called page_loads:

   npx sequelize model:generate --name page_loads \
   --attributes userAgent:string,time:date
   

   The output looks similar to this:

   Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11]
   
   New model was created at <PATH>.
   New migration was created at <PATH>.
   

2. Edit the migration file so that it sets up a migration key:

   'use strict';
   module.exports = {
     up: async (queryInterface, Sequelize) => {
       await queryInterface.createTable('page_loads', {
         userAgent: {
           primaryKey: true,
           type: Sequelize.STRING
         },
         time: {
           primaryKey: true,
           type: Sequelize.DATE
         }
       });
     },
     down: async (queryInterface, Sequelize) => {
       await queryInterface.dropTable('page_loads');
     }
   };
   

3. Migrate the change and make sure that it is reflected in the database:

   npx sequelize db:migrate
   

   The output looks similar to this:

   Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11]
   
   Loaded configuration file "config/config.json".
   Using environment "development".
   == 20200528195725-create-page-loads: migrating =======
   == 20200528195725-create-page-loads: migrated (0.443s)
   

4. Create the PageLoads model in your code. In the index.js file, above the app.use statement, add these lines:

   let PageLoads = sequelize.define('page_loads', {
       userAgent: {type: Sequelize.STRING, primaryKey: true },
       time: {type: Sequelize.DATE, primaryKey: true }
   }, { timestamps: false });
   

5. Instantiate a PageLoads object and save it to the database.

Create a hypertable

When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable.

1. Create a migration to modify the page_loads relational table, and change it to a hypertable by first running the following command:

   npx sequelize migration:generate --name add_hypertable
   

   The output looks similar to this:

   Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11]
   
   migrations folder at <PATH> already exists.
   New migration was created at <PATH>/20200601202912-add_hypertable.js .
   

2. In the migrations folder, there is now a new file. Open the file, and add this content:

   'use strict';
   
   module.exports = {
     up: (queryInterface, Sequelize) => {
       return queryInterface.sequelize.query("SELECT create_hypertable('page_loads', by_range('time'));");
     },
   
     down: (queryInterface, Sequelize) => {
     }
   };
   

   The by_range dimension builder is an addition to TimescaleDB 2.13.

3. At the command prompt, run the migration command:

   npx sequelize db:migrate
   

   The output looks similar to this:

   Sequelize CLI [Node: 12.16.2, CLI: 5.5.1, ORM: 5.21.11]
   
   Loaded configuration file "config/config.json".
   Using environment "development".
   == 20200601202912-add_hypertable: migrating =======
   == 20200601202912-add_hypertable: migrated (0.426s)
   

Insert rows of data

This section covers how to insert data into your hypertables.

1. In the index.js file, modify the / route to get the user-agent from the request object (req) and the current timestamp. Then, call the create method on PageLoads model, supplying the user agent and timestamp parameters. The create call executes an INSERT on the database:

   app.get('/', async (req, res) => {
       // get the user agent and current time
       const userAgent = req.get('user-agent');
       const time = new Date().getTime();
   
       try {
           // insert the record
           await PageLoads.create({
               userAgent, time
           });
   
           // send response
           res.send('Inserted!');
       } catch (e) {
           console.log('Error inserting data', e)
       }
   })
   

Execute a query

This section covers how to execute queries against your database. In this example, every time the page is reloaded, all information currently in the table is displayed.

1. Modify the / route in the index.js file to call the Sequelize findAll function and retrieve all data from the page_loads table using the PageLoads model:

   app.get('/', async (req, res) => {
       // get the user agent and current time
       const userAgent = req.get('user-agent');
       const time = new Date().getTime();
   
       try {
           // insert the record
           await PageLoads.create({
               userAgent, time
           });
   
           // now display everything in the table
           const messages = await PageLoads.findAll();
           res.send(messages);
       } catch (e) {
           console.log('Error inserting data', e)
       }
   })
   

Now, when you reload the page, you should see all of the rows currently in the page_loads table.

===== PAGE: https://docs.tigerdata.com/_partials/_release_notification/ =====

To be notified about the latest releases, in Github click Watch > Custom, then enable Releases.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_set_up_align_db_extensions_postgres_based/ =====

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

===== PAGE: https://docs.tigerdata.com/_partials/_selfhosted_cta/ =====

Deploy a Tiger Cloud service. We tune your database for performance and handle scalability, high availability, backups and management so you can relax.

===== PAGE: https://docs.tigerdata.com/_partials/_preloaded-data/ =====

If you have been provided with a pre-loaded dataset on your Tiger Cloud service, go directly to the queries section.

===== PAGE: https://docs.tigerdata.com/_partials/_cloudtrial/ =====

Your Tiger Cloud trial is completely free for you to use for the first thirty days. This gives you enough time to complete all the tutorials and run a few test projects of your own.

===== PAGE: https://docs.tigerdata.com/_partials/_data_model_metadata/ =====

You might also notice that the metadata fields are missing. Because this is a relational database, metadata can be stored in a secondary table and JOINed at query time. Learn more about TimescaleDB's support for JOINs.

===== PAGE: https://docs.tigerdata.com/_partials/_usage-based-storage-intro/ =====

Tiger Cloud charges are based on the amount of storage you use. You don't pay for fixed storage size, and you don't need to worry about scaling disk size as your data grows—we handle it all for you. To reduce your data costs further, combine hypercore, a data retention policy, and tiered storage.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_using_parallel_copy/ =====

Restoring data into a Tiger Cloud service with timescaledb-parallel-copy

1. At the command prompt, install timescaledb-parallel-copy:

   go get github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy
   

2. Use timescaledb-parallel-copy to import data into your Tiger Cloud service. Set <NUM_WORKERS> to twice the number of CPUs in your database. For example, if you have 4 CPUs, <NUM_WORKERS> should be 8.

   timescaledb-parallel-copy \
   --connection "host=<HOST> \
   user=tsdbadmin password=<PASSWORD> \
   port=<PORT> \
   dbname=tsdb \
   sslmode=require
   " \
   --table  \
   --file <FILE_NAME>.csv \
   --workers <NUM_WORKERS> \
   --reporting-period 30s
   

   Repeat for each table and hypertable you want to migrate.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_step5/ =====

5. Determine the completion point T

After dual-writes have been executing for a while, the target hypertable contains data in three time ranges: missing writes, late-arriving data, and the "consistency" range

Missing writes

If the application is made up of multiple writers, and these writers did not all simultaneously start writing into the target hypertable, there is a period of time in which not all writes have made it into the target hypertable. This period starts when the first writer begins dual-writing, and ends when the last writer begins dual-writing.

Late-arriving data

Some applications have late-arriving data: measurements which have a timestamp in the past, but which weren't written yet (for example from devices which had intermittent connectivity issues). The window of late-arriving data is between the present moment, and the maximum lateness.

Consistency range

The consistency range is the range in which there are no missing writes, and in which all data has arrived, that is between the end of the missing writes range and the beginning of the late-arriving data range.

The length of these ranges is defined by the properties of the application, there is no one-size-fits-all way to determine what they are.

Completion point

The completion point T is an arbitrarily chosen time in the consistency range. It is the point in time to which data can safely be backfilled, ensuring that there is no data loss.

The completion point should be expressed as the type of the time column of the hypertables to be backfilled. For instance, if you're using a TIMESTAMPTZ time column, then the completion point may be 2023-08-10T12:00:00.00Z. If you're using a BIGINT column it may be 1695036737000.

If you are using a mix of types for the time columns of your hypertables, you must determine the completion point for each type individually, and backfill each set of hypertables with the same type independently from those of other types.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-archlinux-based/ =====

ArchLinux packages are built by the community.

1. Install the latest Postgres and TimescaleDB packages

   sudo pacman -Syu timescaledb timescaledb-tune postgresql-libs
   

2. Initalize your Postgres instance

   sudo -u postgres initdb --locale=en_US.UTF-8 --encoding=UTF8 -D /var/lib/postgres/data --data-checksums
   

3. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

4. Enable and start Postgres

   sudo systemctl enable postgresql.service
   sudo systemctl start postgresql.service
   

5. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

6. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_since_2_20_0/ =====

Since TimescaleDB v2.20.0

===== PAGE: https://docs.tigerdata.com/_partials/_consider-cloud/ =====

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dump_roles_schema_data_postgres/ =====

1. Dump the roles from your source database

   Export your role-based security hierarchy. <db_name> has the same value as <db_name> in source. I know, it confuses me as well.

   pg_dumpall -d "source" \
     -l <db_name>
     --quote-all-identifiers \
     --roles-only \
     --file=roles.sql
   

   If you only use the default postgres role, this step is not necessary.

2. Remove roles with superuser access

   Tiger Cloud service do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e '/CREATE ROLE "tsdbadmin";/d' \
   -e '/ALTER ROLE "tsdbadmin"/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

3. Dump the source database schema and data

   The pg_dump flags remove superuser access and tablespaces from your data. When you run pgdump, check the run time, a long-running pg_dump can cause issues.

   pg_dump -d "source" \
   --format=plain \
   --quote-all-identifiers \
   --no-tablespaces \
   --no-owner \
   --no-privileges \
   --file=dump.sql
   

   To dramatically reduce the time taken to dump the source database, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

===== PAGE: https://docs.tigerdata.com/_partials/_timeseries-intro/ =====

Time-series data represents how a system, process, or behavior changes over time. For example, if you are taking measurements from a temperature gauge every five minutes, you are collecting time-series data. Another common example is stock price changes, or even the battery life of your smart phone. As these measurements change over time, each data point is recorded alongside its timestamp, allowing it to be measured, analyzed, and visualized.

Time-series data can be collected very frequently, such as financial data, or infrequently, such as weather or system measurements. It can also be collected regularly, such as every millisecond or every hour, or irregularly, such as only when a change occurs.

Databases have always had time fields, but using a special database for handling time-series data can make your database work much more effectively. Specialized time-series databases, like Timescale, are designed to handle large amounts of database writes, so they work much faster. They are also optimized to handle schema changes, and use more flexible indexing, so you don't need to spend time migrating your data whenever you make a change.

Time-series data is everywhere, but there are some environments where it is especially important to use a specialized time-series database, like Timescale:

• Monitoring computer systems: virtual machines, servers, container metrics, CPU, free memory, net/disk IOPs, service and application metrics such as request rates, and request latency.
• Financial trading systems: securities, cryptocurrencies, payments, and transaction events.
• Internet of things: data from sensors on industrial machines and equipment, wearable devices, vehicles, physical containers, pallets, and consumer devices for smart homes.
• Eventing applications: user or customer interaction data such as clickstreams, pageviews, logins, and signups.
• Business intelligence: Tracking key metrics and the overall health of the business.
• Environmental monitoring: temperature, humidity, pressure, pH, pollen count, air flow, carbon monoxide, nitrogen dioxide, or particulate matter.

===== PAGE: https://docs.tigerdata.com/_partials/_timescaledb_supported_macos/ =====

Operation system	Version
macOS	From 10.15 Catalina to 14 Sonoma

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_setup_environment/ =====

Set your connection strings

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Align the version of TimescaleDB on the source and target

1. Ensure that the source and target databases are running the same version of TimescaleDB.

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target service:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

2. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Tune your source database

You need admin rights to to update the configuration on your source database. If you are using a managed service, follow the instructions in the From MST tab on this page.

1. Install the wal2json extension on your source database

   Install wal2json on your source database.

2. Prevent Postgres from treating the data in a snapshot as outdated

   psql -X -d source -c 'alter system set old_snapshot_threshold=-1'
   

   This is not applicable if the source database is Postgres 17 or later.

3. Set the write-Ahead Log (WAL) to record the information needed for logical decoding

   psql -X -d source -c 'alter system set wal_level=logical'
   

4. Restart the source database

   Your configuration changes are now active. However, verify that the settings are live in your database.

5. Enable live-migration to replicate DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_integration-prereqs-self-only/ =====

To follow the steps on this page:

• Create a target self-hosted TimescaleDB instance.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_run_cleanup_postgres/ =====

1. Validate the migrated data

   The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app.

2. Stop app downtime

   Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service.

3. Cleanup resources associated with live-migration from your migration machine

   This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration.

   docker run --rm -it --name live-migration-clean \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest1 clean --prune
   

===== PAGE: https://docs.tigerdata.com/_partials/_create-hypertable-twelvedata-crypto/ =====

Optimize time-series data in a hypertable

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Create a hypertable to store the real-time cryptocurrency data

   Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data:

   CREATE TABLE crypto_ticks (
       "time" TIMESTAMPTZ,
       symbol TEXT,
       price DOUBLE PRECISION,
       day_volume NUMERIC
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time',
      tsdb.segmentby='symbol',
      tsdb.orderby='time DESC'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

Create a standard Postgres table for relational data

When you have relational data that enhances your time-series data, store that data in standard Postgres relational tables.

1. Add a table to store the asset symbol and name in a relational table

   CREATE TABLE crypto_assets (
       symbol TEXT UNIQUE,
       "name" TEXT
   );
   

You now have two tables within your Tiger Cloud service. A hypertable named crypto_ticks, and a normal Postgres table named crypto_assets.

===== PAGE: https://docs.tigerdata.com/_partials/_cloud-installation/ =====

Create a Tiger Data account

You create a Tiger Data account to manage your services and data in a centralized and efficient manner in Tiger Cloud Console. From there, you can create and delete services, run queries, manage access and billing, integrate other services, contact support, and more.

You create a standalone account to manage Tiger Cloud as a separate unit in your infrastructure, which includes separate billing and invoicing.

To set up Tiger Cloud:

1. Sign up for a 30-day free trial

   Open Sign up for Tiger Cloud and add your details, then click Start your free trial. You receive a confirmation email in your inbox.

2. Confirm your email address

   In the confirmation email, click the link supplied.

3. Select the pricing plan

   You are now logged into Tiger Cloud Console. You can change the pricing plan to better accommodate your growing needs on the Billing page.

To have Tiger Cloud as a part of your AWS infrastructure, you create a Tiger Data account through AWS Marketplace. In this case, Tiger Cloud is a line item in your AWS invoice.

To set up Tiger Cloud via AWS:

1. Open AWS Marketplace and search for Tiger Cloud

   You see two pricing options, pay-as-you-go and annual commit.

2. Select the pricing option that suits you and click View purchase options

3. Review and configure the purchase details, then click Subscribe

4. Click Set up your account at the top of the page

   You are redirected to Tiger Cloud Console.

5. Sign up for a 30-day free trial

   Add your details, then click Start your free trial. If you want to link an existing Tiger Data account to AWS, log in with your existing credentials.

6. Select the pricing plan

   You are now logged into Tiger Cloud Console. You can change the pricing plan later to better accommodate your growing needs on the Billing page.

7. In Confirm AWS Marketplace connection, click Connect

   Your Tiger Cloud and AWS accounts are now connected.

===== PAGE: https://docs.tigerdata.com/_partials/_cloud-connect-service/ =====

1. Check your service is running correctly

   In Tiger Cloud Console, check that your service is marked as Running.

   

2. Connect to your service

   Connect using data mode or SQL editor in Tiger Cloud Console, or psql in the command line:

   This feature is not available under the Free pricing plan.

   1. In Tiger Cloud Console, toggle Data.

   2. Select your service in the connection drop-down in the top right.

      

   3. Run a test query:

      SELECT CURRENT_DATE;
      

      This query gives you the current date, you have successfully connected to your service.

   And that is it, you are up and running. Enjoy developing with Tiger Data.

   1. In Tiger Cloud Console, select your service.

   2. Click SQL editor.

      

   3. Run a test query:

      SELECT CURRENT_DATE;
      

      This query gives you the current date, you have successfully connected to your service.

   And that is it, you are up and running. Enjoy developing with Tiger Data.

   1. Install psql.

   2. Run the following command in the terminal using the service URL from the config file you have saved during service creation:

      psql "<your-service-url>"
      

   3. Run a test query:

      SELECT CURRENT_DATE;
      

      This query returns the current date. You have successfully connected to your service.

   And that is it, you are up and running. Enjoy developing with Tiger Data.

Quick recap. You:

• Manage your services in the ops mode in Tiger Cloud Console: add read replicas and enable high availability, compress data into the columnstore, change parameters, and so on.
• Analyze your data in the data mode in Tiger Cloud Console: write queries with autocomplete, save them in folders, share them, create charts/dashboards, and much more.
• Store configuration and security information in your config file.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_step4/ =====

4. Start application in dual-write mode

With the target database set up, your application can now be started in dual-write mode.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_post_schema_caggs_etal/ =====

Migrate schema post-data

When you have migrated your table and hypertable data, migrate your Postgres schema post-data. This includes information about constraints.

Migrating schema post-data

1. At the command prompt, dump the schema post-data from your source database into a dump_post_data.dump file, using your source database connection details. Exclude Timescale-specific schemas. If you are prompted for a password, use your source database credentials:

   pg_dump -U <SOURCE_DB_USERNAME> -W \
   -h <SOURCE_DB_HOST> -p <SOURCE_DB_PORT> -Fc -v \
   --section=post-data --exclude-schema="_timescaledb*" \
   -f dump_post_data.dump <DATABASE_NAME>
   

2. Restore the dumped schema post-data from the dump_post_data.dump file into your Tiger Cloud service, using your connection details. To avoid permissions errors, include the --no-owner flag:

   pg_restore -U tsdbadmin -W \
   -h <HOST> -p <PORT> --no-owner -Fc \
   -v -d tsdb dump_post_data.dump
   

Troubleshooting

If you see these errors during the migration process, you can safely ignore them. The migration still occurs successfully.

pg_restore: error: could not execute query: ERROR:  relation "<relation_name>" already exists

pg_restore: error: could not execute query: ERROR:  trigger "ts_insert_blocker" for relation "<relation_name>" already exists

Recreate continuous aggregates

Continuous aggregates aren't migrated by default when you transfer your schema and data separately. You can restore them by recreating the continuous aggregate definitions and recomputing the results on your Tiger Cloud service. The recomputed continuous aggregates only aggregate existing data in your Tiger Cloud service. They don't include deleted raw data.

Recreating continuous aggregates

1. Connect to your source database:

   psql "postgres://<SOURCE_DB_USERNAME>:<SOURCE_DB_PASSWORD>@<SOURCE_DB_HOST>:<SOURCE_DB_PORT>/<SOURCE_DB_NAME>?sslmode=require"
   

2. Get a list of your existing continuous aggregate definitions:

   SELECT view_name, view_definition FROM timescaledb_information.continuous_aggregates;
   

   This query returns the names and definitions for all your continuous aggregates. For example:

   view_name       |                                            view_definition
   ----------------+--------------------------------------------------------------------------------------------------------
   avg_fill_levels |  SELECT round(avg(fill_measurements.fill_level), 2) AS avg_fill_level,                                +
                   |     time_bucket('01:00:00'::interval, fill_measurements."time") AS bucket,                            +
                   |     fill_measurements.sensor_id                                                                       +
                   |     FROM fill_measurements                                                                            +
                   |     GROUP BY (time_bucket('01:00:00'::interval, fill_measurements."time")), fill_measurements.sensor_id;
   (1 row)
   

3. Connect to your Tiger Cloud service:

   psql "postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

4. Recreate each continuous aggregate definition:

   CREATE MATERIALIZED VIEW <VIEW_NAME>
   WITH (timescaledb.continuous) AS
   <VIEW_DEFINITION>
   

Recreate policies

By default, policies aren't migrated when you transfer your schema and data separately. Recreate them on your Tiger Cloud service.

Recreating policies

1. Connect to your source database:

   psql "postgres://<SOURCE_DB_USERNAME>:<SOURCE_DB_PASSWORD>@<SOURCE_DB_HOST>:<SOURCE_DB_PORT>/<SOURCE_DB_NAME>?sslmode=require"
   

2. Get a list of your existing policies. This query returns a list of all your policies, including continuous aggregate refresh policies, retention policies, compression policies, and reorder policies:

   SELECT application_name, schedule_interval, retry_period,
       config, hypertable_name
       FROM timescaledb_information.jobs WHERE owner = '<SOURCE_DB_USERNAME>';
   

3. Connect to your Tiger Cloud service:

   psql "postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

4. Recreate each policy. For more information about recreating policies, see the sections on continuous-aggregate refresh policies, retention policies, Hypercore policies, and reorder policies.

Update table statistics

Update your table statistics by running ANALYZE on your entire dataset. Note that this might take some time depending on the size of your database:

ANALYZE;

Troubleshooting

If you see errors of the following form when you run ANALYZE, you can safely ignore them:

WARNING:  skipping "" --- only superuser can analyze it

The skipped tables and indexes correspond to system catalogs that can't be accessed. Skipping them does not affect statistics on your data.

===== PAGE: https://docs.tigerdata.com/_partials/_tutorials-hypercore-intro/ =====

Over time you end up with a lot of data. Since this data is mostly immutable, you can compress it to save space and avoid incurring additional cost.

TimescaleDB is built for handling event-oriented data such as time-series and fast analytical queries, it comes with support of hypercore featuring the columnstore.

Hypercore enables you to store the data in a vastly more efficient format allowing up to 90x compression ratio compared to a normal Postgres table. However, this is highly dependent on the data and configuration.

Hypercore is implemented natively in Postgres and does not require special storage formats. When you convert your data from the rowstore to the columnstore, TimescaleDB uses Postgres features to transform the data into columnar format. The use of a columnar format allows a better compression ratio since similar data is stored adjacently. For more details on the columnar format, see hypercore.

A beneficial side effect of compressing data is that certain queries are significantly faster, since less data has to be read into memory.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_awsrds_migrate_data_downtime/ =====

Prepare to migrate

1. Take the applications that connect to the RDS instance offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss. You should also ensure that your source RDS instance is not receiving any DML queries.

2. Connect to your intermediary EC2 instance

   For example:

   ssh -i "<key-pair>.pem" ubuntu@<EC2 instance's Public IPv4>
   

3. Set your connection strings

   These variables hold the connection information for the RDS instance and target Tiger Cloud service:

   export SOURCE="postgres://<Master username>:<Master password>@<Endpoint>:<Port>/<DB name>"
   export TARGET=postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require
   

   You find the connection information for SOURCE in your RDS configuration. For TARGET in the configuration file you downloaded when you created the Tiger Cloud service.

Align the extensions on the source and target

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Migrate roles from RDS to your Tiger Cloud service

Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service:

1. Dump the roles from your RDS instance

   Export your role-based security hierarchy. If you only use the default postgres role, this step is not necessary.

   pg_dumpall -d "source" \
     --quote-all-identifiers \
     --roles-only \
     --no-role-passwords \
     --file=roles.sql
   

   AWS RDS does not allow you to export passwords with roles. You assign passwords to these roles when you have uploaded them to your Tiger Cloud service.

2. Remove roles with superuser access

   Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e '/CREATE ROLE "rds/d' \
   -e '/ALTER ROLE "rds/d' \
   -e '/TO "rds/d' \
   -e '/GRANT "rds/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

3. Upload the roles to your Tiger Cloud service

   psql -X -d "target" \
     -v ON_ERROR_STOP=1 \
     --echo-errors \
     -f roles.sql
   

4. Manually assign passwords to the roles

   AWS RDS did not allow you to export passwords with roles. For each role, use the following command to manually assign a password to a role:

    psql target -c "ALTER ROLE <role name> WITH PASSWORD '<highly secure password>';"
   

Migrate data from your RDS instance to your Tiger Cloud service

1. Dump the data from your RDS instance to your intermediary EC2 instance

   The pg_dump flags remove superuser access and tablespaces from your data. When you run pgdump, check the run time, a long-running pg_dump can cause issues.

   pg_dump -d "source" \
   --format=plain \
   --quote-all-identifiers \
   --no-tablespaces \
   --no-owner \
   --no-privileges \
   --file=dump.sql
   

   To dramatically reduce the time taken to dump the RDS instance, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

2. Upload your data to your Tiger Cloud service

   psql -d target -v ON_ERROR_STOP=1 --echo-errors \
     -f dump.sql
   

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_validate_data/ =====

Validate your data, then restart your app

1. Validate the migrated data

   The contents of both databases should be the same. To check this you could compare the number of rows, or an aggregate of columns. However, the best validation method depends on your app.

2. Stop app downtime

   Once you are confident that your data is successfully replicated, configure your apps to use your Tiger Cloud service.

3. Cleanup resources associated with live-migration from your migration machine

   This command removes all resources and temporary files used in the migration process. When you run this command, you can no longer resume live-migration.

   docker run --rm -it --name live-migration-clean \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest clean --prune
   

===== PAGE: https://docs.tigerdata.com/_partials/_cloud-integrations-exporter-region/ =====

Your exporter must be in the same AWS region as the Tiger Cloud service it is attached to. If you have Tiger Cloud services running in multiple regions, create an exporter for each region.

===== PAGE: https://docs.tigerdata.com/_partials/_devops-cli-reference/ =====

Commands

You can use the following commands with Tiger CLI. For more information on each command, use the -h flag. For example: tiger auth login -h

Command	Subcommand	Description
auth		Manage authentication and credentials for your Tiger Data account
	login	Create an authenticated connection to your Tiger Data account
	logout	Remove the credentials used to create authenticated connections to Tiger Cloud
	status	Show your current authentication status and project ID
version		Show information about the currently installed version of Tiger CLI
config		Manage your Tiger CLI configuration
	show	Show the current configuration
	set <key> <value>	Set a specific value in your configuration. For example, tiger config set debug true
	unset <key>	Clear the value of a configuration parameter. For example, tiger config unset debug
	reset	Reset the configuration to the defaults. This also logs you out from the current Tiger Cloud project
service		Manage the Tiger Cloud services in this project
	create	Create a new service in this project. Possible flags are: --name: service name (auto-generated if not provided) --addons: addons to enable (time-series, ai, or none for PostgreSQL-only) --region: region code where the service will be deployed --cpu-memory: CPU/memory allocation combination --replicas: number of high-availability replicas --no-wait: don't wait for the operation to complete --wait-timeout: wait timeout duration (for example, 30m, 1h30m, 90s) --no-set-default: don't set this service as the default service --with-password: include password in output --output, -o: output format (json, yaml, table) Possible cpu-memory combinations are: shared/shared 0.5 CPU/2 GB 1 CPU/4 GB 2 CPU/8 GB 4 CPU/16 GB 8 CPU/32 GB 16 CPU/64 GB 32 CPU/128 GB
	delete <service-id>	Delete a service from this project. This operation is irreversible and requires confirmation by typing the service ID
	fork <service-id>	Fork an existing service to create a new independent copy. Key features are: Timing options: --now, --last-snapshot, --to-timestamp Resource configuration: --cpu-memory Naming: --name <name>. Defaults to {source-service-name}-fork Wait behavior: --no-wait, --wait-timeout Default service: --no-set-default
	get <service-id> (aliases: describe, show)	Show detailed information about a specific service in this project
	list	List all the services in this project
	update-password <service-id>	Update the master password for a service
db		Database operations and management
	connect <service-id>	Connect to a service
	connection-string <service-id>	Retrieve the connection string for a service
	save-password <service-id>	Save the password for a service
	test-connection <service-id>	Test the connectivity to a service
mcp		Manage the Tiger Model Context Protocol Server for AI Assistant integration
	install [client]	Install and configure Tiger Model Context Protocol Server for a specific client (claude-code, cursor, windsurf, or other). If no client is specified, you'll be prompted to select one interactively
	start	Start the Tiger Model Context Protocol Server. This is the same as tiger mcp start stdio
	start stdio	Start the Tiger Model Context Protocol Server with stdio transport (default)
	start http	Start the Tiger Model Context Protocol Server with HTTP transport. Includes flags: --port (default: 8080), --host (default: localhost)

Global flags

You can use the following global flags with Tiger CLI:

Flag	Default	Description
--analytics	true	Set to false to disable usage analytics
--color	true	Set to false to disable colored output
--config-dir string	.config/tiger	Set the directory that holds config.yaml
--debug	No debugging	Enable debug logging
--help	-	Print help about the current command. For example, tiger service --help
--password-storage string	keyring	Set the password storage method. Options are keyring, pgpass, or none
--service-id string	-	Set the Tiger Cloud service to manage
--skip-update-check	-	Do not check if a new version of Tiger CLI is available

Configuration parameters

By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml. The name of these variables matches the flags you use to update them. However, you can override them using the following environmental variables:

• Configuration parameters

  • TIGER_CONFIG_DIR: path to configuration directory (default: ~/.config/tiger)
  • TIGER_API_URL: Tiger REST API base endpoint (default: https://console.cloud.timescale.com/public/api/v1)
  • TIGER_CONSOLE_URL: URL to Tiger Cloud Console (default: https://console.cloud.timescale.com)
  • TIGER_GATEWAY_URL: URL to the Tiger Cloud Console gateway (default: https://console.cloud.timescale.com/api)
  • TIGER_DOCS_MCP: enable/disable docs MCP proxy (default: true)
  • TIGER_DOCS_MCP_URL: URL to the Tiger MCP Server for Tiger Data docs (default: https://mcp.tigerdata.com/docs)
  • TIGER_SERVICE_ID: ID for the service updated when you call CLI commands
  • TIGER_ANALYTICS: enable or disable analytics (default: true)
  • TIGER_PASSWORD_STORAGE: password storage method (keyring, pgpass, or none)
  • TIGER_DEBUG: enable/disable debug logging (default: false)
  • TIGER_COLOR: set to false to disable colored output (default: true)

• Authentication parameters

  To authenticate without using the interactive login, either:

  • Set the following parameters with your client credentials, then login:

    TIGER_PUBLIC_KEY=<public_key> TIGER_SECRET_KEY=<secret_key> TIGER_PROJECT_ID=<project_id>\
    tiger auth login
    

  • Add your client credentials to the login command:

    tiger auth login --public-key=<public_key> --secret-key=<secret-key> --project-id=<project_id>
    

===== PAGE: https://docs.tigerdata.com/_partials/_psql-installation-linux/ =====

Install psql on Linux

You can use the apt on Debian-based systems, yum on Red Hat-based systems, and pacman package manager to install the psql tool.

Installing psql using the apt package manager

1. Make sure your apt repository is up to date:

   apt-get update
   

2. Install the postgresql-client package:

   apt-get install postgresql-client
   

Installing psql using the yum package manager

1. Make sure your yum repository is up to date:

   yum update
   

2. Install the postgresql-client package:

   dnf install postgresql14
   

Installing psql using the pacman package manager

1. Make sure your pacman repository is up to date:

   pacman -Syu
   

2. Install the postgresql-client package:

   pacman -S postgresql-libs
   

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_setup_environment_mst/ =====

Set your connection strings

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Align the version of TimescaleDB on the source and target

1. Ensure that the source and target databases are running the same version of TimescaleDB.

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target service:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

2. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Tune your source database

1. Enable live-migration to replicate DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-debian/ =====

1. Install the latest Postgres packages

   sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget
   

2. Run the Postgres package setup script

   sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
   

3. Add the TimescaleDB package

   echo "deb https://packagecloud.io/timescale/timescaledb/debian/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
   

4. Install the TimescaleDB GPG key

   wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg
   

5. Update your local repository list

   sudo apt update
   

6. Install TimescaleDB

   sudo apt install timescaledb-2-postgresql-17 postgresql-client-17
   

   To install a specific TimescaleDB release, set the version. For example:

   sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'

   Older versions of TimescaleDB may not support all the OS versions listed on this page.

7. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune
   

   By default, this script is included with the timescaledb-tools package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see Configuration. If you have an issue, run sudo apt install timescaledb-tools.

8. Restart Postgres

   sudo systemctl restart postgresql
   

9. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

10. Set the password for postgres

    \password postgres
    

    When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_since_2_21_0/ =====

Since TimescaleDB v2.21.0

===== PAGE: https://docs.tigerdata.com/_partials/_where-to-next/ =====

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_backfill_getting_help/ =====

If you get stuck, you can get help by either opening a support request, or take your issue to the #migration channel in the community slack, where the developers of this migration method are there to help.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

===== PAGE: https://docs.tigerdata.com/_partials/_timescaledb/ =====

TimescaleDB is an extension for Postgres that enables time-series workloads, increasing ingest, query, storage and analytics performance.

Best practice is to run TimescaleDB in a Tiger Cloud service, but if you want to self-host you can run TimescaleDB yourself.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_tune_source_database_postgres/ =====

You need admin rights to to update the configuration on your source database. If you are using a managed service, follow the instructions in the From AWS RDS/Aurora tab on this page.

1. Install the wal2json extension on your source database

   Install wal2json on your source database.

2. Prevent Postgres from treating the data in a snapshot as outdated

   psql -X -d source -c 'alter system set old_snapshot_threshold=-1'
   

   This is not applicable if the source database is Postgres 17 or later.

3. Set the write-Ahead Log (WAL) to record the information needed for logical decoding

   psql -X -d source -c 'alter system set wal_level=logical'
   

4. Restart the source database

   Your configuration changes are now active. However, verify that the settings are live in your database.

5. Enable live-migration to replicate DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_tune_source_database/ =====

You need admin rights to to update the configuration on your source database. If you are using a managed service, follow the instructions in the From MST tab on this page.

1. Install the wal2json extension on your source database

   Install wal2json on your source database.

2. Prevent Postgres from treating the data in a snapshot as outdated

   psql -X -d source -c 'alter system set old_snapshot_threshold=-1'
   

   This is not applicable if the source database is Postgres 17 or later.

3. Set the write-Ahead Log (WAL) to record the information needed for logical decoding

   psql -X -d source -c 'alter system set wal_level=logical'
   

4. Restart the source database

   Your configuration changes are now active. However, verify that the settings are live in your database.

5. Enable live-migration to replicate DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

===== PAGE: https://docs.tigerdata.com/_partials/_timescaledb-config/ =====

Just as you can tune settings in Postgres, TimescaleDB provides a number of configuration settings that may be useful to your specific installation and performance needs. These can also be set within the postgresql.conf file or as command-line parameters when starting Postgres.

Query Planning and Execution

timescaledb.enable_chunkwise_aggregation (bool)

If enabled, aggregations are converted into partial aggregations during query planning. The first part of the aggregation is executed on a per-chunk basis. Then, these partial results are combined and finalized. Splitting aggregations decreases the size of the created hash tables and increases data locality, which speeds up queries.

timescaledb.vectorized_aggregation (bool)

Enables or disables the vectorized optimizations in the query executor. For example, the sum() aggregation function on compressed chunks can be optimized in this way.

timescaledb.enable_merge_on_cagg_refresh (bool)

Set to ON to dramatically decrease the amount of data written on a continuous aggregate in the presence of a small number of changes, reduce the i/o cost of refreshing a continuous aggregate, and generate fewer Write-Ahead Logs (WAL). Only works for continuous aggregates that don't have compression enabled.

Please refer to the Grand Unified Configuration (GUC) parameters for a complete list.

Policies

timescaledb.max_background_workers (int)

Max background worker processes allocated to TimescaleDB. Set to at least 1 + the number of databases loaded with the TimescaleDB extension in a Postgres instance. Default value is 16.

Tiger Cloud service tuning

timescaledb.disable_load (bool)

Disable the loading of the actual extension

Administration

timescaledb.restoring (bool)

Set TimescaleDB in restoring mode. It is disabled by default.

timescaledb.license (string)

Change access to features based on the TimescaleDB license in use. For example, setting timescaledb.license to apache limits TimescaleDB to features that are implemented under the Apache 2 license. The default value is timescale, which allows access to all features.

timescaledb.telemetry_level (enum)

Telemetry settings level. Level used to determine which telemetry to send. Can be set to off or basic. Defaults to basic.

timescaledb.last_tuned (string)

Records last time timescaledb-tune ran.

timescaledb.last_tuned_version (string)

Version of timescaledb-tune used to tune when it runs.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_self_postgres_timescaledb_compatibility/ =====

TimescaleDB version	Postgres 17	Postgres 16	Postgres 15	Postgres 14	Postgres 13	Postgres 12	Postgres 11	Postgres 10
2.22.x	✅	✅	✅	❌	❌	❌	❌	❌
2.21.x	✅	✅	✅	❌	❌	❌	❌	❌
2.20.x	✅	✅	✅	❌	❌	❌	❌	❌
2.17 - 2.19	✅	✅	✅	✅	❌	❌	❌	❌
2.16.x	❌	✅	✅	✅	❌	❌	❌	❌
2.13 - 2.15	❌	✅	✅	✅	✅	❌	❌	❌
2.12.x	❌	❌	✅	✅	✅	❌	❌	❌
2.10.x	❌	❌	✅	✅	✅	✅	❌	❌
2.5 - 2.9	❌	❌	❌	✅	✅	✅	❌	❌
2.4	❌	❌	❌	❌	✅	✅	❌	❌
2.1 - 2.3	❌	❌	❌	❌	✅	✅	✅	❌
2.0	❌	❌	❌	❌	❌	✅	✅	❌
1.7	❌	❌	❌	❌	❌	✅	✅	✅

We recommend not using TimescaleDB with Postgres 17.1, 16.5, 15.9, 14.14, 13.17, 12.21. These minor versions introduced a breaking binary interface change that, once identified, was reverted in subsequent minor Postgres versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with Postgres 17.2, 16.6, etc and higher. Users of Tiger Cloud and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected.

===== PAGE: https://docs.tigerdata.com/_partials/_devops-mcp-commands-cli/ =====

You can use the following Tiger CLI commands to run Tiger MCP Server:

Usage: tiger mcp [subcommand] --<flags>

Command	Subcommand	Description
mcp		Manage the Tiger Model Context Protocol Server
	install [client]	Install and configure Tiger MCP Server for a specific client installed on your developer device. Supported clients are: claude-code, cursor, windsurf, codex, gemini/gemini-cli, vscode/code/vs-code. Flags: --no-backup: do not back up the existing configuration --config-path: open the configuration file at a specific location
	start	Start the Tiger MCP Server. This is the same as tiger mcp start stdio
	start stdio	Start the Tiger MCP Server with stdio transport
	start http	Start the Tiger MCP Server with HTTP transport. This option is for users who wish to access Tiger Model Context Protocol Server without using stdio. For example, your AI Assistant does not support stdio, or you do not want to run CLI on your device. Flags are: --port <port number>: the default is 8000 --host <hostname>: the default is localhost

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_to_upload_to_target/ =====

1. Take the applications that connect to the source database offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss.

2. Set your connection strings

   These variables hold the connection information for the source database and target Tiger Cloud service:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

   You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

3. Ensure that the source and target databases are running the same version of TimescaleDB

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target source:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

4. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

5. Dump the roles from your source database

   Export your role-based security hierarchy. If you only use the default postgres role, this step is not necessary.

   pg_dumpall -d "source" \
     --quote-all-identifiers \
     --roles-only \
     --file=roles.sql
   

6. Remove roles with superuser access

   Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

7. Dump the source database schema and data

   The pg_dump flags remove superuser access and tablespaces from your data. When you run pgdump, check the run time, a long-running pg_dump can cause issues.

   pg_dump -d "source" \
   --format=plain \
   --quote-all-identifiers \
   --no-tablespaces \
   --no-owner \
   --no-privileges \
   --file=dump.sql
   

   To dramatically reduce the time taken to dump the source database, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

===== PAGE: https://docs.tigerdata.com/_partials/_devops-cli-global-flags/ =====

Flag	Default	Description
--analytics	true	Set to false to disable usage analytics
--color	true	Set to false to disable colored output
--config-dir string	.config/tiger	Set the directory that holds config.yaml
--debug	No debugging	Enable debug logging
--help	-	Print help about the current command. For example, tiger service --help
--password-storage string	keyring	Set the password storage method. Options are keyring, pgpass, or none
--service-id string	-	Set the Tiger Cloud service to manage
--skip-update-check	-	Do not check if a new version of Tiger CLI is available

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dump_awsrds/ =====

Create an intermediary EC2 Ubuntu instance

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.
2. Click Actions > Set up EC2 connection. Press Create EC2 instance and use the following settings:
   • AMI: Ubuntu Server.
   • Key pair: use an existing pair or create a new one that you will use to access the intermediary machine.
   • VPC: by default, this is the same as the database instance.
   • Configure Storage: adjust the volume to at least the size of RDS/Aurora Postgres instance you are migrating from. You can reduce the space used by your data on Tiger Cloud using Hypercore.
3. Click Lauch instance. AWS creates your EC2 instance, then click Connect to instance > SSH client. Follow the instructions to create the connection to your intermediary EC2 instance.

Install the psql client tools on the intermediary instance

1. Connect to your intermediary EC2 instance. For example:

   ssh -i "<key-pair>.pem" ubuntu@<EC2 instance's Public IPv4>
   

2. On your intermediary EC2 instance, install the Postgres client.

   sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
   wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo tee /etc/apt/trusted.gpg.d/pgdg.asc &>/dev/null
   sudo apt update
   sudo apt install postgresql-client-16 -y # "postgresql-client-16" if your source DB is using PG 16.
   psql --version && pg_dump --version
   

Keep this terminal open, you need it to connect to the RDS/Aurora Postgres instance for migration.

Set up secure connectivity between your RDS/Aurora Postgres and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.

2. Scroll down to Security group rules (1) and select the EC2 Security Group - Inbound group. The Security Groups (1) window opens. Click the Security group ID, then click Edit inbound rules

   

3. On your intermediary EC2 instance, get your local IP address:

   ec2metadata --local-ipv4
   

   Bear with me on this one, you need this IP address to enable access to your RDS/Aurora Postgres instance.

4. In Edit inbound rules, click Add rule, then create a PostgreSQL, TCP rule granting access to the local IP address for your EC2 instance (told you :-)). Then click Save rules.

   

Test the connection between your RDS/Aurora Postgres and EC2 instances

1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS/Aurora Postgres instance to migrate.

2. On your intermediary EC2 instance, use the values of Endpoint, Port, Master username, and DB name to create the postgres connectivity string to the SOURCE variable.

   

   export SOURCE="postgres://<Master username>:<Master password>@<Endpoint>:<Port>/<DB name>"
   

   The value of Master password was supplied when this RDS/Aurora Postgres instance was created.

3. Test your connection:

   psql -d source
   

   You are connected to your RDS/Aurora Postgres instance from your intermediary EC2 instance.

Migrate your data to your Tiger Cloud service

To securely migrate data from your RDS instance:

Prepare to migrate

1. Take the applications that connect to the RDS instance offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss. You should also ensure that your source RDS instance is not receiving any DML queries.

2. Connect to your intermediary EC2 instance

   For example:

   ssh -i "<key-pair>.pem" ubuntu@<EC2 instance's Public IPv4>
   

3. Set your connection strings

   These variables hold the connection information for the RDS instance and target Tiger Cloud service:

   export SOURCE="postgres://<Master username>:<Master password>@<Endpoint>:<Port>/<DB name>"
   export TARGET=postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require
   

   You find the connection information for SOURCE in your RDS configuration. For TARGET in the configuration file you downloaded when you created the Tiger Cloud service.

Align the extensions on the source and target

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Migrate roles from RDS to your Tiger Cloud service

Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service:

1. Dump the roles from your RDS instance

   Export your role-based security hierarchy. If you only use the default postgres role, this step is not necessary.

   pg_dumpall -d "source" \
     --quote-all-identifiers \
     --roles-only \
     --no-role-passwords \
     --file=roles.sql
   

   AWS RDS does not allow you to export passwords with roles. You assign passwords to these roles when you have uploaded them to your Tiger Cloud service.

2. Remove roles with superuser access

   Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e '/CREATE ROLE "rds/d' \
   -e '/ALTER ROLE "rds/d' \
   -e '/TO "rds/d' \
   -e '/GRANT "rds/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

3. Upload the roles to your Tiger Cloud service

   psql -X -d "target" \
     -v ON_ERROR_STOP=1 \
     --echo-errors \
     -f roles.sql
   

4. Manually assign passwords to the roles

   AWS RDS did not allow you to export passwords with roles. For each role, use the following command to manually assign a password to a role:

    psql target -c "ALTER ROLE <role name> WITH PASSWORD '<highly secure password>';"
   

Migrate data from your RDS instance to your Tiger Cloud service

1. Dump the data from your RDS instance to your intermediary EC2 instance

   The pg_dump flags remove superuser access and tablespaces from your data. When you run pgdump, check the run time, a long-running pg_dump can cause issues.

   pg_dump -d "source" \
   --format=plain \
   --quote-all-identifiers \
   --no-tablespaces \
   --no-owner \
   --no-privileges \
   --file=dump.sql
   

   To dramatically reduce the time taken to dump the RDS instance, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

2. Upload your data to your Tiger Cloud service

   psql -d target -v ON_ERROR_STOP=1 --echo-errors \
     -f dump.sql
   

Validate your Tiger Cloud service and restart your app

1. Update the table statistics.

   psql target -c "ANALYZE;"
   

2. Verify the data in the target Tiger Cloud service.

   Check that your data is correct, and returns the results that you expect,

3. Enable any Tiger Cloud features you want to use.

   Migration from Postgres moves the data only. Now manually enable Tiger Cloud features like hypertables, hypercore or data retention while your database is offline.

4. Reconfigure your app to use the target database, then restart it.

===== PAGE: https://docs.tigerdata.com/_partials/_service-overview/ =====

You manage your Tiger Cloud services and interact with your data in Tiger Cloud Console using the following modes:

Ops mode	Data mode
You use the ops mode to: Ensure data security with high availability and read replicas Save money with columnstore compression and tiered storage Enable Postgres extensions to add extra functionality Increase security using VPCs Perform day-to-day administration	Powered by PopSQL, you use the data mode to: Write queries with autocomplete Visualize data with charts and dashboards Schedule queries and dashboards for alerts or recurring reports Share queries and dashboards Interact with your data on auto-pilot with SQL assistant This feature is not available under the Free pricing plan.

===== PAGE: https://docs.tigerdata.com/_partials/_livesync-terminal/ =====

Prerequisites

Best practice is to use an Ubuntu EC2 instance hosted in the same region as your Tiger Cloud service to move data. That is, the machine you run the commands on to move your data from your source database to your target Tiger Cloud service.

Before you move your data:

• Create a target Tiger Cloud service.

  Each Tiger Cloud service has a single Postgres instance that supports the most popular extensions. Tiger Cloud services do not support tablespaces, and there is no superuser associated with a service. Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window.

• To ensure that maintenance does not run while migration is in progress, best practice is to adjust the maintenance window.

• Ensure that the source Postgres instance and the target Tiger Cloud service have the same extensions installed.

  The source Postgres connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table.

• Install Docker on your sync machine.

  For a better experience, use a 4 CPU/16GB EC2 instance or greater to run the source Postgres connector.

• Install the Postgres client tools on your sync machine.

  This includes psql, pg_dump, pg_dumpall, and vacuumdb commands.

Limitations

• The schema is not migrated by the source Postgres connector, you use pg_dump/pg_restore to migrate it.

• This works for Postgres databases only as source. TimescaleDB is not yet supported.

• The source must be running Postgres 13 or later.

• Schema changes must be co-ordinated.

  Make compatible changes to the schema in your Tiger Cloud service first, then make the same changes to the source Postgres instance.

• Ensure that the source Postgres instance and the target Tiger Cloud service have the same extensions installed.

  The source Postgres connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table.

• There is WAL volume growth on the source Postgres instance during large table copy.

• Continuous aggregate invalidation

  The connector uses session_replication_role=replica during data replication, which prevents table triggers from firing. This includes the internal triggers that mark continuous aggregates as invalid when underlying data changes.

  If you have continuous aggregates on your target database, they do not automatically refresh for data inserted during the migration. This limitation only applies to data below the continuous aggregate's materialization watermark. For example, backfilled data. New rows synced above the continuous aggregate watermark are used correctly when refreshing.

  This can lead to:

  • Missing data in continuous aggregates for the migration period.
  • Stale aggregate data.
  • Queries returning incomplete results.

  If the continuous aggregate exists in the source database, best practice is to add it to the Postgres connector publication. If it only exists on the target database, manually refresh the continuous aggregate using the force option of refresh_continuous_aggregate.

Set your connection strings

The <user> in the SOURCE connection must have the replication role granted in order to create a replication slot.

These variables hold the connection information for the source database and target Tiger Cloud service. In Terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"

You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Tune your source database

Updating parameters on a Postgres instance will cause an outage. Choose a time that will cause the least issues to tune this database.

1. Update the DB instance parameter group for your source database

   1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

   2. Click Configuration, scroll down and note the DB instance parameter group, then click Parameter groups

      

   3. Click Create parameter group, fill in the form with the following values, then click Create.

      • Parameter group name - whatever suits your fancy.
      • Description - knock yourself out with this one.
      • Engine type - PostgreSQL
      • Parameter group family - the same as DB instance parameter group in your Configuration.

   4. In Parameter groups, select the parameter group you created, then click Edit.

   5. Update the following parameters, then click Save changes.

      • rds.logical_replication set to 1: record the information needed for logical decoding.
      • wal_sender_timeout set to 0: disable the timeout for the sender process.

   6. In RDS, navigate back to your databases, select the RDS instance to migrate, and click Modify.

   7. Scroll down to Database options, select your new parameter group, and click Continue.

   8. Click Apply immediately or choose a maintenance window, then click Modify DB instance.

      Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.

   9. Verify that the settings are live in your database.

2. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

1. Tune the Write Ahead Log (WAL) on the Postgres source database

   psql source <<EOF
   ALTER SYSTEM SET wal_level='logical';
   ALTER SYSTEM SET max_wal_senders=10;
   ALTER SYSTEM SET wal_sender_timeout=0;
   EOF
   

   • GUC “wal_level” as “logical”
   • GUC “max_wal_senders” as 10
   • GUC “wal_sender_timeout” as 0

   This will require a restart of the Postgres source database.

2. Create a user for the connector and assign permissions

   1. Create <pg connector username>:

      psql source -c "CREATE USER <pg connector username> PASSWORD '<password>'"
      

      You can use an existing user. However, you must ensure that the user has the following permissions.

   2. Grant permissions to create a replication slot:

      psql source -c "ALTER ROLE <pg connector username> REPLICATION"
      

   3. Grant permissions to create a publication:

      psql source -c "GRANT CREATE ON DATABASE <database name> TO <pg connector username>"
      

   4. Assign the user permissions on the source database:

      psql source <<EOF
      GRANT USAGE ON SCHEMA "public" TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

      If the tables you are syncing are not in the public schema, grant the user permissions for each schema you are syncing:

      psql source <<EOF
      GRANT USAGE ON SCHEMA <schema> TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA <schema> GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

   5. On each table you want to sync, make <pg connector username> the owner:

      psql source -c 'ALTER TABLE  OWNER TO <pg connector username>;'
      

      You can skip this step if the replicating user is already the owner of the tables.

3. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

Migrate the table schema to the Tiger Cloud service

Use pg_dump to:

1. Download the schema from the source database

pg_dump source \
--no-privileges \
--no-owner \
--no-publications \
--no-subscriptions \
--no-table-access-method \
--no-tablespaces \
--schema-only \
--file=schema.sql

1. Apply the schema on the target service

psql target -f schema.sql

Convert partitions and tables with time-series data into hypertables

For efficient querying and analysis, you can convert tables which contain time-series or events data, and tables that are already partitioned using Postgres declarative partition into hypertables.

1. Convert tables to hypertables

   Run the following on each table in the target Tiger Cloud service to convert it to a hypertable:

   psql -X -d target -c "SELECT public.create_hypertable('', by_range('<partition column>', '<chunk interval>'::interval));"
   

   For example, to convert the metrics table into a hypertable with time as a partition column and 1 day as a partition interval:

   psql -X -d target -c "SELECT public.create_hypertable('public.metrics', by_range('time', '1 day'::interval));"
   

2. Convert Postgres partitions to hypertables

   Rename the partition and create a new regular table with the same name as the partitioned table, then convert to a hypertable:

   psql target -f - <<'EOF'
      BEGIN;
      ALTER TABLE public.events RENAME TO events_part;
      CREATE TABLE public.events(LIKE public.events_part INCLUDING ALL);
      SELECT create_hypertable('public.events', by_range('time', '1 day'::interval));
      COMMIT;
   

EOF

## Specify the tables to synchronize

After the schema is migrated, you [`CREATE PUBLICATION`][create-publication] on the source database that
specifies the tables to synchronize.

1. **Create a publication that specifies the table to synchronize**

A `PUBLICATION` enables you to synchronize some or all the tables in the schema or database.

```sql
CREATE PUBLICATION <publication_name> FOR TABLE , ;

To add tables after to an existing publication, use [ALTER PUBLICATION][alter-publication]**

ALTER PUBLICATION <publication_name> ADD TABLE ;

1. Publish the Postgres declarative partitioned table

   ALTER PUBLICATION <publication_name> SET(publish_via_partition_root=true);
   

   To convert partitioned table to hypertable, follow Convert partitions and tables with time-series data into hypertables.

2. Stop syncing a table in the PUBLICATION, use DROP TABLE

   ALTER PUBLICATION <publication_name> DROP TABLE ;
   

Synchronize data to your Tiger Cloud service

You use the source Postgres connector docker image to synchronize changes in real time from a Postgres database instance to a Tiger Cloud service:

1. Start the source Postgres connector

   As you run the source Postgres connector continuously, best practice is to run it as a Docker daemon.

   docker run -d --rm --name livesync timescale/live-sync:v0.1.25 run \
      --publication <publication_name> --subscription <subscription_name> \
      --source source --target target --table-map
   

   --publication: The name of the publication as you created in the previous step. To use multiple publications, repeat the --publication flag.

   --subscription: The name that identifies the subscription on the target Tiger Cloud service.

   --source: The connection string to the source Postgres database.

   --target: The connection string to the target Tiger Cloud service.

   --table-map: (Optional) A JSON string that maps source tables to target tables. If not provided, the source and target table names are assumed to be the same. For example, to map the source table metrics to the target table metrics_data:

   --table-map '{"source": {"schema": "public", "table": "metrics"}, "target": {"schema": "public", "table": "metrics_data"}}'
   

   To map only the schema, use:

   --table-map '{"source": {"schema": "public"}, "target": {"schema": "analytics"}}'
   

   This flag can be repeated for multiple table mappings.

2. Capture logs

   Once the source Postgres connector is running as a docker daemon, you can also capture the logs:

   docker logs -f livesync
   

3. View the progress of tables being synchronized

   List the tables being synchronized by the source Postgres connector using the _ts_live_sync.subscription_rel table in the target Tiger Cloud service:

   psql target -c "SELECT * FROM _ts_live_sync.subscription_rel"
   

   You see something like the following:

   subname	pubname	schemaname	tablename	rrelid	state	lsn	updated_at	last_error	created_at	rows_copied	approximate_rows	bytes_copied	approximate_size	target_schema	target_table
   livesync	analytics	public	metrics	20856	r	6/1A8CBA48	2025-06-24 06:16:21.434898+00		2025-06-24 06:03:58.172946+00	18225440	18225440	1387359359	1387359359	public	metrics

   The state column indicates the current state of the table synchronization. Possible values for state are:

   state	description
   d	initial table data sync
   f	initial table data sync completed
   s	catching up with the latest changes
   r	table is ready, syncing live changes

   To see the replication lag, run the following against the SOURCE database:

   psql source -f - <<'EOF'
   SELECT
      slot_name,
      pg_size_pretty(pg_current_wal_flush_lsn() - confirmed_flush_lsn) AS lag
   FROM pg_replication_slots
   WHERE slot_name LIKE 'live_sync_%' AND slot_type = 'logical'
   

EOF

1. **Add or remove tables from the publication**

To add tables, use [ALTER PUBLICATION .. ADD TABLE][alter-publication]**

```sql
ALTER PUBLICATION <publication_name> ADD TABLE ;

To remove tables, use ALTER PUBLICATION .. DROP TABLE**

ALTER PUBLICATION <publication_name> DROP TABLE ;

1. Update table statistics

   If you have a large table, you can run ANALYZE on the target Tiger Cloud service to update the table statistics after the initial sync is complete.

   This helps the query planner make better decisions for query execution plans.

   vacuumdb --analyze --verbose --dbname=target
   

2. Stop the source Postgres connector

   docker stop live-sync
   

3. (Optional) Reset sequence nextval on the target Tiger Cloud service

   The source Postgres connector does not automatically reset the sequence nextval on the target Tiger Cloud service.

   Run the following script to reset the sequence for all tables that have a serial or identity column in the target Tiger Cloud service:

   psql target -f - <<'EOF'
      DO $$
   DECLARE
     rec RECORD;
   BEGIN
     FOR rec IN (
       SELECT
         sr.target_schema  AS table_schema,
         sr.target_table   AS table_name,
         col.column_name,
         pg_get_serial_sequence(
           sr.target_schema || '.' || sr.target_table,
           col.column_name
         ) AS seqname
       FROM _ts_live_sync.subscription_rel AS sr
       JOIN information_schema.columns AS col
         ON col.table_schema = sr.target_schema
        AND col.table_name   = sr.target_table
       WHERE col.column_default LIKE 'nextval(%'  -- only serial/identity columns
     ) LOOP
       EXECUTE format(
         'SELECT setval(%L,
            COALESCE((SELECT MAX(%I) FROM %I.%I), 0) + 1,
            false
          );',
         rec.seqname,       -- the sequence identifier
         rec.column_name,   -- the column to MAX()
         rec.table_schema,  -- schema for MAX()
         rec.table_name     -- table for MAX()
       );
     END LOOP;
   END;
   $$ LANGUAGE plpgsql;
   

EOF

1. **Clean up**

Use the `--drop` flag to remove the replication slots created by the source Postgres connector on the source database.

```shell
docker run -it --rm --name livesync timescale/live-sync:v0.1.25 run \
   --publication <publication_name> --subscription <subscription_name> \
   --source source --target target \
   --drop

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dump_mst/ =====

Prepare to migrate

1. Take the applications that connect to the source database offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss.

2. Set your connection strings

   These variables hold the connection information for the source database and target Tiger Cloud service:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

   You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Align the version of TimescaleDB on the source and target

1. Ensure that the source and target databases are running the same version of TimescaleDB.

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target service:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

2. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Migrate the roles from TimescaleDB to your Tiger Cloud service

Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service:

1. Dump the roles from your source database

   Export your role-based security hierarchy. <db_name> has the same value as <db_name> in source. I know, it confuses me as well.

   pg_dumpall -d "source" \
     -l <db_name>  \
     --quote-all-identifiers \
     --roles-only \
     --no-role-passwords \
     --file=roles.sql
   

   MST does not allow you to export passwords with roles. You assign passwords to these roles when you have uploaded them to your Tiger Cloud service.

2. Remove roles with superuser access

   Tiger Cloud services do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   

-e '/DROP ROLE IF EXISTS "postgres";/d'
-e '/DROP ROLE IF EXISTS "tsdbadmin";/d'
-e '/CREATE ROLE "postgres";/d'
-e '/ALTER ROLE "postgres"/d'
-e '/CREATE ROLE "rds/d'
-e '/ALTER ROLE "rds/d'
-e '/TO "rds/d'
-e '/GRANT "rds/d'
-e '/GRANT "pg_read_all_stats" TO "tsdbadmin"/d'
-e 's/(NO)SUPERUSER//g'
-e 's/(NO)REPLICATION//g'
-e 's/(NO)BYPASSRLS//g'
-e 's/GRANTED BY "[^"]"//g'
-e '/CREATE ROLE "tsdbadmin";/d'
-e '/ALTER ROLE "tsdbadmin"/d'
-e 's/WITH ADMIN OPTION,/WITH /g'
-e 's/WITH ADMIN OPTION//g'
-e 's/GRANTED BY "."//g'
-e '/GRANT "pg_." TO/d'
-e '/CREATE ROLE "_aiven";/d'
-e '/ALTER ROLE "_aiven"/d'
-e '/GRANT SET ON PARAMETER "pgaudit.[^"]+" TO "_tsdbadmin_auditing"/d'
-e '/GRANT SET ON PARAMETER "anon.[^"]+" TO "tsdbadmin_group"/d'
roles.sql

1. **Dump the source database schema and data**

The `pg_dump` flags remove superuser access and tablespaces from your data. When you run
`pgdump`, check the run time, [a long-running `pg_dump` can cause issues][long-running-pgdump].

```bash
pg_dump -d "source" \
--format=plain \
--quote-all-identifiers \
--no-tablespaces \
--no-owner \
--no-privileges \
--file=dump.sql

To dramatically reduce the time taken to dump the source database, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

Upload your data to the target Tiger Cloud service

This command uses the timescaledb_pre_restore and timescaledb_post_restore functions to put your database in the correct state.

1. Upload your data

   psql target -v ON_ERROR_STOP=1 --echo-errors \
   -f roles.sql \
   -c "SELECT timescaledb_pre_restore();" \
   -f dump.sql \
   -c "SELECT timescaledb_post_restore();"
   

2. Manually assign passwords to the roles

   MST did not allow you to export passwords with roles. For each role, use the following command to manually assign a password to a role:

    psql target -c "ALTER ROLE <role name> WITH PASSWORD '<highly secure password>';"
   

Validate your Tiger Cloud service and restart your app

1. Update the table statistics.

   psql target -c "ANALYZE;"
   

2. Verify the data in the target Tiger Cloud service.

   Check that your data is correct, and returns the results that you expect,

3. Enable any Tiger Cloud features you want to use.

   Migration from Postgres moves the data only. Now manually enable Tiger Cloud features like hypertables, hypercore or data retention while your database is offline.

4. Reconfigure your app to use the target database, then restart it.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-rocky/ =====

Tiger Data supports Rocky Linux 8 and 9 on amd64 only.

1. Update your local repository list

   sudo dnf update -y
   sudo dnf install -y epel-release
   

2. Install the latest Postgres packages

   sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
   

3. Add the TimescaleDB repository

   sudo tee /etc/yum.repos.d/timescale_timescaledb.repo <<EOL
   [timescale_timescaledb]
   name=timescale_timescaledb
   baseurl=https://packagecloud.io/timescale/timescaledb/el/9/\$basearch
   repo_gpgcheck=1
   gpgcheck=0
   enabled=1
   gpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey
   sslverify=1
   sslcacert=/etc/pki/tls/certs/ca-bundle.crt
   metadata_expire=300
   EOL
   

4. Disable the built-in PostgreSQL module

   This is for Rocky Linux 9 only.

   sudo dnf module disable postgresql -y
   

5. Install TimescaleDB

   To avoid errors, do not install TimescaleDB Apache 2 Edition and TimescaleDB Community Edition at the same time.

   sudo dnf install -y postgresql16-server postgresql16-contrib timescaledb-2-postgresql-16
   

6. Initialize the Postgres instance

```bash
sudo /usr/pgsql-16/bin/postgresql-16-setup initdb
```

1. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune --pg-config=/usr/pgsql-16/bin/pg_config
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

2. Enable and start Postgres

   sudo systemctl enable postgresql-16
   sudo systemctl start postgresql-16
   

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are now in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_validate_and_restart_app/ =====

1. Update the table statistics.

   psql target -c "ANALYZE;"
   

2. Verify the data in the target Tiger Cloud service.

   Check that your data is correct, and returns the results that you expect,

3. Enable any Tiger Cloud features you want to use.

   Migration from Postgres moves the data only. Now manually enable Tiger Cloud features like hypertables, hypercore or data retention while your database is offline.

4. Reconfigure your app to use the target database, then restart it.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-redhat-based/ =====

1. Install the latest Postgres packages

   sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-$(rpm -E %{rhel})-x86_64/pgdg-redhat-repo-latest.noarch.rpm
   

   sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/F-$(rpm -E %{fedora})-x86_64/pgdg-fedora-repo-latest.noarch.rpm
   

2. Add the TimescaleDB repository

   sudo tee /etc/yum.repos.d/timescale_timescaledb.repo <<EOL
   [timescale_timescaledb]
   name=timescale_timescaledb
   baseurl=https://packagecloud.io/timescale/timescaledb/el/$(rpm -E %{rhel})/\$basearch
   repo_gpgcheck=1
   gpgcheck=0
   enabled=1
   gpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey
   sslverify=1
   sslcacert=/etc/pki/tls/certs/ca-bundle.crt
   metadata_expire=300
   EOL
   

   sudo tee /etc/yum.repos.d/timescale_timescaledb.repo <<EOL
   [timescale_timescaledb]
   name=timescale_timescaledb
   baseurl=https://packagecloud.io/timescale/timescaledb/el/9/\$basearch
   repo_gpgcheck=1
   gpgcheck=0
   enabled=1
   gpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey
   sslverify=1
   sslcacert=/etc/pki/tls/certs/ca-bundle.crt
   metadata_expire=300
   EOL
   

3. Update your local repository list

   sudo yum update
   

4. Install TimescaleDB

   To avoid errors, do not install TimescaleDB Apache 2 Edition and TimescaleDB Community Edition at the same time.

   sudo yum install timescaledb-2-postgresql-17 postgresql17
   

   On Red Hat Enterprise Linux 8 and later, disable the built-in Postgres module:

   sudo dnf -qy module disable postgresql

5. Initialize the Postgres instance

```bash
sudo /usr/pgsql-17/bin/postgresql-17-setup initdb
```

1. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune --pg-config=/usr/pgsql-17/bin/pg_config
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

2. Enable and start Postgres

   sudo systemctl enable postgresql-17
   sudo systemctl start postgresql-17
   

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are now in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_sunsetted_2_14_0/ =====

Sunsetted since TimescaleDB v2.14.0

===== PAGE: https://docs.tigerdata.com/_partials/_real-time-aggregates/ =====

In TimescaleDB v2.13 and later, real-time aggregates are DISABLED by default. In earlier versions, real-time aggregates are ENABLED by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-ubuntu/ =====

1. Install the latest Postgres packages

   sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget
   

2. Run the Postgres package setup script

   sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
   

   echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
   

3. Install the TimescaleDB GPG key

   wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg
   

   For Ubuntu 21.10 and earlier use the following command:

   wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add -

4. Update your local repository list

   sudo apt update
   

5. Install TimescaleDB

   sudo apt install timescaledb-2-postgresql-17 postgresql-client-17
   

   To install a specific TimescaleDB release, set the version. For example:

   sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'

   Older versions of TimescaleDB may not support all the OS versions listed on this page.

6. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune
   

   By default, this script is included with the timescaledb-tools package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see Configuration. If you have an issue, run sudo apt install timescaledb-tools.

7. Restart Postgres

   sudo systemctl restart postgresql
   

8. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

9. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_caggs-one-step-policy/ =====

Use a one-step policy definition to set a {props.policyType} policy on a continuous aggregate

In TimescaleDB 2.8 and above, policy management on continuous aggregates is simplified. You can add, change, or remove the refresh, compression, and data retention policies on a continuous aggregate using a one-step API. For more information, see the APIs for adding policies, altering policies, and removing policies. Note that this feature is experimental.

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

When you change policies with this API, the changes apply to the continuous aggregate, not to the original hypertable. For example, if you use this API to set a retention policy of 20 days, chunks older than 20 days are dropped from the continuous aggregate. The retention policy of the original hypertable remains unchanged.

===== PAGE: https://docs.tigerdata.com/_partials/_start-coding-golang/ =====

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install Go.
• Install the PGX driver for Go.

Connect to your Tiger Cloud service

In this section, you create a connection to Tiger Cloud using the PGX driver. PGX is a toolkit designed to help Go developers work directly with Postgres. You can use it to help your Go application interact directly with TimescaleDB.

1. Locate your TimescaleDB credentials and use them to compose a connection string for PGX.

   You'll need:

   • password
   • username
   • host URL
   • port number
   • database name

2. Compose your connection string variable as a libpq connection string, using this format:

   connStr := "postgres://username:password@host:port/dbname"
   

   If you're using a hosted version of TimescaleDB, or if you need an SSL connection, use this format instead:

   connStr := "postgres://username:password@host:port/dbname?sslmode=require"
   

3. You can check that you're connected to your database with this hello world program:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5"
   )
   
   //connect to database using a single connection
   func main() {
       /***********************************************/
       /* Single Connection to TimescaleDB/ PostgreSQL */
       /***********************************************/
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       conn, err := pgx.Connect(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer conn.Close(ctx)
   
       //run a simple query to check our connection
       var greeting string
       err = conn.QueryRow(ctx, "select 'Hello, Timescale!'").Scan(&greeting)
       if err != nil {
           fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err)
           os.Exit(1)
       }
       fmt.Println(greeting)
   }
   
   

   If you'd like to specify your connection string as an environment variable, you can use this syntax to access it in place of the connStr variable:

   os.Getenv("DATABASE_CONNECTION_STRING")
   

Alternatively, you can connect to TimescaleDB using a connection pool. Connection pooling is useful to conserve computing resources, and can also result in faster database queries:

1. To create a connection pool that can be used for concurrent connections to your database, use the pgxpool.New() function instead of pgx.Connect(). Also note that this script imports github.com/jackc/pgx/v5/pgxpool, instead of pgx/v5 which was used to create a single connection:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
   
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       //run a simple query to check our connection
       var greeting string
       err = dbpool.QueryRow(ctx, "select 'Hello, Tiger Data (but concurrently)'").Scan(&greeting)
       if err != nil {
           fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err)
           os.Exit(1)
       }
       fmt.Println(greeting)
   }
   

Create a relational table

In this section, you create a table called sensors which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called sensor_data which holds the measurements of those sensors. The measurements contain the time, sensor_id, temperature reading, and CPU percentage of the sensors.

1. Compose a string that contains the SQL statement to create a relational table. This example creates a table called sensors, with columns for ID, type, and location:

   queryCreateTable := `CREATE TABLE sensors (id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50));`
   

2. Execute the CREATE TABLE statement with the Exec() function on the dbpool object, using the arguments of the current context and the statement string you created:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* Create relational table                      */
       /********************************************/
   
       //Create relational table called sensors
       queryCreateTable := `CREATE TABLE sensors (id SERIAL PRIMARY KEY, type VARCHAR(50), location VARCHAR(50));`
       _, err = dbpool.Exec(ctx, queryCreateTable)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to create SENSORS table: %v\n", err)
           os.Exit(1)
       }
       fmt.Println("Successfully created relational table SENSORS")
   }
   

Generate a hypertable

When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable.

1. Create a variable for the CREATE TABLE SQL statement for your hypertable. Notice how the hypertable has the compulsory time column:

   queryCreateTable := `CREATE TABLE sensor_data (
           time TIMESTAMPTZ NOT NULL,
           sensor_id INTEGER,
           temperature DOUBLE PRECISION,
           cpu DOUBLE PRECISION,
           FOREIGN KEY (sensor_id) REFERENCES sensors (id));
           `
   

2. Formulate the SELECT statement to convert the table into a hypertable. You must specify the table name to convert to a hypertable, and its time column name as the second argument. For more information, see the create_hypertable docs:

   queryCreateHypertable := `SELECT create_hypertable('sensor_data', by_range('time'));`
   

   The by_range dimension builder is an addition to TimescaleDB 2.13.

3. Execute the CREATE TABLE statement and SELECT statement which converts the table into a hypertable. You can do this by calling the Exec() function on the dbpool object, using the arguments of the current context, and the queryCreateTable and queryCreateHypertable statement strings:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* Create Hypertable                        */
       /********************************************/
       // Create hypertable of time-series data called sensor_data
       queryCreateTable := `CREATE TABLE sensor_data (
           time TIMESTAMPTZ NOT NULL,
           sensor_id INTEGER,
           temperature DOUBLE PRECISION,
           cpu DOUBLE PRECISION,
           FOREIGN KEY (sensor_id) REFERENCES sensors (id));
           `
   
       queryCreateHypertable := `SELECT create_hypertable('sensor_data', by_range('time'));`
   
       //execute statement
       _, err = dbpool.Exec(ctx, queryCreateTable+queryCreateHypertable)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to create the `sensor_data` hypertable: %v\n", err)
           os.Exit(1)
       }
       fmt.Println("Successfully created hypertable `sensor_data`")
   }
   

Insert rows of data

You can insert rows into your database in a couple of different ways. Each of these example inserts the data from the two arrays, sensorTypes and sensorLocations, into the relational table named sensors.

The first example inserts a single row of data at a time. The second example inserts multiple rows of data. The third example uses batch inserts to speed up the process.

1. Open a connection pool to the database, then use the prepared statements to formulate an INSERT SQL statement, and execute it:

   package main
   
   import (
       "context"
       "fmt"
       "os"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* INSERT into  relational table            */
       /********************************************/
       //Insert data into relational table
   
       // Slices of sample data to insert
       // observation i has type sensorTypes[i] and location sensorLocations[i]
       sensorTypes := []string{"a", "a", "b", "b"}
       sensorLocations := []string{"floor", "ceiling", "floor", "ceiling"}
   
       for i := range sensorTypes {
           //INSERT statement in SQL
           queryInsertMetadata := `INSERT INTO sensors (type, location) VALUES ($1, $2);`
   
           //Execute INSERT command
           _, err := dbpool.Exec(ctx, queryInsertMetadata, sensorTypes[i], sensorLocations[i])
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to insert data into database: %v\n", err)
               os.Exit(1)
           }
           fmt.Printf("Inserted sensor (%s, %s) into database \n", sensorTypes[i], sensorLocations[i])
       }
       fmt.Println("Successfully inserted all sensors into database")
   }
   

Instead of inserting a single row of data at a time, you can use this procedure to insert multiple rows of data, instead:

1. This example uses Postgres to generate some sample time-series to insert into the sensor_data hypertable. Define the SQL statement to generate the data, called queryDataGeneration. Then use the .Query() function to execute the statement and return the sample data. The data returned by the query is stored in results, a slice of structs, which is then used as a source to insert data into the hypertable:

   package main
   
   import (
       "context"
       "fmt"
       "os"
       "time"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       // Generate data to insert
   
       //SQL query to generate sample data
       queryDataGeneration := `
           SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
           floor(random() * (3) + 1)::int as sensor_id,
           random()*100 AS temperature,
           random() AS cpu
           `
       //Execute query to generate samples for sensor_data hypertable
       rows, err := dbpool.Query(ctx, queryDataGeneration)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   
       fmt.Println("Successfully generated sensor data")
   
       //Store data generated in slice results
       type result struct {
           Time        time.Time
           SensorId    int
           Temperature float64
           CPU         float64
       }
   
       var results []result
       for rows.Next() {
           var r result
           err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
               os.Exit(1)
           }
           results = append(results, r)
       }
   
       // Any errors encountered by rows.Next or rows.Scan are returned here
       if rows.Err() != nil {
           fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
           os.Exit(1)
       }
   
       // Check contents of results slice
       fmt.Println("Contents of RESULTS slice")
       for i := range results {
           var r result
           r = results[i]
           fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU)
       }
   }
   

2. Formulate an SQL insert statement for the sensor_data hypertable:

   //SQL query to generate sample data
   queryInsertTimeseriesData := `
       INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4);
       `
   

3. Execute the SQL statement for each sample in the results slice:

   //Insert contents of results slice into TimescaleDB
   for i := range results {
       var r result
       r = results[i]
       _, err := dbpool.Exec(ctx, queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to insert sample into TimescaleDB %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   }
   fmt.Println("Successfully inserted samples into sensor_data hypertable")
   

4. This example main.go generates sample data and inserts it into the sensor_data hypertable:

   package main
   
   import (
       "context"
       "fmt"
       "os"
       "time"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       /********************************************/
       /* Connect using Connection Pool            */
       /********************************************/
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* Insert data into hypertable              */
       /********************************************/
       // Generate data to insert
   
       //SQL query to generate sample data
       queryDataGeneration := `
           SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
           floor(random() * (3) + 1)::int as sensor_id,
           random()*100 AS temperature,
           random() AS cpu
           `
       //Execute query to generate samples for sensor_data hypertable
       rows, err := dbpool.Query(ctx, queryDataGeneration)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   
       fmt.Println("Successfully generated sensor data")
   
       //Store data generated in slice results
       type result struct {
           Time        time.Time
           SensorId    int
           Temperature float64
           CPU         float64
       }
       var results []result
       for rows.Next() {
           var r result
           err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
               os.Exit(1)
           }
           results = append(results, r)
       }
       // Any errors encountered by rows.Next or rows.Scan are returned here
       if rows.Err() != nil {
           fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
           os.Exit(1)
       }
   
       // Check contents of results slice
       fmt.Println("Contents of RESULTS slice")
       for i := range results {
           var r result
           r = results[i]
           fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU)
       }
   
       //Insert contents of results slice into TimescaleDB
       //SQL query to generate sample data
       queryInsertTimeseriesData := `
           INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4);
           `
   
       //Insert contents of results slice into TimescaleDB
       for i := range results {
           var r result
           r = results[i]
           _, err := dbpool.Exec(ctx, queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to insert sample into TimescaleDB %v\n", err)
               os.Exit(1)
           }
           defer rows.Close()
       }
       fmt.Println("Successfully inserted samples into sensor_data hypertable")
   }
   

Inserting multiple rows of data using this method executes as many insert statements as there are samples to be inserted. This can make ingestion of data slow. To speed up ingestion, you can batch insert data instead.

Here's a sample pattern for how to do so, using the sample data you generated in the previous procedure. It uses the pgx Batch object:

1. This example batch inserts data into the database:

   package main
   
   import (
       "context"
       "fmt"
       "os"
       "time"
   
       "github.com/jackc/pgx/v5"
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       /********************************************/
       /* Connect using Connection Pool            */
       /********************************************/
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       // Generate data to insert
   
       //SQL query to generate sample data
       queryDataGeneration := `
           SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
           floor(random() * (3) + 1)::int as sensor_id,
           random()*100 AS temperature,
           random() AS cpu
           `
   
       //Execute query to generate samples for sensor_data hypertable
       rows, err := dbpool.Query(ctx, queryDataGeneration)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to generate sensor data: %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   
       fmt.Println("Successfully generated sensor data")
   
       //Store data generated in slice results
       type result struct {
           Time        time.Time
           SensorId    int
           Temperature float64
           CPU         float64
       }
       var results []result
       for rows.Next() {
           var r result
           err = rows.Scan(&r.Time, &r.SensorId, &r.Temperature, &r.CPU)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
               os.Exit(1)
           }
           results = append(results, r)
       }
       // Any errors encountered by rows.Next or rows.Scan are returned here
       if rows.Err() != nil {
           fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
           os.Exit(1)
       }
   
       // Check contents of results slice
       /*fmt.Println("Contents of RESULTS slice")
       for i := range results {
           var r result
           r = results[i]
           fmt.Printf("Time: %s | ID: %d | Temperature: %f | CPU: %f |\n", &r.Time, r.SensorId, r.Temperature, r.CPU)
       }*/
   
       //Insert contents of results slice into TimescaleDB
       //SQL query to generate sample data
       queryInsertTimeseriesData := `
           INSERT INTO sensor_data (time, sensor_id, temperature, cpu) VALUES ($1, $2, $3, $4);
           `
   
       /********************************************/
       /* Batch Insert into TimescaleDB            */
       /********************************************/
       //create batch
       batch := &pgx.Batch{}
       //load insert statements into batch queue
       for i := range results {
           var r result
           r = results[i]
           batch.Queue(queryInsertTimeseriesData, r.Time, r.SensorId, r.Temperature, r.CPU)
       }
       batch.Queue("select count(*) from sensor_data")
   
       //send batch to connection pool
       br := dbpool.SendBatch(ctx, batch)
       //execute statements in batch queue
       _, err = br.Exec()
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to execute statement in batch queue %v\n", err)
           os.Exit(1)
       }
       fmt.Println("Successfully batch inserted data")
   
       //Compare length of results slice to size of table
       fmt.Printf("size of results: %d\n", len(results))
       //check size of table for number of rows inserted
       // result of last SELECT statement
       var rowsInserted int
       err = br.QueryRow().Scan(&rowsInserted)
       fmt.Printf("size of table: %d\n", rowsInserted)
   
       err = br.Close()
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to closer batch %v\n", err)
           os.Exit(1)
       }
   }
   

Execute a query

This section covers how to execute queries against your database.

1. Define the SQL query you'd like to run on the database. This example uses a SQL query that combines time-series and relational data. It returns the average CPU values for every 5 minute interval, for sensors located on location ceiling and of type a:

   // Formulate query in SQL
   // Note the use of prepared statement placeholders $1 and $2
   queryTimebucketFiveMin := `
       SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
       FROM sensor_data
       JOIN sensors ON sensors.id = sensor_data.sensor_id
       WHERE sensors.location = $1 AND sensors.type = $2
       GROUP BY five_min
       ORDER BY five_min DESC;
       `
   

2. Use the .Query() function to execute the query string. Make sure you specify the relevant placeholders:

   //Execute query on TimescaleDB
   rows, err := dbpool.Query(ctx, queryTimebucketFiveMin, "ceiling", "a")
   if err != nil {
       fmt.Fprintf(os.Stderr, "Unable to execute query %v\n", err)
       os.Exit(1)
   }
   defer rows.Close()
   
   fmt.Println("Successfully executed query")
   

3. Access the rows returned by .Query(). Create a struct with fields representing the columns that you expect to be returned, then use the rows.Next() function to iterate through the rows returned and fill results with the array of structs. This uses the rows.Scan() function, passing in pointers to the fields that you want to scan for results.

   This example prints out the results returned from the query, but you might want to use those results for some other purpose. Once you've scanned through all the rows returned you can then use the results array however you like.

   //Do something with the results of query
   // Struct for results
   type result2 struct {
       Bucket time.Time
       Avg    float64
   }
   
   // Print rows returned and fill up results slice for later use
   var results []result2
   for rows.Next() {
       var r result2
       err = rows.Scan(&r.Bucket, &r.Avg)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
           os.Exit(1)
       }
       results = append(results, r)
       fmt.Printf("Time bucket: %s | Avg: %f\n", &r.Bucket, r.Avg)
   }
   
   // Any errors encountered by rows.Next or rows.Scan are returned here
   if rows.Err() != nil {
       fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
       os.Exit(1)
   }
   
   // use results here…
   

4. This example program runs a query, and accesses the results of that query:

   package main
   
   import (
       "context"
       "fmt"
       "os"
       "time"
   
       "github.com/jackc/pgx/v5/pgxpool"
   )
   
   func main() {
       ctx := context.Background()
       connStr := "yourConnectionStringHere"
       dbpool, err := pgxpool.New(ctx, connStr)
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
           os.Exit(1)
       }
       defer dbpool.Close()
   
       /********************************************/
       /* Execute a query                          */
       /********************************************/
   
       // Formulate query in SQL
       // Note the use of prepared statement placeholders $1 and $2
       queryTimebucketFiveMin := `
           SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
           FROM sensor_data
           JOIN sensors ON sensors.id = sensor_data.sensor_id
           WHERE sensors.location = $1 AND sensors.type = $2
           GROUP BY five_min
           ORDER BY five_min DESC;
           `
   
       //Execute query on TimescaleDB
       rows, err := dbpool.Query(ctx, queryTimebucketFiveMin, "ceiling", "a")
       if err != nil {
           fmt.Fprintf(os.Stderr, "Unable to execute query %v\n", err)
           os.Exit(1)
       }
       defer rows.Close()
   
       fmt.Println("Successfully executed query")
   
       //Do something with the results of query
       // Struct for results
       type result2 struct {
           Bucket time.Time
           Avg    float64
       }
   
       // Print rows returned and fill up results slice for later use
       var results []result2
       for rows.Next() {
           var r result2
           err = rows.Scan(&r.Bucket, &r.Avg)
           if err != nil {
               fmt.Fprintf(os.Stderr, "Unable to scan %v\n", err)
               os.Exit(1)
           }
           results = append(results, r)
           fmt.Printf("Time bucket: %s | Avg: %f\n", &r.Bucket, r.Avg)
       }
       // Any errors encountered by rows.Next or rows.Scan are returned here
       if rows.Err() != nil {
           fmt.Fprintf(os.Stderr, "rows Error: %v\n", rows.Err())
           os.Exit(1)
       }
   }
   

Next steps

Now that you're able to connect, read, and write to a TimescaleDB instance from your Go application, be sure to check out these advanced TimescaleDB tutorials:

• Refer to the pgx documentation for more information about pgx.
• Get up and running with TimescaleDB with the Getting Started tutorial.
• Want fast inserts on CSV data? Check out TimescaleDB parallel copy, a tool for fast inserts, written in Go.

===== PAGE: https://docs.tigerdata.com/_partials/_start-coding-python/ =====

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install the psycopg2 library.

For more information, see the psycopg2 documentation.

• Create a Python virtual environment.

Connect to TimescaleDB

In this section, you create a connection to TimescaleDB using the psycopg2 library. This library is one of the most popular Postgres libraries for Python. It allows you to execute raw SQL queries efficiently and safely, and prevents common attacks such as SQL injection.

1. Import the psycogpg2 library:

   import psycopg2
   

2. Locate your TimescaleDB credentials and use them to compose a connection string for psycopg2.

   You'll need:

   • password
   • username
   • host URL
   • port
   • database name

3. Compose your connection string variable as a libpq connection string, using this format:

   CONNECTION = "postgres://username:password@host:port/dbname"
   

   If you're using a hosted version of TimescaleDB, or generally require an SSL connection, use this version instead:

   CONNECTION = "postgres://username:password@host:port/dbname?sslmode=require"
   

   Alternatively you can specify each parameter in the connection string as follows

   CONNECTION = "dbname=tsdb user=tsdbadmin password=secret host=host.com port=5432 sslmode=require"
   

   This method of composing a connection string is for test or development purposes only. For production, use environment variables for sensitive details like your password, hostname, and port number.

4. Use the psycopg2 connect function to create a new database session and create a new cursor object to interact with the database.

   In your main function, add these lines:

   CONNECTION = "postgres://username:password@host:port/dbname"
   with psycopg2.connect(CONNECTION) as conn:
       cursor = conn.cursor()
       # use the cursor to interact with your database
       # cursor.execute("SELECT * FROM table")
   

   Alternatively, you can create a connection object and pass the object around as needed, like opening a cursor to perform database operations:

   CONNECTION = "postgres://username:password@host:port/dbname"
   conn = psycopg2.connect(CONNECTION)
   cursor = conn.cursor()
   # use the cursor to interact with your database
   cursor.execute("SELECT 'hello world'")
   print(cursor.fetchone())
   

Create a relational table

In this section, you create a table called sensors which holds the ID, type, and location of your fictional sensors. Additionally, you create a hypertable called sensor_data which holds the measurements of those sensors. The measurements contain the time, sensor_id, temperature reading, and CPU percentage of the sensors.

1. Compose a string which contains the SQL statement to create a relational table. This example creates a table called sensors, with columns id, type and location:

   query_create_sensors_table = """CREATE TABLE sensors (
                                       id SERIAL PRIMARY KEY,
                                       type VARCHAR(50),
                                       location VARCHAR(50)
                                   );
                                   """
   

2. Open a cursor, execute the query you created in the previous step, and commit the query to make the changes persistent. Afterward, close the cursor to clean up:

   cursor = conn.cursor()
   # see definition in Step 1
   cursor.execute(query_create_sensors_table)
   conn.commit()
   cursor.close()
   

Create a hypertable

When you have created the relational table, you can create a hypertable. Creating tables and indexes, altering tables, inserting data, selecting data, and most other tasks are executed on the hypertable.

1. Create a string variable that contains the CREATE TABLE SQL statement for your hypertable. Notice how the hypertable has the compulsory time column:

   # create sensor data hypertable
   query_create_sensordata_table = """CREATE TABLE sensor_data (
                                           time TIMESTAMPTZ NOT NULL,
                                           sensor_id INTEGER,
                                           temperature DOUBLE PRECISION,
                                           cpu DOUBLE PRECISION,
                                           FOREIGN KEY (sensor_id) REFERENCES sensors (id)
                                       );
                                       """
   

2. Formulate a SELECT statement that converts the sensor_data table to a hypertable. You must specify the table name to convert to a hypertable, and the name of the time column as the two arguments. For more information, see the create_hypertable docs:

   query_create_sensordata_hypertable = "SELECT create_hypertable('sensor_data', by_range('time'));"
   

   The by_range dimension builder is an addition to TimescaleDB 2.13.

3. Open a cursor with the connection, execute the statements from the previous steps, commit your changes, and close the cursor:

   cursor = conn.cursor()
   cursor.execute(query_create_sensordata_table)
   cursor.execute(query_create_sensordata_hypertable)
   # commit changes to the database to make changes persistent
   conn.commit()
   cursor.close()
   

Insert rows of data

You can insert data into your hypertables in several different ways. In this section, you can use psycopg2 with prepared statements, or you can use pgcopy for a faster insert.

1. This example inserts a list of tuples, or relational data, called sensors, into the relational table named sensors. Open a cursor with a connection to the database, use prepared statements to formulate the INSERT SQL statement, and then execute that statement:

   sensors = [('a', 'floor'), ('a', 'ceiling'), ('b', 'floor'), ('b', 'ceiling')]
   cursor = conn.cursor()
   for sensor in sensors:
     try:
       cursor.execute("INSERT INTO sensors (type, location) VALUES (%s, %s);",
                   (sensor[0], sensor[1]))
     except (Exception, psycopg2.Error) as error:
       print(error.pgerror)
   conn.commit()
   

2. Alternatively, you can pass variables to the cursor.execute function and separate the formulation of the SQL statement, SQL, from the data being passed with it into the prepared statement, data:

   SQL = "INSERT INTO sensors (type, location) VALUES (%s, %s);"
   sensors = [('a', 'floor'), ('a', 'ceiling'), ('b', 'floor'), ('b', 'ceiling')]
   cursor = conn.cursor()
   for sensor in sensors:
     try:
       data = (sensor[0], sensor[1])
       cursor.execute(SQL, data)
     except (Exception, psycopg2.Error) as error:
       print(error.pgerror)
   conn.commit()
   

If you choose to use pgcopy instead, install the pgcopy package using pip, and then add this line to your list of import statements:

from pgcopy import CopyManager

1. Generate some random sensor data using the generate_series function provided by Postgres. This example inserts a total of 480 rows of data (4 readings, every 5 minutes, for 24 hours). In your application, this would be the query that saves your time-series data into the hypertable:

   # for sensors with ids 1-4
   for id in range(1, 4, 1):
       data = (id,)
       # create random data
       simulate_query = """SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
                               %s as sensor_id,
                               random()*100 AS temperature,
                               random() AS cpu;
                               """
       cursor.execute(simulate_query, data)
       values = cursor.fetchall()
   

2. Define the column names of the table you want to insert data into. This example uses the sensor_data hypertable created earlier. This hypertable consists of columns named time, sensor_id, temperature and cpu. The column names are defined in a list of strings called cols:

   cols = ['time', 'sensor_id', 'temperature', 'cpu']
   

3. Create an instance of the pgcopy CopyManager, mgr, and pass the connection variable, hypertable name, and list of column names. Then use the copy function of the CopyManager to insert the data into the database quickly using pgcopy.

   mgr = CopyManager(conn, 'sensor_data', cols)
   mgr.copy(values)
   

4. Commit to persist changes:

   conn.commit()
   

5. The full sample code to insert data into TimescaleDB using pgcopy, using the example of sensor data from four sensors:

   # insert using pgcopy
   def fast_insert(conn):
       cursor = conn.cursor()
   
       # for sensors with ids 1-4
       for id in range(1, 4, 1):
           data = (id,)
           # create random data
           simulate_query = """SELECT generate_series(now() - interval '24 hour', now(), interval '5 minute') AS time,
                                   %s as sensor_id,
                                   random()*100 AS temperature,
                                   random() AS cpu;
                                   """
           cursor.execute(simulate_query, data)
           values = cursor.fetchall()
   
           # column names of the table you're inserting into
           cols = ['time', 'sensor_id', 'temperature', 'cpu']
   
           # create copy manager with the target table and insert
           mgr = CopyManager(conn, 'sensor_data', cols)
           mgr.copy(values)
   
       # commit after all sensor data is inserted
       # could also commit after each sensor insert is done
       conn.commit()
   

6. You can also check if the insertion worked:

   cursor.execute("SELECT * FROM sensor_data LIMIT 5;")
   print(cursor.fetchall())
   

Execute a query

This section covers how to execute queries against your database.

The first procedure shows a simple SELECT * query. For more complex queries, you can use prepared statements to ensure queries are executed safely against the database.

For more information about properly using placeholders in psycopg2, see the basic module usage document. For more information about how to execute more complex queries in psycopg2, see the psycopg2 documentation.

Execute a query

1. Define the SQL query you'd like to run on the database. This example is a simple SELECT statement querying each row from the previously created sensor_data table.

   query = "SELECT * FROM sensor_data;"
   

2. Open a cursor from the existing database connection, conn, and then execute the query you defined:

   cursor = conn.cursor()
   query = "SELECT * FROM sensor_data;"
   cursor.execute(query)
   

3. To access all resulting rows returned by your query, use one of pyscopg2's results retrieval methods, such as fetchall() or fetchmany(). This example prints the results of the query, row by row. Note that the result of fetchall() is a list of tuples, so you can handle them accordingly:

   cursor = conn.cursor()
   query = "SELECT * FROM sensor_data;"
   cursor.execute(query)
   for row in cursor.fetchall():
       print(row)
   cursor.close()
   

4. If you want a list of dictionaries instead, you can define the cursor using DictCursor:

   cursor = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
   

   Using this cursor, cursor.fetchall() returns a list of dictionary-like objects.

For more complex queries, you can use prepared statements to ensure queries are executed safely against the database.

Execute queries using prepared statements

1. Write the query using prepared statements:

   # query with placeholders
   cursor = conn.cursor()
   query = """
              SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
              FROM sensor_data
              JOIN sensors ON sensors.id = sensor_data.sensor_id
              WHERE sensors.location = %s AND sensors.type = %s
              GROUP BY five_min
              ORDER BY five_min DESC;
              """
   location = "floor"
   sensor_type = "a"
   data = (location, sensor_type)
   cursor.execute(query, data)
   results = cursor.fetchall()
   

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_pg_dump_do_not_recommend_for_large_migration/ =====

If you want to migrate more than 400GB of data, create a Tiger Cloud Console support request, or send us an email at support@tigerdata.com saying how much data you want to migrate. We pre-provision your Tiger Cloud service for you.

===== PAGE: https://docs.tigerdata.com/_partials/_livesync-console/ =====

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

  You need your connection details.

• Install the Postgres client tools on your sync machine.

• Ensure that the source Postgres instance and the target Tiger Cloud service have the same extensions installed.

  The source Postgres connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table.

Limitations

• The source Postgres instance must be accessible from the Internet.

  Services hosted behind a firewall or VPC are not supported. This functionality is on the roadmap.

• Indexes, including the primary key and unique constraints, are not migrated to the target Tiger Cloud service.

  We recommend that, depending on your query patterns, you create only the necessary indexes on the target Tiger Cloud service.

• This works for Postgres databases only as source. TimescaleDB is not yet supported.

• The source must be running Postgres 13 or later.

• Schema changes must be co-ordinated.

  Make compatible changes to the schema in your Tiger Cloud service first, then make the same changes to the source Postgres instance.

• Ensure that the source Postgres instance and the target Tiger Cloud service have the same extensions installed.

  The source Postgres connector does not create extensions on the target. If the table uses column types from an extension, first create the extension on the target Tiger Cloud service before syncing the table.

• There is WAL volume growth on the source Postgres instance during large table copy.

• Continuous aggregate invalidation

  The connector uses session_replication_role=replica during data replication, which prevents table triggers from firing. This includes the internal triggers that mark continuous aggregates as invalid when underlying data changes.

  If you have continuous aggregates on your target database, they do not automatically refresh for data inserted during the migration. This limitation only applies to data below the continuous aggregate's materialization watermark. For example, backfilled data. New rows synced above the continuous aggregate watermark are used correctly when refreshing.

  This can lead to:

  • Missing data in continuous aggregates for the migration period.
  • Stale aggregate data.
  • Queries returning incomplete results.

  If the continuous aggregate exists in the source database, best practice is to add it to the Postgres connector publication. If it only exists on the target database, manually refresh the continuous aggregate using the force option of refresh_continuous_aggregate.

Set your connection string

This variable holds the connection information for the source database. In the terminal on your migration machine, set the following:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"

Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. This tool requires a direct connection to the database to function properly.

Tune your source database

Updating parameters on a Postgres instance will cause an outage. Choose a time that will cause the least issues to tune this database.

1. Tune the Write Ahead Log (WAL) on the RDS/Aurora Postgres source database

   1. In https://console.aws.amazon.com/rds/home#databases:, select the RDS instance to migrate.

   2. Click Configuration, scroll down and note the DB instance parameter group, then click Parameter Groups

      

   3. Click Create parameter group, fill in the form with the following values, then click Create.

      • Parameter group name - whatever suits your fancy.
      • Description - knock yourself out with this one.
      • Engine type - PostgreSQL
      • Parameter group family - the same as DB instance parameter group in your Configuration.

   4. In Parameter groups, select the parameter group you created, then click Edit.

   5. Update the following parameters, then click Save changes.

      • rds.logical_replication set to 1: record the information needed for logical decoding.
      • wal_sender_timeout set to 0: disable the timeout for the sender process.

   6. In RDS, navigate back to your databases, select the RDS instance to migrate, and click Modify.

   7. Scroll down to Database options, select your new parameter group, and click Continue.

   8. Click Apply immediately or choose a maintenance window, then click Modify DB instance.

      Changing parameters will cause an outage. Wait for the database instance to reboot before continuing.

   9. Verify that the settings are live in your database.

2. Create a user for the source Postgres connector and assign permissions

   1. Create <pg connector username>:

      psql source -c "CREATE USER <pg connector username> PASSWORD '<password>'"
      

      You can use an existing user. However, you must ensure that the user has the following permissions.

   2. Grant permissions to create a replication slot:

      psql source -c "GRANT rds_replication TO <pg connector username>"
      

   3. Grant permissions to create a publication:

      psql source -c "GRANT CREATE ON DATABASE <database name> TO <pg connector username>"
      

   4. Assign the user permissions on the source database:

      psql source <<EOF
      GRANT USAGE ON SCHEMA "public" TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

      If the tables you are syncing are not in the public schema, grant the user permissions for each schema you are syncing:

      psql source <<EOF
      GRANT USAGE ON SCHEMA <schema> TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA <schema> GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

   5. On each table you want to sync, make <pg connector username> the owner:

      psql source -c 'ALTER TABLE  OWNER TO <pg connector username>;'
      

      You can skip this step if the replicating user is already the owner of the tables.

3. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

1. Tune the Write Ahead Log (WAL) on the Postgres source database

   psql source <<EOF
   ALTER SYSTEM SET wal_level='logical';
   ALTER SYSTEM SET max_wal_senders=10;
   ALTER SYSTEM SET wal_sender_timeout=0;
   EOF
   

   • GUC “wal_level” as “logical”
   • GUC “max_wal_senders” as 10
   • GUC “wal_sender_timeout” as 0

   This will require a restart of the Postgres source database.

2. Create a user for the connector and assign permissions

   1. Create <pg connector username>:

      psql source -c "CREATE USER <pg connector username> PASSWORD '<password>'"
      

      You can use an existing user. However, you must ensure that the user has the following permissions.

   2. Grant permissions to create a replication slot:

      psql source -c "ALTER ROLE <pg connector username> REPLICATION"
      

   3. Grant permissions to create a publication:

      psql source -c "GRANT CREATE ON DATABASE <database name> TO <pg connector username>"
      

   4. Assign the user permissions on the source database:

      psql source <<EOF
      GRANT USAGE ON SCHEMA "public" TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

      If the tables you are syncing are not in the public schema, grant the user permissions for each schema you are syncing:

      psql source <<EOF
      GRANT USAGE ON SCHEMA <schema> TO <pg connector username>;
      GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <pg connector username>;
      ALTER DEFAULT PRIVILEGES IN SCHEMA <schema> GRANT SELECT ON TABLES TO <pg connector username>;
      EOF
      

   5. On each table you want to sync, make <pg connector username> the owner:

      psql source -c 'ALTER TABLE  OWNER TO <pg connector username>;'
      

      You can skip this step if the replicating user is already the owner of the tables.

3. Enable replication DELETE andUPDATE operations

   Replica identity assists data replication by identifying the rows being modified. Your options are that each table and hypertable in the source database should either have:

• A primary key: data replication defaults to the primary key of the table being replicated. Nothing to do.

• A viable unique index: each table has a unique, non-partial, non-deferrable index that includes only columns marked as NOT NULL. If a UNIQUE index does not exist, create one to assist the migration. You can delete if after migration.

  For each table, set REPLICA IDENTITY to the viable unique index:

  psql -X -d source -c 'ALTER TABLE  REPLICA IDENTITY USING INDEX <_index_name>'
  

• No primary key or viable unique index: use brute force.

  For each table, set REPLICA IDENTITY to FULL:

  psql -X -d source -c 'ALTER TABLE {table_name} REPLICA IDENTITY FULL'
  

  For each UPDATE or DELETE statement, Postgres reads the whole table to find all matching rows. This results in significantly slower replication. If you are expecting a large number of UPDATE or DELETE operations on the table, best practice is to not use FULL.

Synchronize data to your Tiger Cloud service

To sync data from your Postgres database to your Tiger Cloud service using Tiger Cloud Console:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console, select the service to sync live data to.

2. Connect the source database and the target service

   

   1. Click Connectors > PostgreSQL.
   2. Set the name for the new connector by clicking the pencil icon.
   3. Check the boxes for Set wal_level to logical and Update your credentials, then click Continue.
   4. Enter your database credentials or a Postgres connection string, then click Connect to database. This is the connection string for <pg connector username>. Tiger Cloud Console connects to the source database and retrieves the schema information.

3. Optimize the data to synchronize in hypertables

   

   1. In the Select table dropdown, select the tables to sync.

   2. Click Select tables + .

      Tiger Cloud Console checks the table schema and, if possible, suggests the column to use as the time dimension in a hypertable.

   3. Click Create Connector.

      Tiger Cloud Console starts source Postgres connector between the source database and the target service and displays the progress.

4. Monitor synchronization

   

   1. To view the amount of data replicated, click Connectors. The diagram in Connector data flow gives you an overview of the connectors you have created, their status, and how much data has been replicated.

   2. To review the syncing progress for each table, click Connectors > Source connectors, then select the name of your connector in the table.

5. Manage the connector

   

   1. To edit the connector, click Connectors > Source connectors, then select the name of your connector in the table. You can rename the connector, delete or add new tables for syncing.

   2. To pause a connector, click Connectors > Source connectors, then open the three-dot menu on the right and select Pause.

   3. To delete a connector, click Connectors > Source connectors, then open the three-dot menu on the right and select Delete. You must pause the connector before deleting it.

And that is it, you are using the source Postgres connector to synchronize all the data, or specific tables, from a Postgres database instance to your Tiger Cloud service, in real time.

===== PAGE: https://docs.tigerdata.com/_partials/_2-step-aggregation/ =====

This group of functions uses the two-step aggregation pattern.

Rather than calculating the final result in one step, you first create an intermediate aggregate by using the aggregate function.

Then, use any of the accessors on the intermediate aggregate to calculate a final result. You can also roll up multiple intermediate aggregates with the rollup functions.

The two-step aggregation pattern has several advantages:

1. More efficient because multiple accessors can reuse the same aggregate
2. Easier to reason about performance, because aggregation is separate from final computation
3. Easier to understand when calculations can be rolled up into larger intervals, especially in window functions and continuous aggregates
4. Can perform retrospective analysis even when underlying data is dropped, because the intermediate aggregate stores extra information not available in the final result

To learn more, see the blog post on two-step aggregates.

===== PAGE: https://docs.tigerdata.com/_partials/_timescaledb-gucs/ =====

Name	Type	Default	Description
GUC_CAGG_HIGH_WORK_MEM_NAME	INTEGER	GUC_CAGG_HIGH_WORK_MEM_VALUE	The high working memory limit for the continuous aggregate invalidation processing. min: 64, max: MAX_KILOBYTES
GUC_CAGG_LOW_WORK_MEM_NAME	INTEGER	GUC_CAGG_LOW_WORK_MEM_VALUE	The low working memory limit for the continuous aggregate invalidation processing. min: 64, max: MAX_KILOBYTES
auto_sparse_indexes	BOOLEAN	true	The hypertable columns that are used as index keys will have suitable sparse indexes when compressed. Must be set at the moment of chunk compression, e.g. when the compress_chunk() is called.
bgw_log_level	ENUM	WARNING	Log level for the scheduler and workers of the background worker subsystem. Requires configuration reload to change.
cagg_processing_wal_batch_size	INTEGER	10000	Number of entries processed from the WAL at a go. Larger values take more memory but might be more efficient. min: 1000, max: 10000000
compress_truncate_behaviour	ENUM	COMPRESS_TRUNCATE_ONLY	Defines how truncate behaves at the end of compression. 'truncate_only' forces truncation. 'truncate_disabled' deletes rows instead of truncate. 'truncate_or_delete' allows falling back to deletion.
compression_batch_size_limit	INTEGER	1000	Setting this option to a number between 1 and 999 will force compression to limit the size of compressed batches to that amount of uncompressed tuples.Setting this to 0 defaults to the max batch size of 1000. min: 1, max: 1000
compression_orderby_default_function	STRING	"_timescaledb_functions.get_orderby_defaults"	Function to use for calculating default order_by setting for compression
compression_segmentby_default_function	STRING	"_timescaledb_functions.get_segmentby_defaults"	Function to use for calculating default segment_by setting for compression
current_timestamp_mock	STRING	NULL	this is for debugging purposes
debug_allow_cagg_with_deprecated_funcs	BOOLEAN	false	this is for debugging/testing purposes
debug_bgw_scheduler_exit_status	INTEGER	0	this is for debugging purposes min: 0, max: 255
debug_compression_path_info	BOOLEAN	false	this is for debugging/information purposes
debug_have_int128	BOOLEAN	#ifdef HAVE_INT128 true	this is for debugging purposes
debug_require_batch_sorted_merge	ENUM	DRO_Allow	this is for debugging purposes
debug_require_vector_agg	ENUM	DRO_Allow	this is for debugging purposes
debug_require_vector_qual	ENUM	DRO_Allow	this is for debugging purposes, to let us check if the vectorized quals are used or not. EXPLAIN differs after PG15 for custom nodes, and using the test templates is a pain
debug_skip_scan_info	BOOLEAN	false	Print debug info about SkipScan distinct columns
debug_toast_tuple_target	INTEGER	/* bootValue = */ 128	this is for debugging purposes min: /* minValue = */ 1, max: /* maxValue = */ 65535
enable_bool_compression	BOOLEAN	true	Enable bool compression
enable_bulk_decompression	BOOLEAN	true	Increases throughput of decompression, but might increase query memory usage
enable_cagg_reorder_groupby	BOOLEAN	true	Enable group by clause reordering for continuous aggregates
enable_cagg_sort_pushdown	BOOLEAN	true	Enable pushdown of ORDER BY clause for continuous aggregates
enable_cagg_watermark_constify	BOOLEAN	true	Enable constifying cagg watermark for real-time caggs
enable_cagg_window_functions	BOOLEAN	false	Allow window functions in continuous aggregate views
enable_chunk_append	BOOLEAN	true	Enable using chunk append node
enable_chunk_skipping	BOOLEAN	false	Enable using chunk column stats to filter chunks based on column filters
enable_chunkwise_aggregation	BOOLEAN	true	Enable the pushdown of aggregations to the chunk level
enable_columnarscan	BOOLEAN	true	A columnar scan replaces sequence scans for columnar-oriented storage and enables storage-specific optimizations like vectorized filters. Disabling columnar scan will make PostgreSQL fall back to regular sequence scans.
enable_compressed_direct_batch_delete	BOOLEAN	true	Enable direct batch deletion in compressed chunks
enable_compressed_skipscan	BOOLEAN	true	Enable SkipScan for distinct inputs over compressed chunks
enable_compression_indexscan	BOOLEAN	false	Enable indexscan during compression, if matching index is found
enable_compression_ratio_warnings	BOOLEAN	true	Enable warnings for poor compression ratio
enable_compression_wal_markers	BOOLEAN	true	Enable the generation of markers in the WAL stream which mark the start and end of compression operations
enable_compressor_batch_limit	BOOLEAN	false	Enable compressor batch limit for compressors which can go over the allocation limit (1 GB). This feature willlimit those compressors by reducing the size of the batch and thus avoid hitting the limit.
enable_constraint_aware_append	BOOLEAN	true	Enable constraint exclusion at execution time
enable_constraint_exclusion	BOOLEAN	true	Enable planner constraint exclusion
enable_custom_hashagg	BOOLEAN	false	Enable creating custom hash aggregation plans
enable_decompression_sorted_merge	BOOLEAN	true	Enable the merge of compressed batches to preserve the compression order by
enable_delete_after_compression	BOOLEAN	false	Delete all rows after compression instead of truncate
enable_deprecation_warnings	BOOLEAN	true	Enable warnings when using deprecated functionality
enable_direct_compress_copy	BOOLEAN	false	Enable experimental support for direct compression during COPY
enable_direct_compress_copy_client_sorted	BOOLEAN	false	Correct handling of data sorting by the user is required for this option.
enable_direct_compress_copy_sort_batches	BOOLEAN	true	Enable batch sorting during direct compress COPY
enable_dml_decompression	BOOLEAN	true	Enable DML decompression when modifying compressed hypertable
enable_dml_decompression_tuple_filtering	BOOLEAN	true	Recheck tuples during DML decompression to only decompress batches with matching tuples
enable_event_triggers	BOOLEAN	false	Enable event triggers for chunks creation
enable_exclusive_locking_recompression	BOOLEAN	false	Enable getting exclusive lock on chunk during segmentwise recompression
enable_foreign_key_propagation	BOOLEAN	true	Adjust foreign key lookup queries to target whole hypertable
enable_job_execution_logging	BOOLEAN	false	Retain job run status in logging table
enable_merge_on_cagg_refresh	BOOLEAN	false	Enable MERGE statement on cagg refresh
enable_multikey_skipscan	BOOLEAN	true	Enable SkipScan for multiple distinct inputs
enable_now_constify	BOOLEAN	true	Enable constifying now() in query constraints
enable_null_compression	BOOLEAN	true	Enable null compression
enable_optimizations	BOOLEAN	true	Enable TimescaleDB query optimizations
enable_ordered_append	BOOLEAN	true	Enable ordered append optimization for queries that are ordered by the time dimension
enable_parallel_chunk_append	BOOLEAN	true	Enable using parallel aware chunk append node
enable_qual_propagation	BOOLEAN	true	Enable propagation of qualifiers in JOINs
enable_rowlevel_compression_locking	BOOLEAN	false	Use only if you know what you are doing
enable_runtime_exclusion	BOOLEAN	true	Enable runtime chunk exclusion in ChunkAppend node
enable_segmentwise_recompression	BOOLEAN	true	Enable segmentwise recompression
enable_skipscan	BOOLEAN	true	Enable SkipScan for DISTINCT queries
enable_skipscan_for_distinct_aggregates	BOOLEAN	true	Enable SkipScan for DISTINCT aggregates
enable_sparse_index_bloom	BOOLEAN	true	This sparse index speeds up the equality queries on compressed columns, and can be disabled when not desired.
enable_tiered_reads	BOOLEAN	true	Enable reading of tiered data by including a foreign table representing the data in the object storage into the query plan
enable_transparent_decompression	BOOLEAN	true	Enable transparent decompression when querying hypertable
enable_tss_callbacks	BOOLEAN	true	Enable ts_stat_statements callbacks
enable_uuid_compression	BOOLEAN	false	Enable uuid compression
enable_vectorized_aggregation	BOOLEAN	true	Enable vectorized aggregation for compressed data
last_tuned	STRING	NULL	records last time timescaledb-tune ran
last_tuned_version	STRING	NULL	version of timescaledb-tune used to tune
license	STRING	TS_LICENSE_DEFAULT	Determines which features are enabled
materializations_per_refresh_window	INTEGER	10	The maximal number of individual refreshes per cagg refresh. If more refreshes need to be performed, they are merged into a larger single refresh. min: 0, max: INT_MAX
max_cached_chunks_per_hypertable	INTEGER	1024	Maximum number of chunks stored in the cache min: 0, max: 65536
max_open_chunks_per_insert	INTEGER	1024	Maximum number of open chunk tables per insert min: 0, max: PG_INT16_MAX
max_tuples_decompressed_per_dml_transaction	INTEGER	100000	If the number of tuples exceeds this value, an error will be thrown and transaction rolled back. Setting this to 0 sets this value to unlimited number of tuples decompressed. min: 0, max: 2147483647
restoring	BOOLEAN	false	In restoring mode all timescaledb internal hooks are disabled. This mode is required for restoring logical dumps of databases with timescaledb.
shutdown_bgw_scheduler	BOOLEAN	false	this is for debugging purposes
skip_scan_run_cost_multiplier	REAL	1.0	Default is 1.0 i.e. regularly estimated SkipScan run cost, 0.0 will make SkipScan to have run cost = 0 min: 0.0, max: 1.0
telemetry_level	ENUM	TELEMETRY_DEFAULT	Level used to determine which telemetry to send

Version: 2.22.1

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_run_live_migration_timescaledb/ =====

2. Pull the live-migration docker image to you migration machine

   sudo docker pull timescale/live-migration:latest
   

   To list the available commands, run:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest --help
   

   To see the available flags for each command, run --help for that command. For example:

   sudo docker run --rm -it -e PGCOPYDB_SOURCE_PGURI=source  timescale/live-migration:latest migrate --help
   

3. Create a snapshot image of your source database in your Tiger Cloud service

   This process checks that you have tuned your source database and target service correctly for replication, then creates a snapshot of your data on the migration machine:

   docker run --rm -it --name live-migration-snapshot \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest snapshot
   

   Live-migration supplies information about updates you need to make to the source database and target service. For example:

   2024-03-25T12:40:40.884 WARNING: The following tables in the Source DB have neither a primary key nor a REPLICA IDENTITY (FULL/INDEX)
   2024-03-25T12:40:40.884 WARNING: UPDATE and DELETE statements on these tables will not be replicated to the Target DB
   2024-03-25T12:40:40.884 WARNING:        - public.metrics
   

   If you have warnings, stop live-migration, make the suggested changes and start again.

4. Synchronize data between your source database and your Tiger Cloud service

   This command migrates data from the snapshot to your Tiger Cloud service, then streams transactions from the source to the target.

   docker run --rm -it --name live-migration-migrate \
       -e PGCOPYDB_SOURCE_PGURI=source \
       -e PGCOPYDB_TARGET_PGURI=target \
       --pid=host \
       -v ~/live-migration:/opt/timescale/ts_cdc \
       timescale/live-migration:latest migrate
   

   If the source Postgres version is 17 or later, you need to pass additional flag -e PGVERSION=17 to the migrate command.

   During this process, you see the migration process:

   Live-replay will complete in 1 minute 38.631 seconds (source_wal_rate: 106.0B/s, target_replay_rate: 589.0KiB/s, replay_lag: 56MiB)
   

   If migrate stops add --resume to start from where it left off.

   Once the data in your target Tiger Cloud service has almost caught up with the source database, you see the following message:

   Target has caught up with source (source_wal_rate: 751.0B/s, target_replay_rate: 0B/s, replay_lag: 7KiB)
       To stop replication, hit 'c' and then ENTER
   

   Wait until replay_lag is down to a few kilobytes before you move to the next step. Otherwise, data replication may not have finished.

5. Start app downtime

   1. Stop your app writing to the source database, then let the the remaining transactions finish to fully sync with the target. You can use tools like the pg_top CLI or pg_stat_activity to view the current transaction on the source database.

   2. Stop Live-migration.

      hit 'c' and then ENTER
      

      Live-migration continues the remaining work. This includes copying TimescaleDB metadata, sequences, and run policies. When the migration completes, you see the following message:

      Migration successfully completed
      

===== PAGE: https://docs.tigerdata.com/_partials/_caggs-types/ =====

There are three main ways to make aggregation easier: materialized views, continuous aggregates, and real-time aggregates.

Materialized views are a standard Postgres function. They are used to cache the result of a complex query so that you can reuse it later on. Materialized views do not update regularly, although you can manually refresh them as required.

Continuous aggregates are a TimescaleDB-only feature. They work in a similar way to a materialized view, but they are updated automatically in the background, as new data is added to your database. Continuous aggregates are updated continuously and incrementally, which means they are less resource intensive to maintain than materialized views. Continuous aggregates are based on hypertables, and you can query them in the same way as you do your other tables.

Real-time aggregates are a TimescaleDB-only feature. They are the same as continuous aggregates, but they add the most recent raw data to the previously aggregated data to provide accurate and up-to-date results, without needing to aggregate data as it is being written.

===== PAGE: https://docs.tigerdata.com/_partials/_devops-rest-api-get-started/ =====

Tiger REST API is a comprehensive RESTful API you use to manage Tiger Cloud resources including VPCs, services, and read replicas.

This page shows you how to set up secure authentication for the Tiger REST API and create your first service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Data account.

• Install curl.

Configure secure authentication

Tiger REST API uses HTTP Basic Authentication with access keys and secret keys. All API requests must include proper authentication headers.

1. Set up API credentials

   1. In Tiger Cloud Console copy your project ID and store it securely using an environment variable:

   export TIGERDATA_PROJECT_ID="your-project-id"
   

   1. In Tiger Cloud Console create your client credentials and store them securely using environment variables:

      export TIGERDATA_ACCESS_KEY="Public key"
      export TIGERDATA_SECRET_KEY="Secret key"
      

2. Configure the API endpoint

   Set the base URL in your environment:

   export API_BASE_URL="https://console.cloud.timescale.com/public/api/v1"
   

3. Test your authenticated connection to Tiger REST API by listing the services in the current Tiger Cloud project

   curl -X GET "${API_BASE_URL}/projects/${TIGERDATA_PROJECT_ID}/services" \
     -u "${TIGERDATA_ACCESS_KEY}:${TIGERDATA_SECRET_KEY}" \
     -H "Content-Type: application/json"
   

   This call returns something like:

   • No services:

     []%
     

   • One or more services:

     [{"service_id":"tgrservice","project_id":"tgrproject","name":"tiger-eon",
     "region_code":"us-east-1","service_type":"TIMESCALEDB",
     "created":"2025-10-20T12:21:28.216172Z","paused":false,"status":"READY",
     "resources":[{"id":"104977","spec":{"cpu_millis":500,"memory_gbs":2,"volume_type":""}}],
     "metadata":{"environment":"DEV"},
     "endpoint":{"host":"tgrservice.tgrproject.tsdb.cloud.timescale.com","port":11111}}]
     

Create your first Tiger Cloud service

Create a new service using the Tiger REST API:

1. Create a service using the POST endpoint

   curl -X POST "${API_BASE_URL}/projects/${TIGERDATA_PROJECT_ID}/services" \
     -u "${TIGERDATA_ACCESS_KEY}:${TIGERDATA_SECRET_KEY}" \
     -H "Content-Type: application/json" \
     -d '{
        "name": "my-first-service",
        "addons": ["time-series"],
        "region_code": "us-east-1",
        "replica_count": 1,
        "cpu_millis": "1000",
        "memory_gbs": "4"
        }'
   

   Tiger Cloud creates a Development environment for you. That is, no delete protection, high-availability, spooling or read replication. You see something like:

    {"service_id":"tgrservice","project_id":"tgrproject","name":"my-first-service",
    "region_code":"us-east-1","service_type":"TIMESCALEDB",
    "created":"2025-10-20T22:29:33.052075713Z","paused":false,"status":"QUEUED",
    "resources":[{"id":"105120","spec":{"cpu_millis":1000,"memory_gbs":4,"volume_type":""}}],
    "metadata":{"environment":"PROD"},
    "endpoint":{"host":"tgrservice.tgrproject.tsdb.cloud.timescale.com","port":00001},
    "initial_password":"notTellingYou",
    "ha_replicas":{"sync_replica_count":0,"replica_count":1}}
   

2. Save service_id from the response to a variable:

   # Extract service_id from the JSON response
   export SERVICE_ID="service_id-from-response"
   

3. Check the configuration for the service

  curl -X GET "${API_BASE_URL}/projects/${TIGERDATA_PROJECT_ID}/services/${SERVICE_ID}" \
    -u "${TIGERDATA_ACCESS_KEY}:${TIGERDATA_SECRET_KEY}" \
    -H "Content-Type: application/json"

You see something like:

  {"service_id":"tgrservice","project_id":"tgrproject","name":"my-first-service",
  "region_code":"us-east-1","service_type":"TIMESCALEDB",
  "created":"2025-10-20T22:29:33.052075Z","paused":false,"status":"READY",
  "resources":[{"id":"105120","spec":{"cpu_millis":1000,"memory_gbs":4,"volume_type":""}}],
  "metadata":{"environment":"DEV"},
  "endpoint":{"host":"tgrservice.tgrproject.tsdb.cloud.timescale.com","port":11111},
  "ha_replicas":{"sync_replica_count":0,"replica_count":1}}

And that is it, you are ready to use the Tiger REST API to manage your services in Tiger Cloud.

Security best practices

Follow these security guidelines when working with the Tiger REST API:

• Credential management

  • Store API credentials as environment variables, not in code
  • Use credential rotation policies for production environments
  • Never commit credentials to version control systems

• Network security

  • Use HTTPS endpoints exclusively for API communication
  • Implement proper certificate validation in your HTTP clients

• Data protection

  • Use secure storage for service connection strings and passwords
  • Implement proper backup and recovery procedures for created services
  • Follow data residency requirements for your region

===== PAGE: https://docs.tigerdata.com/_partials/_dimensions_info/ =====

Dimension info

To create a _timescaledb_internal.dimension_info instance, you call add_dimension to an existing hypertable.

Samples

Hypertables must always have a primary range dimension, followed by an arbitrary number of additional dimensions that can be either range or hash, Typically this is just one hash. For example:

SELECT add_dimension('conditions', by_range('time'));
SELECT add_dimension('conditions', by_hash('location', 2));

For incompatible data types such as jsonb, you can specify a function to the partition_func argument of the dimension build to extract a compatible data type. Look in the example section below.

Custom partitioning

By default, TimescaleDB calls Postgres's internal hash function for the given type. You use a custom partitioning function for value types that do not have a native Postgres hash function.

You can specify a custom partitioning function for both range and hash partitioning. A partitioning function should take a anyelement argument as the only parameter and return a positive integer hash value. This hash value is not a partition identifier, but rather the inserted value's position in the dimension's key space, which is then divided across the partitions.

by_range()

Create a by-range dimension builder. You can partition by_range on it's own.

Samples

• Partition on time using CREATE TABLE

  The simplest usage is to partition on a time column:

  CREATE TABLE conditions (
     time        TIMESTAMPTZ       NOT NULL,
     location    TEXT              NOT NULL,
     device      TEXT              NOT NULL,
     temperature DOUBLE PRECISION  NULL,
     humidity    DOUBLE PRECISION  NULL
  ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='time'
  );
  

  If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

  This is the default partition, you do not need to add it explicitly.

• Extract time from a non-time column using create_hypertable

  If you have a table with a non-time column containing the time, such as a JSON column, add a partition function to extract the time:

  CREATE TABLE my_table (
     metric_id serial not null,
     data jsonb,
  );
  
  CREATE FUNCTION get_time(jsonb) RETURNS timestamptz AS $$
    SELECT ($1->>'time')::timestamptz
  $$ LANGUAGE sql IMMUTABLE;
  
  SELECT create_hypertable('my_table', by_range('data', '1 day', 'get_time'));
  

Arguments

Name	Type	Default	Required	Description
column_name	NAME	-	✔	Name of column to partition on.
partition_func	REGPROC	-	✖	The function to use for calculating the partition of a value.
partition_interval	ANYELEMENT	-	✖	Interval to partition column on.

If the column to be partitioned is a:

• TIMESTAMP, TIMESTAMPTZ, or DATE: specify partition_interval either as an INTERVAL type or an integer value in microseconds.

• Another integer type: specify partition_interval as an integer that reflects the column's underlying semantics. For example, if this column is in UNIX time, specify partition_interval in milliseconds.

The partition type and default value depending on column type is:

Column Type	Partition Type	Default value
TIMESTAMP WITHOUT TIMEZONE	INTERVAL/INTEGER	1 week
TIMESTAMP WITH TIMEZONE	INTERVAL/INTEGER	1 week
DATE	INTERVAL/INTEGER	1 week
SMALLINT	SMALLINT	10000
INT	INT	100000
BIGINT	BIGINT	1000000

by_hash()

The main purpose of hash partitioning is to enable parallelization across multiple disks within the same time interval. Every distinct item in hash partitioning is hashed to one of N buckets. By default, TimescaleDB uses flexible range intervals to manage chunk sizes.

Parallelizing disk I/O

You use Parallel I/O in the following scenarios:

• Two or more concurrent queries should be able to read from different disks in parallel.
• A single query should be able to use query parallelization to read from multiple disks in parallel.

For the following options:

• RAID: use a RAID setup across multiple physical disks, and expose a single logical disk to the hypertable. That is, using a single tablespace.

  Best practice is to use RAID when possible, as you do not need to manually manage tablespaces in the database.

• Multiple tablespaces: for each physical disk, add a separate tablespace to the database. TimescaleDB allows you to add multiple tablespaces to a single hypertable. However, although under the hood, a hypertable's chunks are spread across the tablespaces associated with that hypertable.

  When using multiple tablespaces, a best practice is to also add a second hash-partitioned dimension to your hypertable and to have at least one hash partition per disk. While a single time dimension would also work, it would mean that the first chunk is written to one tablespace, the second to another, and so on, and thus would parallelize only if a query's time range exceeds a single chunk.

When adding a hash partitioned dimension, set the number of partitions to a multiple of number of disks. For example, the number of partitions P=N*Pd where N is the number of disks and Pd is the number of partitions per disk. This enables you to add more disks later and move partitions to the new disk from other disks.

TimescaleDB does not benefit from a very large number of hash partitions, such as the number of unique items you expect in partition field. A very large number of hash partitions leads both to poorer per-partition load balancing (the mapping of items to partitions using hashing), as well as much increased planning latency for some types of queries.

Samples

CREATE TABLE conditions (
   "time"      TIMESTAMPTZ       NOT NULL,
   location    TEXT              NOT NULL,
   device      TEXT              NOT NULL,
   temperature DOUBLE PRECISION  NULL,
   humidity    DOUBLE PRECISION  NULL
) WITH (
   tsdb.hypertable,
   tsdb.partition_column='time',
   tsdb.chunk_interval='1 day'
);

SELECT add_dimension('conditions', by_hash('location', 2));

Arguments

Name	Type	Default	Required	Description
column_name	NAME	-	✔	Name of column to partition on.
partition_func	REGPROC	-	✖	The function to use to calcule the partition of a value.
number_partitions	ANYELEMENT	-	✔	Number of hash partitions to use for partitioning_column. Must be greater than 0.

Returns

by_range and by-hash return an opaque _timescaledb_internal.dimension_info instance, holding the dimension information used by this function.

===== PAGE: https://docs.tigerdata.com/_partials/_selfhosted_production_alert/ =====

The following instructions are for development and testing installations. For a production environment, we strongly recommend that you implement the following, many of which you can achieve using Postgres tooling:

• Incremental backup and database snapshots, with efficient point-in-time recovery.
• High availability replication, ideally with nodes across multiple availability zones.
• Automatic failure detection with fast restarts, for both non-replicated and replicated deployments.
• Asynchronous replicas for scaling reads when needed.
• Connection poolers for scaling client connections.
• Zero-down-time minor version and extension upgrades.
• Forking workflows for major version upgrades and other feature testing.
• Monitoring and observability.

Deploying for production? With a Tiger Cloud service we tune your database for performance and handle scalability, high availability, backups, and management, so you can relax.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-redhat-x-platform/ =====

1. Update your local repository list

   sudo yum update
   

2. Install TimescaleDB

   To avoid errors, do not install TimescaleDB Apache 2 Edition and TimescaleDB Community Edition at the same time.

   sudo yum install timescaledb-2-postgresql-17 postgresql17
   

   On Red Hat Enterprise Linux 8 and later, disable the built-in Postgres module:

   sudo dnf -qy module disable postgresql

3. Initialize the Postgres instance

```bash
sudo /usr/pgsql-17/bin/postgresql-17-setup initdb
```

1. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune --pg-config=/usr/pgsql-17/bin/pg_config
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

2. Enable and start Postgres

   sudo systemctl enable postgresql-17
   sudo systemctl start postgresql-17
   

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are now in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_since_2_2_0/ =====

Since TimescaleDB v2.2.0

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dual_write_6a_through_c/ =====

Dump the data from your source database on a per-table basis into CSV format, and restore those CSVs into the target database using the timescaledb-parallel-copy tool.

6a. Determine the time range of data to be copied

Determine the window of data that to be copied from the source database to the target. Depending on the volume of data in the source table, it may be sensible to split the source table into multiple chunks of data to move independently. In the following steps, this time range is called <start> and <end>.

Usually the time column is of type timestamp with time zone, so the values of <start> and <end> must be something like 2023-08-01T00:00:00Z. If the time column is not a timestamp with time zone then the values of <start> and <end> must be the correct type for the column.

If you intend to copy all historic data from the source table, then the value of <start> can be '-infinity', and the <end> value is the value of the completion point T that you determined.

6b. Remove overlapping data in the target

The dual-write process may have already written data into the target database in the time range that you want to move. In this case, the dual-written data must be removed. This can be achieved with a DELETE statement, as follows:

psql target -c "DELETE FROM <hypertable> WHERE time >= <start> AND time < <end>);"

The BETWEEN operator is inclusive of both the start and end ranges, so it is not recommended to use it.

===== PAGE: https://docs.tigerdata.com/_partials/_psql-installation-homebrew/ =====

Installing psql using Homebrew

1. Install psql:

   brew install libpq
   

2. Update your path to include the psql tool.

   brew link --force libpq
   

   On Intel chips, the symbolic link is added to /usr/local/bin. On Apple Silicon, the symbolic link is added to /opt/homebrew/bin.

===== PAGE: https://docs.tigerdata.com/_partials/_early_access_2_17_1/ =====

Early access: TimescaleDB v2.17.1

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dump_postgresql/ =====

Prepare to migrate

1. Take the applications that connect to the source database offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss.

2. Set your connection strings

   These variables hold the connection information for the source database and target Tiger Cloud service:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

   You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Align the extensions on the source and target

1. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Migrate the roles from TimescaleDB to your Tiger Cloud service

Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service:

1. Dump the roles from your source database

   Export your role-based security hierarchy. <db_name> has the same value as <db_name> in source. I know, it confuses me as well.

   pg_dumpall -d "source" \
     -l <db_name>
     --quote-all-identifiers \
     --roles-only \
     --file=roles.sql
   

   If you only use the default postgres role, this step is not necessary.

2. Remove roles with superuser access

   Tiger Cloud service do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e '/CREATE ROLE "tsdbadmin";/d' \
   -e '/ALTER ROLE "tsdbadmin"/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

3. Dump the source database schema and data

   The pg_dump flags remove superuser access and tablespaces from your data. When you run pgdump, check the run time, a long-running pg_dump can cause issues.

   pg_dump -d "source" \
   --format=plain \
   --quote-all-identifiers \
   --no-tablespaces \
   --no-owner \
   --no-privileges \
   --file=dump.sql
   

   To dramatically reduce the time taken to dump the source database, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

Upload your data to the target Tiger Cloud service

psql target -v ON_ERROR_STOP=1 --echo-errors \
-f roles.sql \
-f dump.sql

Validate your Tiger Cloud service and restart your app

1. Update the table statistics.

   psql target -c "ANALYZE;"
   

2. Verify the data in the target Tiger Cloud service.

   Check that your data is correct, and returns the results that you expect,

3. Enable any Tiger Cloud features you want to use.

   Migration from Postgres moves the data only. Now manually enable Tiger Cloud features like hypertables, hypercore or data retention while your database is offline.

4. Reconfigure your app to use the target database, then restart it.

===== PAGE: https://docs.tigerdata.com/_partials/_hypercore-conversion-overview/ =====

When you convert chunks from the rowstore to the columnstore, multiple records are grouped into a single row. The columns of this row hold an array-like structure that stores all the data. For example, data in the following rowstore chunk:

Timestamp	Device ID	Device Type	CPU	Disk IO
12:00:01	A	SSD	70.11	13.4
12:00:01	B	HDD	69.70	20.5
12:00:02	A	SSD	70.12	13.2
12:00:02	B	HDD	69.69	23.4
12:00:03	A	SSD	70.14	13.0
12:00:03	B	HDD	69.70	25.2

Is converted and compressed into arrays in a row in the columnstore:

Timestamp	Device ID	Device Type	CPU	Disk IO
[12:00:01, 12:00:01, 12:00:02, 12:00:02, 12:00:03, 12:00:03]	[A, B, A, B, A, B]	[SSD, HDD, SSD, HDD, SSD, HDD]	[70.11, 69.70, 70.12, 69.69, 70.14, 69.70]	[13.4, 20.5, 13.2, 23.4, 13.0, 25.2]

Because a single row takes up less disk space, you can reduce your chunk size by up to 98%, and can also speed up your queries. This saves on storage costs, and keeps your queries operating at lightning speed.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_migration_cleanup/ =====

To clean up resources associated with live migration, use the following command:

docker run --rm -it --name live-migration-clean \
    -e PGCOPYDB_SOURCE_PGURI=source \
    -e PGCOPYDB_TARGET_PGURI=target \
    --pid=host \
    -v ~/live-migration:/opt/timescale/ts_cdc \
    timescale/live-migration:latest clean --prune

The --prune flag is used to delete temporary files in the ~/live-migration directory that were needed for the migration process. It's important to note that executing the clean command means you cannot resume the interrupted live migration.

===== PAGE: https://docs.tigerdata.com/_partials/_devops-cli-get-started/ =====

Tiger CLI is a command-line interface that you use to manage Tiger Cloud resources including VPCs, services, read replicas, and related infrastructure. Tiger CLI calls Tiger REST API to communicate with Tiger Cloud.

This page shows you how to install and set up secure authentication for Tiger CLI, then create your first service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Data account.

Install and configure Tiger CLI

1. Install Tiger CLI

   Use the terminal to install the CLI:

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   brew install --cask timescale/tap/tiger-cli
   

   curl -fsSL https://cli.tigerdata.com | sh
   

2. Set up API credentials

   1. Log Tiger CLI into your Tiger Data account:

      tiger auth login
      

      Tiger CLI opens Console in your browser. Log in, then click Authorize.

      You can have a maximum of 10 active client credentials. If you get an error, open credentials and delete an unused credential.

   2. Select a Tiger Cloud project:

      Auth URL is: https://console.cloud.timescale.com/oauth/authorize?client_id=lotsOfURLstuff
      Opening browser for authentication...
      Select a project:
      
      > 1. Tiger Project (tgrproject)
      2. YourCompany (Company wide project) (cpnproject)
      3. YourCompany Department (dptproject)
      
      Use ↑/↓ arrows or number keys to navigate, enter to select, q to quit
      

      If only one project is associated with your account, this step is not shown.

      Where possible, Tiger CLI stores your authentication information in the system keychain/credential manager. If that fails, the credentials are stored in ~/.config/tiger/credentials with restricted file permissions (600). By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml.

3. Test your authenticated connection to Tiger Cloud by listing services

   tiger service list
   

   This call returns something like:

   • No services:

     🏜️  No services found! Your project is looking a bit empty.
     🚀 Ready to get started? Create your first service with: tiger service create
     

   • One or more services:

     ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐
     │ SERVICE ID │        NAME         │ STATUS │    TYPE     │    REGION    │     CREATED      │
     ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤
     │ tgrservice │ tiger-agent-service │ READY  │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │
     └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘
     

Create your first Tiger Cloud service

Create a new Tiger Cloud service using Tiger CLI:

1. Submit a service creation request

   By default, Tiger CLI creates a service for you that matches your pricing plan:

   • Free plan: shared CPU/memory and the time-series and ai capabilities
   • Paid plan: 0.5 CPU and 2 GB memory with the time-series capability

   tiger service create
   

   Tiger Cloud creates a Development environment for you. That is, no delete protection, high-availability, spooling or read replication. You see something like:

    🚀 Creating service 'db-11111' (auto-generated name)...
    ✅ Service creation request accepted!
    📋 Service ID: tgrservice
    🔐 Password saved to system keyring for automatic authentication
    🎯 Set service 'tgrservice' as default service.
    ⏳ Waiting for service to be ready (wait timeout: 30m0s)...
    🎉 Service is ready and running!
   🔌 Run 'tiger db connect' to connect to your new service
   ┌───────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
   │     PROPERTY      │                                              VALUE                                               │
   ├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ Service ID        │ tgrservice                                                                                       │
   │ Name              │ db-11111                                                                                         │
   │ Status            │ READY                                                                                            │
   │ Type              │ TIMESCALEDB                                                                                      │
   │ Region            │ us-east-1                                                                                        │
   │ CPU               │ 0.5 cores (500m)                                                                                 │
   │ Memory            │ 2 GB                                                                                             │
   │ Direct Endpoint   │ tgrservice.tgrproject.tsdb.cloud.timescale.com:39004                                             │
   │ Created           │ 2025-10-20 20:33:46 UTC                                                                          │
   │ Connection String │ postgresql://tsdbadmin@tgrservice.tgrproject.tsdb.cloud.timescale.com:0007/tsdb?sslmode=require │
   │ Console URL       │ https://console.cloud.timescale.com/dashboard/services/tgrservice                                │
   └───────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
   

   This service is set as default by the CLI.

2. Check the CLI configuration

   tiger config show
   

   You see something like:

   api_url:     https://console.cloud.timescale.com/public/api/v1
   console_url: https://console.cloud.timescale.com
   gateway_url: https://console.cloud.timescale.com/api
   docs_mcp:       true
   docs_mcp_url:   https://mcp.tigerdata.com/docs
   project_id:  tgrproject
   service_id:  tgrservice
   output:      table
   analytics:   true
   password_storage: keyring
   debug:       false
   config_dir:  /Users/<username>/.config/tiger
   

And that is it, you are ready to use Tiger CLI to manage your services in Tiger Cloud.

Commands

You can use the following commands with Tiger CLI. For more information on each command, use the -h flag. For example: tiger auth login -h

Command	Subcommand	Description
auth		Manage authentication and credentials for your Tiger Data account
	login	Create an authenticated connection to your Tiger Data account
	logout	Remove the credentials used to create authenticated connections to Tiger Cloud
	status	Show your current authentication status and project ID
version		Show information about the currently installed version of Tiger CLI
config		Manage your Tiger CLI configuration
	show	Show the current configuration
	set <key> <value>	Set a specific value in your configuration. For example, tiger config set debug true
	unset <key>	Clear the value of a configuration parameter. For example, tiger config unset debug
	reset	Reset the configuration to the defaults. This also logs you out from the current Tiger Cloud project
service		Manage the Tiger Cloud services in this project
	create	Create a new service in this project. Possible flags are: --name: service name (auto-generated if not provided) --addons: addons to enable (time-series, ai, or none for PostgreSQL-only) --region: region code where the service will be deployed --cpu-memory: CPU/memory allocation combination --replicas: number of high-availability replicas --no-wait: don't wait for the operation to complete --wait-timeout: wait timeout duration (for example, 30m, 1h30m, 90s) --no-set-default: don't set this service as the default service --with-password: include password in output --output, -o: output format (json, yaml, table) Possible cpu-memory combinations are: shared/shared 0.5 CPU/2 GB 1 CPU/4 GB 2 CPU/8 GB 4 CPU/16 GB 8 CPU/32 GB 16 CPU/64 GB 32 CPU/128 GB
	delete <service-id>	Delete a service from this project. This operation is irreversible and requires confirmation by typing the service ID
	fork <service-id>	Fork an existing service to create a new independent copy. Key features are: Timing options: --now, --last-snapshot, --to-timestamp Resource configuration: --cpu-memory Naming: --name <name>. Defaults to {source-service-name}-fork Wait behavior: --no-wait, --wait-timeout Default service: --no-set-default
	get <service-id> (aliases: describe, show)	Show detailed information about a specific service in this project
	list	List all the services in this project
	update-password <service-id>	Update the master password for a service
db		Database operations and management
	connect <service-id>	Connect to a service
	connection-string <service-id>	Retrieve the connection string for a service
	save-password <service-id>	Save the password for a service
	test-connection <service-id>	Test the connectivity to a service
mcp		Manage the Tiger Model Context Protocol Server for AI Assistant integration
	install [client]	Install and configure Tiger Model Context Protocol Server for a specific client (claude-code, cursor, windsurf, or other). If no client is specified, you'll be prompted to select one interactively
	start	Start the Tiger Model Context Protocol Server. This is the same as tiger mcp start stdio
	start stdio	Start the Tiger Model Context Protocol Server with stdio transport (default)
	start http	Start the Tiger Model Context Protocol Server with HTTP transport. Includes flags: --port (default: 8080), --host (default: localhost)

Global flags

You can use the following global flags with Tiger CLI:

Flag	Default	Description
--analytics	true	Set to false to disable usage analytics
--color	true	Set to false to disable colored output
--config-dir string	.config/tiger	Set the directory that holds config.yaml
--debug	No debugging	Enable debug logging
--help	-	Print help about the current command. For example, tiger service --help
--password-storage string	keyring	Set the password storage method. Options are keyring, pgpass, or none
--service-id string	-	Set the Tiger Cloud service to manage
--skip-update-check	-	Do not check if a new version of Tiger CLI is available

Configuration parameters

By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml. The name of these variables matches the flags you use to update them. However, you can override them using the following environmental variables:

• Configuration parameters

  • TIGER_CONFIG_DIR: path to configuration directory (default: ~/.config/tiger)
  • TIGER_API_URL: Tiger REST API base endpoint (default: https://console.cloud.timescale.com/public/api/v1)
  • TIGER_CONSOLE_URL: URL to Tiger Cloud Console (default: https://console.cloud.timescale.com)
  • TIGER_GATEWAY_URL: URL to the Tiger Cloud Console gateway (default: https://console.cloud.timescale.com/api)
  • TIGER_DOCS_MCP: enable/disable docs MCP proxy (default: true)
  • TIGER_DOCS_MCP_URL: URL to the Tiger MCP Server for Tiger Data docs (default: https://mcp.tigerdata.com/docs)
  • TIGER_SERVICE_ID: ID for the service updated when you call CLI commands
  • TIGER_ANALYTICS: enable or disable analytics (default: true)
  • TIGER_PASSWORD_STORAGE: password storage method (keyring, pgpass, or none)
  • TIGER_DEBUG: enable/disable debug logging (default: false)
  • TIGER_COLOR: set to false to disable colored output (default: true)

• Authentication parameters

  To authenticate without using the interactive login, either:

  • Set the following parameters with your client credentials, then login:

    TIGER_PUBLIC_KEY=<public_key> TIGER_SECRET_KEY=<secret_key> TIGER_PROJECT_ID=<project_id>\
    tiger auth login
    

  • Add your client credentials to the login command:

    tiger auth login --public-key=<public_key> --secret-key=<secret-key> --project-id=<project_id>
    

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_self_postgres_plan_migration_path/ =====

Best practice is to always use the latest version of TimescaleDB. Subscribe to our releases on GitHub or use Tiger Cloud and always run the latest update without any hassle.

Check the following support matrix against the versions of TimescaleDB and Postgres that you are running currently and the versions you want to update to, then choose your upgrade path.

For example, to upgrade from TimescaleDB 2.13 on Postgres 13 to TimescaleDB 2.18.2 you need to:

1. Upgrade TimescaleDB to 2.15
2. Upgrade Postgres to 14, 15 or 16.
3. Upgrade TimescaleDB to 2.18.2.

You may need to upgrade to the latest Postgres version before you upgrade TimescaleDB. Also, if you use TimescaleDB Toolkit, ensure the timescaledb_toolkit extension is >= v1.6.0 before you upgrade TimescaleDB extension.

TimescaleDB version	Postgres 17	Postgres 16	Postgres 15	Postgres 14	Postgres 13	Postgres 12	Postgres 11	Postgres 10
2.22.x	✅	✅	✅	❌	❌	❌	❌	❌
2.21.x	✅	✅	✅	❌	❌	❌	❌	❌
2.20.x	✅	✅	✅	❌	❌	❌	❌	❌
2.17 - 2.19	✅	✅	✅	✅	❌	❌	❌	❌
2.16.x	❌	✅	✅	✅	❌	❌	❌	❌
2.13 - 2.15	❌	✅	✅	✅	✅	❌	❌	❌
2.12.x	❌	❌	✅	✅	✅	❌	❌	❌
2.10.x	❌	❌	✅	✅	✅	✅	❌	❌
2.5 - 2.9	❌	❌	❌	✅	✅	✅	❌	❌
2.4	❌	❌	❌	❌	✅	✅	❌	❌
2.1 - 2.3	❌	❌	❌	❌	✅	✅	✅	❌
2.0	❌	❌	❌	❌	❌	✅	✅	❌
1.7	❌	❌	❌	❌	❌	✅	✅	✅

We recommend not using TimescaleDB with Postgres 17.1, 16.5, 15.9, 14.14, 13.17, 12.21. These minor versions introduced a breaking binary interface change that, once identified, was reverted in subsequent minor Postgres versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with Postgres 17.2, 16.6, etc and higher. Users of Tiger Cloud and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_dump_timescaledb/ =====

Prepare to migrate

1. Take the applications that connect to the source database offline

   The duration of the migration is proportional to the amount of data stored in your database. By disconnection your app from your database you avoid and possible data loss.

2. Set your connection strings

   These variables hold the connection information for the source database and target Tiger Cloud service:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   export TARGET="postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

   You find the connection information for your Tiger Cloud service in the configuration file you downloaded when you created the service.

Align the version of TimescaleDB on the source and target

1. Ensure that the source and target databases are running the same version of TimescaleDB.

   1. Check the version of TimescaleDB running on your Tiger Cloud service:

      psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"
      

   2. Update the TimescaleDB extension in your source database to match the target service:

      If the TimescaleDB extension is the same version on the source database and target service, you do not need to do this.

      psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"
      

      For more information and guidance, see Upgrade TimescaleDB.

2. Ensure that the Tiger Cloud service is running the Postgres extensions used in your source database.

   1. Check the extensions on the source database:

      psql source  -c "SELECT * FROM pg_extension;"
      

   2. For each extension, enable it on your target Tiger Cloud service:

      psql target  -c "CREATE EXTENSION IF NOT EXISTS <extension name> CASCADE;"
      

Migrate the roles from TimescaleDB to your Tiger Cloud service

Roles manage database access permissions. To migrate your role-based security hierarchy to your Tiger Cloud service:

1. Dump the roles from your source database

   Export your role-based security hierarchy. <db_name> has the same value as <db_name> in source. I know, it confuses me as well.

   pg_dumpall -d "source" \
     -l <db_name>
     --quote-all-identifiers \
     --roles-only \
     --file=roles.sql
   

   If you only use the default postgres role, this step is not necessary.

2. Remove roles with superuser access

   Tiger Cloud service do not support roles with superuser access. Run the following script to remove statements, permissions and clauses that require superuser permissions from roles.sql:

   sed -i -E \
   -e '/CREATE ROLE "postgres";/d' \
   -e '/ALTER ROLE "postgres"/d' \
   -e '/CREATE ROLE "tsdbadmin";/d' \
   -e '/ALTER ROLE "tsdbadmin"/d' \
   -e 's/(NO)*SUPERUSER//g' \
   -e 's/(NO)*REPLICATION//g' \
   -e 's/(NO)*BYPASSRLS//g' \
   -e 's/GRANTED BY "[^"]*"//g' \
   roles.sql
   

3. Dump the source database schema and data

   The pg_dump flags remove superuser access and tablespaces from your data. When you run pgdump, check the run time, a long-running pg_dump can cause issues.

   pg_dump -d "source" \
   --format=plain \
   --quote-all-identifiers \
   --no-tablespaces \
   --no-owner \
   --no-privileges \
   --file=dump.sql
   

   To dramatically reduce the time taken to dump the source database, using multiple connections. For more information, see dumping with concurrency and restoring with concurrency.

Upload your data to the target Tiger Cloud service

This command uses the timescaledb_pre_restore and timescaledb_post_restore functions to put your database in the correct state.

psql target -v ON_ERROR_STOP=1 --echo-errors \
-f roles.sql \
-c "SELECT timescaledb_pre_restore();" \
-f dump.sql \
-c "SELECT timescaledb_post_restore();"

Validate your Tiger Cloud service and restart your app

1. Update the table statistics.

   psql target -c "ANALYZE;"
   

2. Verify the data in the target Tiger Cloud service.

   Check that your data is correct, and returns the results that you expect,

3. Enable any Tiger Cloud features you want to use.

   Migration from Postgres moves the data only. Now manually enable Tiger Cloud features like hypertables, hypercore or data retention while your database is offline.

4. Reconfigure your app to use the target database, then restart it.

===== PAGE: https://docs.tigerdata.com/_partials/_early_access/ =====

Early access

===== PAGE: https://docs.tigerdata.com/_partials/_add-data-twelvedata-crypto/ =====

Load financial data

This tutorial uses real-time cryptocurrency data, also known as tick data, from Twelve Data. To ingest data into the tables that you created, you need to download the dataset, then upload the data to your Tiger Cloud service.

1. Unzip crypto_sample.zip to a <local folder>.

   This test dataset contains second-by-second trade data for the most-traded crypto-assets and a regular table of asset symbols and company names.

   To import up to 100GB of data directly from your current Postgres-based database, migrate with downtime using native Postgres tooling. To seamlessly import 100GB-10TB+ of data, use the live migration tooling supplied by Tiger Data. To add data from non-Postgres data sources, see Import and ingest data.

2. In Terminal, navigate to <local folder> and connect to your service.

   psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>"
   

   The connection information for a service is available in the file you downloaded when you created it.

3. At the psql prompt, use the COPY command to transfer data into your Tiger Cloud service. If the .csv files aren't in your current directory, specify the file paths in these commands:

   \COPY crypto_ticks FROM 'tutorial_sample_tick.csv' CSV HEADER;
   

   \COPY crypto_assets FROM 'tutorial_sample_assets.csv' CSV HEADER;
   

   Because there are millions of rows of data, the COPY process could take a few minutes depending on your internet connection and local client resources.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-fedora/ =====

1. Install the latest Postgres packages

   sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/F-$(rpm -E %{fedora})-x86_64/pgdg-fedora-repo-latest.noarch.rpm
   

2. Add the TimescaleDB repository

   sudo tee /etc/yum.repos.d/timescale_timescaledb.repo <<EOL
   [timescale_timescaledb]
   name=timescale_timescaledb
   baseurl=https://packagecloud.io/timescale/timescaledb/el/9/\$basearch
   repo_gpgcheck=1
   gpgcheck=0
   enabled=1
   gpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey
   sslverify=1
   sslcacert=/etc/pki/tls/certs/ca-bundle.crt
   metadata_expire=300
   EOL
   

3. Update your local repository list

   sudo yum update
   

4. Install TimescaleDB

   To avoid errors, do not install TimescaleDB Apache 2 Edition and TimescaleDB Community Edition at the same time.

   sudo yum install timescaledb-2-postgresql-17 postgresql17
   

   On Red Hat Enterprise Linux 8 and later, disable the built-in Postgres module:

   sudo dnf -qy module disable postgresql

5. Initialize the Postgres instance

```bash
sudo /usr/pgsql-17/bin/postgresql-17-setup initdb
```

1. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune --pg-config=/usr/pgsql-17/bin/pg_config
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

2. Enable and start Postgres

   sudo systemctl enable postgresql-17
   sudo systemctl start postgresql-17
   

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are now in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_add-data-blockchain/ =====

Load financial data

The dataset contains around 1.5 million Bitcoin transactions, the trades for five days. It includes information about each transaction, along with the value in satoshi. It also states if a trade is a coinbase transaction, and the reward a coin miner receives for mining the coin.

To ingest data into the tables that you created, you need to download the dataset and copy the data to your database.

1. Download the bitcoin_sample.zip file. The file contains a .csv file that contains Bitcoin transactions for the past five days. Download:

   bitcoin_sample.zip

2. In a new terminal window, run this command to unzip the .csv files:

   unzip bitcoin_sample.zip
   

3. In Terminal, navigate to the folder where you unzipped the Bitcoin transactions, then connect to your service using psql.

4. At the psql prompt, use the COPY command to transfer data into your Tiger Cloud service. If the .csv files aren't in your current directory, specify the file paths in these commands:

   \COPY transactions FROM 'tutorial_bitcoin_sample.csv' CSV HEADER;
   

   Because there is over a million rows of data, the COPY process could take a few minutes depending on your internet connection and local client resources.

===== PAGE: https://docs.tigerdata.com/_partials/_hypercore-intro/ =====

Hypercore is a hybrid row-columnar storage engine in TimescaleDB. It is designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage, delivering the best of both worlds:

Hypercore solves the key challenges in real-time analytics:

• High ingest throughput
• Low-latency ingestion
• Fast query performance
• Efficient handling of data updates and late-arriving data
• Streamlined data management

Hypercore’s hybrid approach combines the benefits of row-oriented and column-oriented formats:

• Fast ingest with rowstore: new data is initially written to the rowstore, which is optimized for high-speed inserts and updates. This process ensures that real-time applications easily handle rapid streams of incoming data. Mutability—upserts, updates, and deletes happen seamlessly.

• Efficient analytics with columnstore: as the data cools and becomes more suited for analytics, it is automatically converted to the columnstore. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space.

• Faster queries on compressed data in columnstore: in the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. Combined with chunk skipping, this helps you save on storage costs and keeps your queries operating at lightning speed.

• Fast modification of compressed data in columnstore: just use SQL to add or modify data in the columnstore. TimescaleDB is optimized for superfast INSERT and UPSERT performance.

• Full mutability with transactional semantics: regardless of where data is stored, hypercore provides full ACID support. Like in a vanilla Postgres database, inserts and updates to the rowstore and columnstore are always consistent, and available to queries as soon as they are completed.

For an in-depth explanation of how hypertables and hypercore work, see the Data model.

===== PAGE: https://docs.tigerdata.com/_partials/_experimental-schema-upgrade/ =====

When you upgrade the timescaledb extension, the experimental schema is removed by default. To use experimental features after an upgrade, you need to add the experimental schema again.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_import_setup_connection_strings_parquet/ =====

This variable holds the connection information for the target Tiger Cloud service.

In the terminal on the source machine, set the following:

export TARGET=postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require

See where to find your connection details.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_pg_dump_minimal_downtime/ =====

For minimal downtime, run the migration commands from a machine with a low-latency, high-throughput link to the source and target databases. If you are using an AWS EC2 instance to run the migration commands, use one in the same region as your target Tiger Cloud service.

===== PAGE: https://docs.tigerdata.com/_partials/_migrate_live_migrate_faq_all/ =====

ERROR: relation "xxx.yy" does not exist

This may happen when a relation is removed after executing the snapshot command. A relation can be a table, index, view, or materialized view. When you see you this error:

• Do not perform any explicit DDL operation on the source database during the course of migration.

• If you are migrating from self-hosted TimescaleDB or MST, disable the chunk retention policy on your source database until you have finished migration.

FATAL: remaining connection slots are reserved for non-replication superuser connections

This may happen when the number of connections exhaust max_connections defined in your target Tiger Cloud service. By default, live-migration needs around ~6 connections on the source and ~12 connections on the target.

Migration seems to be stuck with “x GB copied to Target DB (Source DB is y GB)”

When you are migrating a lot of data involved in aggregation, or there are many materialized views taking time to complete the materialization, this may be due to REFRESH MATERIALIZED VIEWS happening at the end of initial data migration.

To resolve this issue:

1. See what is happening on the target Tiger Cloud service:

   psql target -c "select * from pg_stat_activity where application_name ilike '%pgcopydb%';"
   

2. When you run the migrate, add the following flags to exclude specific materialized views being materialized:

   --skip-table-data <matview1> <matview2>”
   

3. When migrate has finished, manually refresh the materialized views you excluded.

Restart migration from scratch after a non-resumable failure

If the migration halts due to a failure, such as a misconfiguration of the source or target database, you may need to restart the migration from scratch. In such cases, you can reuse the original target Tiger Cloud service created for the migration by utilizing the --drop-if-exists flag with the migrate command.

This flag ensures that the existing target objects created by the previous migration are dropped, allowing the migration to proceed without trouble.

Note: This flag also requires you to manually recreate the TimescaleDB extension on the target.

Here’s an example command sequence to restart the migration:

psql target -c "DROP EXTENSION timescaledb CASCADE"

psql target -c 'CREATE EXTENSION timescaledb VERSION "<desired version>"'

docker run --rm -it --name live-migration-migrate \
    -e PGCOPYDB_SOURCE_PGURI=source \
    -e PGCOPYDB_TARGET_PGURI=target \
    --pid=host \
    -v ~/live-migration:/opt/timescale/ts_cdc \
    timescale/live-migration:latest migrate --drop-if-exists

This approach provides a clean slate for the migration process while reusing the existing target instance.

Inactive or lagging replication slots

If you encounter an “Inactive or lagging replication slots” warning on your cloud provider console after using live-migration, it might be due to lingering replication slots created by the live-migration tool on your source database.

To clean up resources associated with live migration, use the following command:

docker run --rm -it --name live-migration-clean \
    -e PGCOPYDB_SOURCE_PGURI=source \
    -e PGCOPYDB_TARGET_PGURI=target \
    --pid=host \
    -v ~/live-migration:/opt/timescale/ts_cdc \
    timescale/live-migration:latest clean --prune

The --prune flag is used to delete temporary files in the ~/live-migration directory that were needed for the migration process. It's important to note that executing the clean command means you cannot resume the interrupted live migration.

Role passwords

Because of issues dumping passwords from various managed service providers, Live-migration migrates roles without passwords. You have to migrate passwords manually.

Table privileges

Live-migration does not migrate table privileges. After completing Live-migration:

1. Grant all roles to tsdbadmin.

   psql -d source -t -A -c "SELECT FORMAT('GRANT %I TO tsdbadmin;', rolname) FROM
   pg_catalog.pg_roles WHERE rolname not like 'pg_%' AND rolname != 'tsdbadmin'
   AND NOT rolsuper" | psql -d target -f -
   

2. On your migration machine, edit /tmp/grants.psql to match table privileges on your source database.

   pg_dump --schema-only --quote-all-identifiers
   --exclude-schema=_timescaledb_catalog --format=plain --dbname "source" | grep
   "(ALTER.*OWNER.*|GRANT|REVOKE)"  > /tmp/grants.psql
   

3. Run grants.psql on your target Tiger Cloud service.

   psql -d target -f /tmp/grants.psql
   

Postgres to Tiger Cloud: “live-replay not keeping up with source load”

1. Go to Tiger Cloud Console -> Monitoring -> Insights tab and find the query which takes significant time
2. If the query is either UPDATE/DELETE, make sure the columns used on the WHERE clause have necessary indexes.
3. If the query is either UPDATE/DELETE on the tables which are converted as hypertables, make sure the REPLIDA IDENTITY(defaults to primary key) on the source is compatible with the target primary key. If not, create an UNIQUE index source database by including the hypertable partition column and make it as a REPLICA IDENTITY. Also, create the same UNIQUE index on target.

ERROR: out of memory (or) Failed on request of size xxx in memory context "yyy" on a Tiger Cloud service

This error occurs when the Out of Memory (OOM) guard is triggered due to memory allocations exceeding safe limits. It typically happens when multiple concurrent connections to the TimescaleDB instance are performing memory-intensive operations. For example, during live migrations, this error can occur when large indexes are being created simultaneously.

The live-migration tool includes a retry mechanism to handle such errors. However, frequent OOM crashes may significantly delay the migration process.

One of the following can be used to avoid the OOM errors:

1. Upgrade to Higher Memory Spec Instances: To mitigate memory constraints, consider using a TimescaleDB instance with higher specifications, such as an instance with 8 CPUs and 32 GB RAM (or more). Higher memory capacity can handle larger workloads and reduce the likelihood of OOM errors.

2. Reduce Concurrency: If upgrading your instance is not feasible, you can reduce the concurrency of the index migration process using the --index-jobs=<value> flag in the migration command. By default, the value of --index-jobs matches the GUC max_parallel_workers. Lowering this value reduces the memory usage during migration but may increase the total migration time.

By taking these steps, you can prevent OOM errors and ensure a smoother migration experience with TimescaleDB.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-debian-based/ =====

1. Install the latest Postgres packages

   sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget
   

2. Run the Postgres package setup script

   sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
   

   If you want to do some development on Postgres, add the libraries:

   sudo apt install postgresql-server-dev-17
   

3. Add the TimescaleDB package

   echo "deb https://packagecloud.io/timescale/timescaledb/debian/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
   

   echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
   

4. Install the TimescaleDB GPG key

   wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg
   

   For Ubuntu 21.10 and earlier use the following command:

   wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add -

5. Update your local repository list

   sudo apt update
   

6. Install TimescaleDB

   sudo apt install timescaledb-2-postgresql-17 postgresql-client-17
   

   To install a specific TimescaleDB release, set the version. For example:

   sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'

   Older versions of TimescaleDB may not support all the OS versions listed on this page.

7. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune
   

   By default, this script is included with the timescaledb-tools package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see Configuration. If you have an issue, run sudo apt install timescaledb-tools.

8. Restart Postgres

   sudo systemctl restart postgresql
   

9. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

10. Set the password for postgres

    \password postgres
    

    When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_use-case-setup-blockchain-dataset/ =====

Ingest data into a Tiger Cloud service

This tutorial uses a dataset that contains Bitcoin blockchain data for the past five days, in a hypertable named transactions.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Optimize time-series data using hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed. You can also connect to your service using psql.

2. Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data:

   CREATE TABLE transactions (
      time TIMESTAMPTZ NOT NULL,
      block_id INT,
      hash TEXT,
      size INT,
      weight INT,
      is_coinbase BOOLEAN,
      output_total BIGINT,
      output_total_usd DOUBLE PRECISION,
      fee BIGINT,
      fee_usd DOUBLE PRECISION,
      details JSONB
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time',
      tsdb.segmentby='block_id',
      tsdb.orderby='time DESC'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

3. Create an index on the hash column to make queries for individual transactions faster:

   CREATE INDEX hash_idx ON public.transactions USING HASH (hash);
   

4. Create an index on the block_id column to make block-level queries faster:

When you create a hypertable, it is partitioned on the time column. TimescaleDB automatically creates an index on the time column. However, you'll often filter your time-series data on other columns as well. You use indexes to improve query performance.

```sql
CREATE INDEX block_idx ON public.transactions (block_id);
```

1. Create a unique index on the time and hash columns to make sure you don't accidentally insert duplicate records:

   CREATE UNIQUE INDEX time_hash_idx ON public.transactions (time, hash);
   

Load financial data

The dataset contains around 1.5 million Bitcoin transactions, the trades for five days. It includes information about each transaction, along with the value in satoshi. It also states if a trade is a coinbase transaction, and the reward a coin miner receives for mining the coin.

To ingest data into the tables that you created, you need to download the dataset and copy the data to your database.

1. Download the bitcoin_sample.zip file. The file contains a .csv file that contains Bitcoin transactions for the past five days. Download:

   bitcoin_sample.zip

2. In a new terminal window, run this command to unzip the .csv files:

   unzip bitcoin_sample.zip
   

3. In Terminal, navigate to the folder where you unzipped the Bitcoin transactions, then connect to your service using psql.

4. At the psql prompt, use the COPY command to transfer data into your Tiger Cloud service. If the .csv files aren't in your current directory, specify the file paths in these commands:

   \COPY transactions FROM 'tutorial_bitcoin_sample.csv' CSV HEADER;
   

   Because there is over a million rows of data, the COPY process could take a few minutes depending on your internet connection and local client resources.

===== PAGE: https://docs.tigerdata.com/_partials/_import-data-iot/ =====

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Import time-series data into a hypertable

   1. Unzip metrics.csv.gz to a <local folder>.

      This test dataset contains energy consumption data.

      To import up to 100GB of data directly from your current Postgres based database, migrate with downtime using native Postgres tooling. To seamlessly import 100GB-10TB+ of data, use the live migration tooling supplied by Tiger Data. To add data from non-Postgres data sources, see Import and ingest data.

   2. In Terminal, navigate to <local folder> and update the following string with your connection details to connect to your service.

      psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>?sslmode=require"
      

   3. Create an optimized hypertable for your time-series data:

      1. Create a hypertable with hypercore enabled by default for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data.

         In your sql client, run the following command:

         CREATE TABLE "metrics"(
           created timestamp with time zone default now() not null,
           type_id integer                                not null,
           value   double precision                       not null
         ) WITH (
           tsdb.hypertable,
           tsdb.partition_column='created',
           tsdb.segmentby = 'type_id',
           tsdb.orderby = 'created DESC'
         );
         

         If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

   4. Upload the dataset to your service

      \COPY metrics FROM metrics.csv CSV;
      

2. Have a quick look at your data

   You query hypertables in exactly the same way as you would a relational Postgres table. Use one of the following SQL editors to run a query and see the data you uploaded:

   • Data mode: write queries, visualize data, and share your results in Tiger Cloud Console for all your Tiger Cloud services.
   • SQL editor: write, fix, and organize SQL faster and more accurately in Tiger Cloud Console for a Tiger Cloud service.
   • psql: easily run queries on your Tiger Cloud services or self-hosted TimescaleDB deployment from Terminal.

   SELECT time_bucket('1 day', created, 'Europe/Berlin') AS "time",
   round((last(value, created) - first(value, created)) * 100.) / 100. AS value
   FROM metrics
   WHERE type_id = 5
   GROUP BY 1;
   

   On this amount of data, this query on data in the rowstore takes about 3.6 seconds. You see something like:

   Time	value
   2023-05-29 22:00:00+00	23.1
   2023-05-28 22:00:00+00	19.5
   2023-05-30 22:00:00+00	25
   2023-05-31 22:00:00+00	8.1

===== PAGE: https://docs.tigerdata.com/_partials/_toolkit-install-update-debian-base/ =====

Prerequisites

To follow this procedure:

• Install TimescaleDB.
• Add the TimescaleDB repository and the GPG key.

Install TimescaleDB Toolkit

These instructions use the apt package manager.

1. Update your local repository list:

   sudo apt update
   

2. Install TimescaleDB Toolkit:

   sudo apt install timescaledb-toolkit-postgresql-17
   

3. Connect to the database where you want to use Toolkit.

4. Create the Toolkit extension in the database:

   CREATE EXTENSION timescaledb_toolkit;
   

Update TimescaleDB Toolkit

Update Toolkit by installing the latest version and running ALTER EXTENSION.

1. Update your local repository list:

   apt update
   

2. Install the latest version of TimescaleDB Toolkit:

   apt install timescaledb-toolkit-postgresql-17
   

3. Connect to the database where you want to use the new version of Toolkit.

4. Update the Toolkit extension in the database:

   ALTER EXTENSION timescaledb_toolkit UPDATE;
   

   For some Toolkit versions, you might need to disconnect and reconnect active sessions.

===== PAGE: https://docs.tigerdata.com/_partials/_grafana-viz-prereqs/ =====

Before you begin, make sure you have:

• Created a Timescale service.
• Installed a self-managed Grafana account, or signed up for Grafana Cloud.
• Ingested some data to your database. You can use the stock trade data from the Getting Started Guide.

The examples in this section use these variables and Grafana functions:

• $symbol: a variable used to filter results by stock symbols.
• _timeFrom()::timestamptz & _timeTo()::timestamptz: Grafana variables. You change the values of these variables by using the dashboard's date chooser when viewing your graph.
• $bucket_interval: the interval size to pass to the time_bucket function when aggregating data.

===== PAGE: https://docs.tigerdata.com/_partials/_cloud-mst-comparison/ =====

Tiger Cloud is a high-performance developer focused cloud that provides Postgres services enhanced with our blazing fast vector search. You can securely integrate Tiger Cloud with your AWS, GCS or Azure infrastructure. Create a Tiger Cloud service and try for free.

If you need to run TimescaleDB on GCP or Azure, you're in the right place — keep reading.

===== PAGE: https://docs.tigerdata.com/_partials/_plan_upgrade/ =====

• Install the Postgres client tools on your migration machine. This includes psql, and pg_dump.
• Read the release notes for the version of TimescaleDB that you are upgrading to.
• Perform a backup of your database. While TimescaleDB upgrades are performed in-place, upgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster.

===== PAGE: https://docs.tigerdata.com/_partials/_use-case-iot-create-cagg/ =====

1. Monitor energy consumption on a day-to-day basis

   1. Create a continuous aggregate kwh_day_by_day for energy consumption:

      CREATE MATERIALIZED VIEW kwh_day_by_day(time, value)
         with (timescaledb.continuous) as
      SELECT time_bucket('1 day', created, 'Europe/Berlin') AS "time",
             round((last(value, created) - first(value, created)) * 100.) / 100. AS value
      FROM metrics
      WHERE type_id = 5
      GROUP BY 1;
      

   2. Add a refresh policy to keep kwh_day_by_day up-to-date:

      SELECT add_continuous_aggregate_policy('kwh_day_by_day',
         start_offset => NULL,
         end_offset => INTERVAL '1 hour',
         schedule_interval => INTERVAL '1 hour');
      

2. Monitor energy consumption on an hourly basis

   1. Create a continuous aggregate kwh_hour_by_hour for energy consumption:

      CREATE MATERIALIZED VIEW kwh_hour_by_hour(time, value)
        with (timescaledb.continuous) as
      SELECT time_bucket('01:00:00', metrics.created, 'Europe/Berlin') AS "time",
             round((last(value, created) - first(value, created)) * 100.) / 100. AS value
      FROM metrics
      WHERE type_id = 5
      GROUP BY 1;
      

   2. Add a refresh policy to keep the continuous aggregate up-to-date:

   SELECT add_continuous_aggregate_policy('kwh_hour_by_hour',
    start_offset => NULL,
       end_offset => INTERVAL '1 hour',
       schedule_interval => INTERVAL '1 hour');
   

3. Analyze your data

   Now you have made continuous aggregates, it could be a good idea to use them to perform analytics on your data. For example, to see how average energy consumption changes during weekdays over the last year, run the following query:

     WITH per_day AS (
      SELECT
        time,
        value
      FROM kwh_day_by_day
      WHERE "time" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year'
      ORDER BY 1
     ), daily AS (
         SELECT
            to_char(time, 'Dy') as day,
            value
         FROM per_day
     ), percentile AS (
         SELECT
             day,
             approx_percentile(0.50, percentile_agg(value)) as value
         FROM daily
         GROUP BY 1
         ORDER BY 1
     )
     SELECT
         d.day,
         d.ordinal,
         pd.value
     FROM unnest(array['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']) WITH ORDINALITY AS d(day, ordinal)
     LEFT JOIN percentile pd ON lower(pd.day) = lower(d.day);
   

   You see something like:

   day	ordinal	value
   Mon	2	23.08078714975423
   Sun	1	19.511430831944395
   Tue	3	25.003118897837307
   Wed	4	8.09300571759772

===== PAGE: https://docs.tigerdata.com/_partials/_use-case-transport-geolocation/ =====

Set up your data for geospatial queries

To add geospatial analysis to your ride count visualization, you need geospatial data to work out which trips originated where. As TimescaleDB is compatible with all Postgres extensions, use PostGIS to slice data by time and location.

1. Connect to your Tiger Cloud service and add the PostGIS extension:

   CREATE EXTENSION postgis;
   

2. Add geometry columns for pick up and drop off locations:

   ALTER TABLE rides ADD COLUMN pickup_geom geometry(POINT,2163);
   ALTER TABLE rides ADD COLUMN dropoff_geom geometry(POINT,2163);
   

3. Convert the latitude and longitude points into geometry coordinates that work with PostGIS:

   UPDATE rides SET pickup_geom = ST_Transform(ST_SetSRID(ST_MakePoint(pickup_longitude,pickup_latitude),4326),2163),
      dropoff_geom = ST_Transform(ST_SetSRID(ST_MakePoint(dropoff_longitude,dropoff_latitude),4326),2163);
   

   This updates 10,906,860 rows of data on both columns, it takes a while. Coffee is your friend.

Visualize the area where you can make the most money

In this section you visualize a query that returns rides longer than 5 miles for trips taken within 2 km of Times Square. The data includes the distance travelled and is GROUP BY trip_distance and location so that Grafana can plot the data properly.

This enables you to see where a taxi driver is most likely to pick up a passenger who wants a longer ride, and make more money.

1. Create a geolocalization dashboard

   1. In Grafana, create a new dashboard that is connected to your Tiger Cloud service data source with a Geomap visualization.

   2. In the Queries section, select Code, then select the Time series Format.

      

   3. To find rides longer than 5 miles in Manhattan, paste the following query:

      SELECT time_bucket('5m', rides.pickup_datetime) AS time,
             rides.trip_distance AS value,
             rides.pickup_latitude AS latitude,
             rides.pickup_longitude AS longitude
      FROM rides
      WHERE rides.pickup_datetime BETWEEN '2016-01-01T01:41:55.986Z' AND '2016-01-01T07:41:55.986Z' AND
        ST_Distance(pickup_geom,
                    ST_Transform(ST_SetSRID(ST_MakePoint(-73.9851,40.7589),4326),2163)
        ) < 2000
      GROUP BY time,
               rides.trip_distance,
               rides.pickup_latitude,
               rides.pickup_longitude
      ORDER BY time
      LIMIT 500;
      

      You see a world map with a dot on New York.

   4. Zoom into your map to see the visualization clearly.

2. Customize the visualization

   1. In the Geomap options, under Map Layers, click + Add layer and select Heatmap. You now see the areas where a taxi driver is most likely to pick up a passenger who wants a longer ride, and make more money.

      

===== PAGE: https://docs.tigerdata.com/_partials/_old-api-create-hypertable/ =====

If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

===== PAGE: https://docs.tigerdata.com/_partials/_timescale-cloud-regions/ =====

Tiger Cloud services run in the following Amazon Web Services (AWS) regions:

Region	Zone	Location
ap-south-1	Asia Pacific	Mumbai
ap-southeast-1	Asia Pacific	Singapore
ap-southeast-2	Asia Pacific	Sydney
ap-northeast-1	Asia Pacific	Tokyo
ca-central-1	Canada	Central
eu-central-1	Europe	Frankfurt
eu-west-1	Europe	Ireland
eu-west-2	Europe	London
sa-east-1	South America	São Paulo
us-east-1	United States	North Virginia
us-east-2	United States	Ohio
us-west-2	United States	Oregon

===== PAGE: https://docs.tigerdata.com/_partials/_timescale-intro/ =====

Tiger Data extends Postgres for all of your resource-intensive production workloads, so you can build faster, scale further, and stay under budget.

===== PAGE: https://docs.tigerdata.com/_partials/_devops-mcp-commands/ =====

Tiger Model Context Protocol Server exposes the following MCP tools to your AI Assistant:

Command	Parameter	Required	Description
service_list	-	-	Returns a list of the services in the current project.
service_get	-	-	Returns detailed information about a service.
	service_id	✓	The unique identifier of the service (10-character alphanumeric string).
	with_password	-	Set to true to include the password in the response and connection string. WARNING: never do this unless the user explicitly requests the password.
service_create	-	-	Create a new service in Tiger Cloud. WARNING: creates billable resources.
	name	-	Set the human-readable name of up to 128 characters for this service.
	addons	-	Set the array of addons to enable for the service. Options: time-series: enables TimescaleDB ai: enables the AI and vector extensions Set an empty array for Postgres-only.
	region	-	Set the AWS region to deploy this service in.
	cpu_memory	-	CPU and memory allocation combination. Available configurations are: shared/shared 0.5 CPU/2 GB 1 CPU/4 GB 2 CPU/8 GB 4 CPU/16 GB 8 CPU/32 GB 16 CPU/64 GB 32 CPU/128 GB
	replicas	-	Set the number of high-availability replicas for fault tolerance.
	wait	-	Set to true to wait for service to be fully ready before returning.
	timeout_minutes	-	Set the timeout in minutes to wait for service to be ready. Only used when wait=true. Default: 30 minutes
	set_default	-	By default, the new service is the default for following commands in CLI. Set to false to keep the previous service as the default.
	with_password	-	Set to true to include the password for this service in response and connection string. WARNING: never set to true unless user explicitly requests the password.
service_update_password	-	-	Update the password for the tsdbadmin for this service. The password change takes effect immediately and may terminate existing connections.
	service_id	✓	The unique identifier of the service you want to update the password for.
	password	✓	The new password for the tsdbadmin user.
db_execute_query	-	-	Execute a single SQL query against a service. This command returns column metadata, result rows, affected row count, and execution time. Multi-statement queries are not supported. WARNING: can execute destructive SQL including INSERT, UPDATE, DELETE, and DDL commands.
	service_id	✓	The unique identifier of the service. Use tiger_service_list to find service IDs.
	query	✓	The SQL query to execute. Single statement queries are supported.
	parameters	-	Query parameters for parameterized queries. Values are substituted for the $n placeholders in the query.
	timeout_seconds	-	The query timeout in seconds. Default: 30.
	role	-	The service role/username to connect as. Default: tsdbadmin.
	pooled	-	Use connection pooling. This is only available if you have already enabled it for the service. Default: false.

===== PAGE: https://docs.tigerdata.com/_partials/_cloudwatch-data-exporter/ =====

1. In Tiger Cloud Console, open Exporters

2. Click New exporter

3. Select the data type and specify AWS CloudWatch for provider

   

4. Provide your AWS CloudWatch configuration

   • The AWS region must be the same for your Tiger Cloud exporter and AWS CloudWatch Log group.
   • The exporter name appears in Tiger Cloud Console, best practice is to make this name easily understandable.
   • For CloudWatch credentials, either use an existing CloudWatch Log group or create a new one. If you're uncertain, use the default values. For more information, see Working with log groups and log streams.

5. Choose the authentication method to use for the exporter

   

   1. In AWS, navigate to IAM > Identity providers, then click Add provider.

   2. Update the new identity provider with your details:

      Set Provider URL to the region where you are creating your exporter.

      

   3. Click Add provider.

   4. In AWS, navigate to IAM > Roles, then click Create role.

   5. Add your identity provider as a Web identity role and click Next.

      

   6. Set the following permission and trust policies:

      • Permission policy:

        {
          "Version": "2012-10-17",
          "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
                     "logs:PutLogEvents",
                     "logs:CreateLogGroup",
                     "logs:CreateLogStream",
                     "logs:DescribeLogStreams",
                     "logs:DescribeLogGroups",
                     "logs:PutRetentionPolicy",
                     "xray:PutTraceSegments",
                     "xray:PutTelemetryRecords",
                     "xray:GetSamplingRules",
                     "xray:GetSamplingTargets",
                     "xray:GetSamplingStatisticSummaries",
                     "ssm:GetParameters"
                 ],
                 "Resource": "*"
             }
         ]
        }
        

      • Role with a Trust Policy:

        {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "Federated": "arn:aws:iam::12345678910:oidc-provider/irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com"
                  },
                  "Action": "sts:AssumeRoleWithWebIdentity",
                  "Condition": {
                      "StringEquals": {
                          "irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com:aud": "sts.amazonaws.com"
                      }
                  }
              },
              {
                  "Sid": "Statement1",
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::12345678910:role/my-exporter-role"
                  },
                  "Action": "sts:AssumeRole"
              }
          ]
        }
        

   7. Click Add role.

   When you use CloudWatch credentials, you link an Identity and Access Management (IAM) user with access to CloudWatch only with your Tiger Cloud service:

   1. Retrieve the user information from IAM > Users in AWS console.

      If you do not have an AWS user with access restricted to CloudWatch only, create one. For more information, see Creating IAM users (console).

   2. Enter the credentials for the AWS IAM user.

      AWS keys give access to your AWS services. To keep your AWS account secure, restrict users to the minimum required permissions. Always store your keys in a safe location. To avoid this issue, use the IAM role authentication method.

6. Select the AWS Region your CloudWatch services run in, then click Create exporter.

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-srt-candlestick/ =====

SELECT time_bucket('1 day', "time") AS day, symbol, max(price) AS high, first(price, time) AS open, last(price, time) AS close, min(price) AS low FROM stocks_real_time srt GROUP BY day, symbol ORDER BY day DESC, symbol LIMIT 10;

-- Output

day | symbol | high | open | close | low -----------------------+--------+--------------+----------+----------+-------------- 2023-06-07 00:00:00+00 | AAPL | 179.25 | 178.91 | 179.04 | 178.17 2023-06-07 00:00:00+00 | ABNB | 117.99 | 117.4 | 117.9694 | 117 2023-06-07 00:00:00+00 | AMAT | 134.8964 | 133.73 | 134.8964 | 133.13 2023-06-07 00:00:00+00 | AMD | 125.33 | 124.11 | 125.13 | 123.82 2023-06-07 00:00:00+00 | AMZN | 127.45 | 126.22 | 126.69 | 125.81 ...

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-crypto-cagg/ =====

SELECT * FROM assets_candlestick_daily ORDER BY day DESC, symbol LIMIT 10;

-- Output

day | symbol | high | open | close | low -----------------------+--------+----------+--------+----------+---------- 2025-01-30 00:00:00+00 | ADA/USD | 0.9708 | 0.9396 | 0.9607 | 0.9365 2025-01-30 00:00:00+00 | ATOM/USD | 6.114 | 5.825 | 6.063 | 5.776 2025-01-30 00:00:00+00 | AVAX/USD | 34.1 | 32.8 | 33.95 | 32.44 2025-01-30 00:00:00+00 | BNB/USD | 679.3 | 668.12 | 677.81 | 666.08 2025-01-30 00:00:00+00 | BTC/USD | 105595.65 | 103735.84 | 105157.21 | 103298.84 2025-01-30 00:00:00+00 | CRO/USD | 0.13233 | 0.12869 | 0.13138 | 0.12805 2025-01-30 00:00:00+00 | DAI/USD | 1 | 1 | 0.9999 | 0.99989998 2025-01-30 00:00:00+00 | DOGE/USD | 0.33359 | 0.32392 | 0.33172 | 0.32231 2025-01-30 00:00:00+00 | DOT/USD | 6.01 | 5.779 | 6.004 | 5.732 2025-01-30 00:00:00+00 | ETH/USD | 3228.9 | 3113.36 | 3219.25 | 3092.92 (10 rows)

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-cagg-tesla/ =====

SELECT * FROM stock_candlestick_daily WHERE symbol='TSLA' ORDER BY day DESC LIMIT 10;

-- Output

day | symbol | high | open | close | low -----------------------+--------+----------+----------+----------+---------- 2023-07-31 00:00:00+00 | TSLA | 269 | 266.42 | 266.995 | 263.8422 2023-07-28 00:00:00+00 | TSLA | 267.4 | 259.32 | 266.8 | 258.06 2023-07-27 00:00:00+00 | TSLA | 269.98 | 268.3 | 256.8 | 241.5539 2023-07-26 00:00:00+00 | TSLA | 271.5168 | 265.48 | 265.3283 | 258.0418 2023-07-25 00:00:00+00 | TSLA | 270.22 | 267.5099 | 264.55 | 257.21 2023-07-20 00:00:00+00 | TSLA | 267.58 | 267.34 | 260.6 | 247.4588 2023-07-14 00:00:00+00 | TSLA | 285.27 | 277.29 | 281.7 | 264.7567 2023-07-13 00:00:00+00 | TSLA | 290.0683 | 274.07 | 277.4509 | 270.6127 2023-07-12 00:00:00+00 | TSLA | 277.68 | 271.26 | 272.94 | 258.0418 2023-07-11 00:00:00+00 | TSLA | 271.44 | 270.83 | 269.8303 | 266.3885 (10 rows)

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-srt-4-days/ =====

SELECT * FROM stocks_real_time srt LIMIT 10;

-- Output

time | symbol | price | day_volume -----------------------+--------+----------+------------ 2023-07-31 16:32:16+00 | PEP | 187.755 | 1618189 2023-07-31 16:32:16+00 | TSLA | 268.275 | 51902030 2023-07-31 16:32:16+00 | INTC | 36.035 | 22736715 2023-07-31 16:32:15+00 | CHTR | 402.27 | 626719 2023-07-31 16:32:15+00 | TSLA | 268.2925 | 51899210 2023-07-31 16:32:15+00 | AMD | 113.72 | 29136618 2023-07-31 16:32:15+00 | NVDA | 467.72 | 13951198 2023-07-31 16:32:15+00 | AMD | 113.72 | 29137753 2023-07-31 16:32:15+00 | RTX | 87.74 | 4295687 2023-07-31 16:32:15+00 | RTX | 87.74 | 4295907 (10 rows)

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-srt-bucket-first-last/ =====

SELECT time_bucket('1 hour', time) AS bucket, first(price,time), last(price, time) FROM stocks_real_time srt WHERE time > now() - INTERVAL '4 days' GROUP BY bucket;

-- Output

     bucket         | first  |  last

------------------------+--------+-------- 2023-08-07 08:00:00+00 | 88.75 | 182.87 2023-08-07 09:00:00+00 | 140.85 | 35.16 2023-08-07 10:00:00+00 | 182.89 | 52.58 2023-08-07 11:00:00+00 | 86.69 | 255.15

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-srt-orderby/ =====

SELECT * FROM stocks_real_time srt WHERE symbol='TSLA' ORDER BY time DESC LIMIT 10;

-- Output

time | symbol | price | day_volume -----------------------+--------+----------+------------ 2025-01-30 00:51:00+00 | TSLA | 405.32 | NULL 2025-01-30 00:41:00+00 | TSLA | 406.05 | NULL 2025-01-30 00:39:00+00 | TSLA | 406.25 | NULL 2025-01-30 00:32:00+00 | TSLA | 406.02 | NULL 2025-01-30 00:32:00+00 | TSLA | 406.10 | NULL 2025-01-30 00:25:00+00 | TSLA | 405.95 | NULL 2025-01-30 00:24:00+00 | TSLA | 406.04 | NULL 2025-01-30 00:24:00+00 | TSLA | 406.04 | NULL 2025-01-30 00:22:00+00 | TSLA | 406.38 | NULL 2025-01-30 00:21:00+00 | TSLA | 405.77 | NULL (10 rows)

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-cagg/ =====

SELECT * FROM stock_candlestick_daily ORDER BY day DESC, symbol LIMIT 10;

-- Output

day | symbol | high | open | close | low -----------------------+--------+----------+--------+----------+---------- 2023-07-31 00:00:00+00 | AAPL | 196.71 | 195.9 | 196.1099 | 195.2699 2023-07-31 00:00:00+00 | ABBV | 151.25 | 151.25 | 148.03 | 148.02 2023-07-31 00:00:00+00 | ABNB | 154.95 | 153.43 | 152.95 | 151.65 2023-07-31 00:00:00+00 | ABT | 113 | 112.4 | 111.49 | 111.44 2023-07-31 00:00:00+00 | ADBE | 552.87 | 536.74 | 550.835 | 536.74 2023-07-31 00:00:00+00 | AMAT | 153.9786 | 152.5 | 151.84 | 150.52 2023-07-31 00:00:00+00 | AMD | 114.57 | 113.47 | 113.15 | 112.35 2023-07-31 00:00:00+00 | AMGN | 237 | 236.61 | 233.6 | 233.515 2023-07-31 00:00:00+00 | AMT | 191.69 | 189.75 | 190.55 | 188.97 2023-07-31 00:00:00+00 | AMZN | 133.89 | 132.42 | 133.055 | 132.32 (10 rows)

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-srt-aggregation/ =====

SELECT time_bucket('1 day', time) AS bucket, symbol, max(price) AS high, first(price, time) AS open, last(price, time) AS close, min(price) AS low FROM stocks_real_time srt WHERE time > now() - INTERVAL '1 week' GROUP BY bucket, symbol ORDER BY bucket, symbol LIMIT 10;

-- Output

day | symbol | high | open | close | low -----------------------+--------+--------------+----------+----------+-------------- 2023-06-07 00:00:00+00 | AAPL | 179.25 | 178.91 | 179.04 | 178.17 2023-06-07 00:00:00+00 | ABNB | 117.99 | 117.4 | 117.9694 | 117 2023-06-07 00:00:00+00 | AMAT | 134.8964 | 133.73 | 134.8964 | 133.13 2023-06-07 00:00:00+00 | AMD | 125.33 | 124.11 | 125.13 | 123.82 2023-06-07 00:00:00+00 | AMZN | 127.45 | 126.22 | 126.69 | 125.81 ...

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-srt-first-last/ =====

SELECT symbol, first(price,time), last(price, time) FROM stocks_real_time srt WHERE time > now() - INTERVAL '4 days' GROUP BY symbol ORDER BY symbol LIMIT 10;

-- Output

symbol | first | last -------+----------+---------- AAPL | 179.0507 | 179.04 ABNB | 118.83 | 117.9694 AMAT | 133.55 | 134.8964 AMD | 122.6476 | 125.13 AMZN | 126.5599 | 126.69 ...

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-crypto-srt-orderby/ =====

SELECT * FROM crypto_ticks srt WHERE symbol='ETH/USD' ORDER BY time DESC LIMIT 10;

-- Output

time | symbol | price | day_volume -----------------------+--------+----------+------------ 2025-01-30 12:05:09+00 | ETH/USD | 3219.25 | 39425 2025-01-30 12:05:00+00 | ETH/USD | 3219.26 | 39425 2025-01-30 12:04:42+00 | ETH/USD | 3219.26 | 39459 2025-01-30 12:04:33+00 | ETH/USD | 3219.91 | 39458 2025-01-30 12:04:15+00 | ETH/USD | 3219.6 | 39458 2025-01-30 12:04:06+00 | ETH/USD | 3220.68 | 39458 2025-01-30 12:03:57+00 | ETH/USD | 3220.68 | 39483 2025-01-30 12:03:48+00 | ETH/USD | 3220.12 | 39483 2025-01-30 12:03:20+00 | ETH/USD | 3219.79 | 39482 2025-01-30 12:03:11+00 | ETH/USD | 3220.06 | 39472 (10 rows)

===== PAGE: https://docs.tigerdata.com/_queries/getting-started-week-average/ =====

SELECT time_bucket('1 day', time) AS bucket, symbol, avg(price) FROM stocks_real_time srt WHERE time > now() - INTERVAL '1 week' GROUP BY bucket, symbol ORDER BY bucket, symbol LIMIT 10;

-- Output

bucket | symbol | avg -----------------------+--------+-------------------- 2023-06-01 00:00:00+00 | AAPL | 179.3242530284364 2023-06-01 00:00:00+00 | ABNB | 112.05498586371293 2023-06-01 00:00:00+00 | AMAT | 134.41263567849518 2023-06-01 00:00:00+00 | AMD | 119.43332772033834 2023-06-01 00:00:00+00 | AMZN | 122.3446364966392 ...

===== PAGE: https://docs.tigerdata.com/integrations/corporate-data-center/ =====

Integrate your data center with Tiger Cloud

This page explains how to integrate your corporate on-premise infrastructure with Tiger Cloud using AWS Transit Gateway.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details.

• Set up AWS Transit Gateway.

Connect your on-premise infrastructure to your Tiger Cloud services

To connect to Tiger Cloud:

1. Connect your infrastructure to AWS Transit Gateway

   Establish connectivity between your on-premise infrastructure and AWS. See the Centralize network connectivity using AWS Transit Gateway.

2. Create a Peering VPC in Tiger Cloud Console

   1. In Security > VPC, click Create a VPC:

      

   2. Choose your region and IP range, name your VPC, then click Create VPC:

      

      Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your plan in Tiger Cloud Console.

   3. Add a peering connection:

      1. In the VPC Peering column, click Add.
      2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID.

      

      1. Click Add connection.

3. Accept and configure peering connection in your AWS account

   Once your peering connection appears as Processing, you can accept and configure it in AWS:

   1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as Connected in Tiger Cloud Console.

   2. Configure at least the following in your AWS account networking:

      • Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs.
      • Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs.
      • Security groups to allow outbound TCP 5432.

4. Attach a Tiger Cloud service to the Peering VPC In Tiger Cloud Console

   1. Select the service you want to connect to the Peering VPC.
   2. Click Operations > Security > VPC.
   3. Select the VPC, then click Attach VPC.

   You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time.

You have successfully integrated your Microsoft Azure infrastructure with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/cloudwatch/ =====

Integrate Amazon CloudWatch with Tiger Cloud

Amazon CloudWatch is a monitoring and observability service designed to help collect, analyze, and act on data from applications, infrastructure, and services running in AWS and on-premises environments.

You can export telemetry data from your Tiger Cloud services with the time-series and analytics capability enabled to CloudWatch. The available metrics include CPU usage, RAM usage, and storage. This integration is available for Scale and Enterprise pricing tiers.

This pages explains how to export telemetry data from your Tiger Cloud service into CloudWatch by creating a Tiger Cloud data exporter, then attaching it to the service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details.

• Sign up for Amazon CloudWatch.

Create a data exporter

A Tiger Cloud data exporter sends telemetry data from a Tiger Cloud service to a third-party monitoring tool. You create an exporter on the project level, in the same AWS region as your service:

1. In Tiger Cloud Console, open Exporters

2. Click New exporter

3. Select the data type and specify AWS CloudWatch for provider

   

4. Provide your AWS CloudWatch configuration

   • The AWS region must be the same for your Tiger Cloud exporter and AWS CloudWatch Log group.
   • The exporter name appears in Tiger Cloud Console, best practice is to make this name easily understandable.
   • For CloudWatch credentials, either use an existing CloudWatch Log group or create a new one. If you're uncertain, use the default values. For more information, see Working with log groups and log streams.

5. Choose the authentication method to use for the exporter

   

   1. In AWS, navigate to IAM > Identity providers, then click Add provider.

   2. Update the new identity provider with your details:

      Set Provider URL to the region where you are creating your exporter.

      

   3. Click Add provider.

   4. In AWS, navigate to IAM > Roles, then click Create role.

   5. Add your identity provider as a Web identity role and click Next.

      

   6. Set the following permission and trust policies:

      • Permission policy:

        {
          "Version": "2012-10-17",
          "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
                     "logs:PutLogEvents",
                     "logs:CreateLogGroup",
                     "logs:CreateLogStream",
                     "logs:DescribeLogStreams",
                     "logs:DescribeLogGroups",
                     "logs:PutRetentionPolicy",
                     "xray:PutTraceSegments",
                     "xray:PutTelemetryRecords",
                     "xray:GetSamplingRules",
                     "xray:GetSamplingTargets",
                     "xray:GetSamplingStatisticSummaries",
                     "ssm:GetParameters"
                 ],
                 "Resource": "*"
             }
         ]
        }
        

      • Role with a Trust Policy:

        {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "Federated": "arn:aws:iam::12345678910:oidc-provider/irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com"
                  },
                  "Action": "sts:AssumeRoleWithWebIdentity",
                  "Condition": {
                      "StringEquals": {
                          "irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com:aud": "sts.amazonaws.com"
                      }
                  }
              },
              {
                  "Sid": "Statement1",
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::12345678910:role/my-exporter-role"
                  },
                  "Action": "sts:AssumeRole"
              }
          ]
        }
        

   7. Click Add role.

   When you use CloudWatch credentials, you link an Identity and Access Management (IAM) user with access to CloudWatch only with your Tiger Cloud service:

   1. Retrieve the user information from IAM > Users in AWS console.

      If you do not have an AWS user with access restricted to CloudWatch only, create one. For more information, see Creating IAM users (console).

   2. Enter the credentials for the AWS IAM user.

      AWS keys give access to your AWS services. To keep your AWS account secure, restrict users to the minimum required permissions. Always store your keys in a safe location. To avoid this issue, use the IAM role authentication method.

6. Select the AWS Region your CloudWatch services run in, then click Create exporter.

Attach a data exporter to a Tiger Cloud service

To send telemetry data to an external monitoring tool, you attach a data exporter to your Tiger Cloud service. You can attach only one exporter to a service.

To attach an exporter:

1. In Tiger Cloud Console, choose the service
2. Click Operations > Exporters
3. Select the exporter, then click Attach exporter
4. If you are attaching a first Logs data type exporter, restart the service

Monitor Tiger Cloud service metrics

You can now monitor your service metrics. Use the following metrics to check the service is running correctly:

• timescale.cloud.system.cpu.usage.millicores
• timescale.cloud.system.cpu.total.millicores
• timescale.cloud.system.memory.usage.bytes
• timescale.cloud.system.memory.total.bytes
• timescale.cloud.system.disk.usage.bytes
• timescale.cloud.system.disk.total.bytes

Additionally, use the following tags to filter your results.

Tag	Example variable	Description
host	us-east-1.timescale.cloud
project-id
service-id
region	us-east-1	AWS region
role	replica or primary	For service with replicas
node-id		For multi-node services

Edit a data exporter

To update a data exporter:

1. In Tiger Cloud Console, open Exporters
2. Next to the exporter you want to edit, click the menu > Edit
3. Edit the exporter fields and save your changes

You cannot change fields such as the provider or the AWS region.

Delete a data exporter

To remove a data exporter that you no longer need:

1. Disconnect the data exporter from your Tiger Cloud services

   1. In Tiger Cloud Console, choose the service.
   2. Click Operations > Exporters.
   3. Click the trash can icon.
   4. Repeat for every service attached to the exporter you want to remove.

   The data exporter is now unattached from all services. However, it still exists in your project.

2. Delete the exporter on the project level

   1. In Tiger Cloud Console, open Exporters
   2. Next to the exporter you want to edit, click menu > Delete
   3. Confirm that you want to delete the data exporter.

Reference

When you create the IAM OIDC provider, the URL must match the region you create the exporter in. It must be one of the following:

Region	Zone	Location	URL
ap-southeast-1	Asia Pacific	Singapore	irsa-oidc-discovery-prod-ap-southeast-1.s3.ap-southeast-1.amazonaws.com
ap-southeast-2	Asia Pacific	Sydney	irsa-oidc-discovery-prod-ap-southeast-2.s3.ap-southeast-2.amazonaws.com
ap-northeast-1	Asia Pacific	Tokyo	irsa-oidc-discovery-prod-ap-northeast-1.s3.ap-northeast-1.amazonaws.com
ca-central-1	Canada	Central	irsa-oidc-discovery-prod-ca-central-1.s3.ca-central-1.amazonaws.com
eu-central-1	Europe	Frankfurt	irsa-oidc-discovery-prod-eu-central-1.s3.eu-central-1.amazonaws.com
eu-west-1	Europe	Ireland	irsa-oidc-discovery-prod-eu-west-1.s3.eu-west-1.amazonaws.com
eu-west-2	Europe	London	irsa-oidc-discovery-prod-eu-west-2.s3.eu-west-2.amazonaws.com
sa-east-1	South America	São Paulo	irsa-oidc-discovery-prod-sa-east-1.s3.sa-east-1.amazonaws.com
us-east-1	United States	North Virginia	irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com
us-east-2	United States	Ohio	irsa-oidc-discovery-prod-us-east-2.s3.us-east-2.amazonaws.com
us-west-2	United States	Oregon	irsa-oidc-discovery-prod-us-west-2.s3.us-west-2.amazonaws.com

===== PAGE: https://docs.tigerdata.com/integrations/pgadmin/ =====

Integrate pgAdmin with Tiger

pgAdmin is a feature-rich open-source administration and development platform for Postgres. It is available for Chrome, Firefox, Edge, and Safari browsers, or can be installed on Microsoft Windows, Apple macOS, or various Linux flavors.

This page explains how to integrate pgAdmin with your Tiger Cloud service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Download and install pgAdmin.

Connect pgAdmin to your Tiger Cloud service

To connect to Tiger Cloud:

1. Start pgAdmin
2. In the Quick Links section of the Dashboard tab, click Add New Server
3. In Register - Server > General, fill in the Name and Comments fields with the server name and description, respectively
4. Configure the connection
   1. In the Connection tab, configure the connection using your connection details.
   2. If you configured your service to connect using a stricter SSL mode, then in the SSL tab check Use SSL, set SSL mode to the configured mode, and in the CA Certificate field type the location of the SSL root CA certificate to use.
5. Click Save

You have successfully integrated pgAdmin with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/kubernetes/ =====

Integrate Kubernetes with Tiger

Kubernetes is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. You can connect Kubernetes to Tiger Cloud, and deploy TimescaleDB within your Kubernetes clusters.

This guide explains how to connect a Kubernetes cluster to Tiger Cloud, configure persistent storage, and deploy TimescaleDB in your kubernetes cluster.

Prerequisites

To follow the steps on this page:

• Install self-managed Kubernetes or sign up for a Kubernetes Turnkey Cloud Solution.
• Install kubectl for command-line interaction with your cluster.

Integrate TimescaleDB in a Kubernetes cluster

To connect your Kubernetes cluster to your Tiger Cloud service:

1. Create a default namespace for your Tiger Cloud components

   1. Create a namespace:

      kubectl create namespace timescale
      

   2. Set this namespace as the default for your session:

      kubectl config set-context --current --namespace=timescale
      

   For more information, see Kubernetes Namespaces.

2. Create a Kubernetes secret that stores your Tiger Cloud service credentials

   Update the following command with your connection details, then run it:

   kubectl create secret generic timescale-secret \
    --from-literal=PGHOST=<host> \
    --from-literal=PGPORT=<port> \
    --from-literal=PGDATABASE=<dbname> \
    --from-literal=PGUSER=<user> \
    --from-literal=PGPASSWORD=<password>
   

3. Configure network access to Tiger Cloud

   • Managed Kubernetes: outbound connections to external databases like Tiger Cloud work by default. Make sure your cluster’s security group or firewall rules allow outbound traffic to Tiger Cloud IP.

   • Self-hosted Kubernetes: If your cluster is behind a firewall or running on-premise, you may need to allow egress traffic to Tiger Cloud. Test connectivity using your connection details:

     nc -zv <host> <port>
     

     If the connection fails, check your firewall rules.

4. Create a Kubernetes deployment that can access your Tiger Cloud

   Run the following command to apply the deployment:

   kubectl apply -f - <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: timescale-app
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: timescale-app
     template:
       metadata:
         labels:
           app: timescale-app
       spec:
         containers:
         - name: timescale-container
           image: postgres:latest
           envFrom:
             - secretRef:
                 name: timescale-secret
   EOF
   

5. Test the connection

   1. Create and run a pod that uses the connection details you added to timescale-secret in the timescale namespace:

      kubectl run test-pod --image=postgres --restart=Never \
       --env="PGHOST=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGHOST}' | base64 --decode)" \
       --env="PGPORT=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGPORT}' | base64 --decode)" \
       --env="PGDATABASE=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGDATABASE}' | base64 --decode)" \
       --env="PGUSER=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGUSER}' | base64 --decode)" \
       --env="PGPASSWORD=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGPASSWORD}' | base64 --decode)" \
       -- sleep infinity
      

   2. Launch a psql shell in the test-pod you just created:

      kubectl exec -it test-pod -- bash -c "psql -h \$PGHOST -U \$PGUSER -d \$PGDATABASE"
      

   You start a psql session connected to your Tiger Cloud service.

Running TimescaleDB on Kubernetes is similar to running Postgres. This procedure outlines the steps for a non-distributed system.

To connect your Kubernetes cluster to self-hosted TimescaleDB running in the cluster:

1. Create a default namespace for Tiger Data components

   1. Create the Tiger Data namespace:

      kubectl create namespace timescale
      

   2. Set this namespace as the default for your session:

      kubectl config set-context --current --namespace=timescale
      

   For more information, see Kubernetes Namespaces.

2. Set up a persistent volume claim (PVC) storage

   To manually set up a persistent volume and claim for self-hosted Kubernetes, run the following command:

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: timescale-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 10Gi
   EOF
   

3. Deploy TimescaleDB as a StatefulSet

   By default, the TimescaleDB Docker image you are installing on Kubernetes uses the default Postgres database, user and password. To deploy TimescaleDB on Kubernetes, run the following command:

   kubectl apply -f - <<EOF
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: timescaledb
   spec:
     serviceName: timescaledb
     replicas: 1
     selector:
       matchLabels:
         app: timescaledb
     template:
       metadata:
         labels:
           app: timescaledb
       spec:
         containers:
           - name: timescaledb
             image: 'timescale/timescaledb:latest-pg17'
             env:
               - name: POSTGRES_USER
                 value: postgres
               - name: POSTGRES_PASSWORD
                 value: postgres
               - name: POSTGRES_DB
                 value: postgres
               - name: PGDATA
                 value: /var/lib/postgresql/data/pgdata
             ports:
               - containerPort: 5432
             volumeMounts:
               - mountPath: /var/lib/postgresql/data
                 name: timescale-storage
         volumes:
           - name: timescale-storage
             persistentVolumeClaim:
               claimName: timescale-pvc
   EOF
   

4. Allow applications to connect by exposing TimescaleDB within Kubernetes

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: timescaledb
spec:
  selector:
    app: timescaledb
  ports:
    - protocol: TCP
      port: 5432
      targetPort: 5432
  type: ClusterIP
EOF

1. Create a Kubernetes secret to store the database credentials

   kubectl create secret generic timescale-secret \
   --from-literal=PGHOST=timescaledb \
   --from-literal=PGPORT=5432 \
   --from-literal=PGDATABASE=postgres \
   --from-literal=PGUSER=postgres \
   --from-literal=PGPASSWORD=postgres
   

2. Deploy an application that connects to TimescaleDB

   kubectl apply -f - <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: timescale-app
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: timescale-app
     template:
       metadata:
         labels:
           app: timescale-app
       spec:
         containers:
         - name: timescale-container
           image: postgres:latest
           envFrom:
             - secretRef:
                 name: timescale-secret
   EOF
   

3. Test the database connection

   1. Create and run a pod to verify database connectivity using your connection details saved in timescale-secret:

      kubectl run test-pod --image=postgres --restart=Never \
      --env="PGHOST=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGHOST}' | base64 --decode)" \
      --env="PGPORT=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGPORT}' | base64 --decode)" \
      --env="PGDATABASE=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGDATABASE}' | base64 --decode)" \
      --env="PGUSER=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGUSER}' | base64 --decode)" \
      --env="PGPASSWORD=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGPASSWORD}' | base64 --decode)" \
      -- sleep infinity
      

   2. Launch the Postgres interactive shell within the created test-pod:

      kubectl exec -it test-pod -- bash -c "psql -h \$PGHOST -U \$PGUSER -d \$PGDATABASE"
      

   You see the Postgres interactive terminal.

You have successfully integrated Kubernetes with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/prometheus/ =====

Integrate Prometheus with Tiger

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, and a modern alerting approach.

This page shows you how to export your service telemetry to Prometheus:

• For Tiger Cloud, using a dedicated Prometheus exporter in Tiger Cloud Console.
• For self-hosted TimescaleDB, using Postgres Exporter.

Prerequisites

To follow the steps on this page:

• Download and run Prometheus.

• For Tiger Cloud:

  Create a target Tiger Cloud service with the time-series and analytics capability enabled.

• For self-hosted TimescaleDB:

  • Create a target self-hosted TimescaleDB instance. You need your connection details.
  • Install Postgres Exporter. To reduce latency and potential data transfer costs, install Prometheus and Postgres Exporter on a machine in the same AWS region as your Tiger Cloud service.

Export Tiger Cloud service telemetry to Prometheus

To export your data, do the following:

To export metrics from a Tiger Cloud service, you create a dedicated Prometheus exporter in Tiger Cloud Console, attach it to your service, then configure Prometheus to scrape metrics using the exposed URL. The Prometheus exporter exposes the metrics related to the Tiger Cloud service like CPU, memory, and storage. To scrape other metrics, use Postgres Exporter as described for self-hosted TimescaleDB. The Prometheus exporter is available for Scale and Enterprise pricing plans.

1. Create a Prometheus exporter

   1. In Tiger Cloud Console, click Exporters > + New exporter.

   2. Select Metrics for data type and Prometheus for provider.

      

   3. Choose the region for the exporter. Only services in the same project and region can be attached to this exporter.

   4. Name your exporter.

   5. Change the auto-generated Prometheus credentials, if needed. See official documentation on basic authentication in Prometheus.

2. Attach the exporter to a service

   1. Select a service, then click Operations > Exporters.

   2. Select the exporter in the drop-down, then click Attach exporter.

      

   The exporter is now attached to your service. To unattach it, click the trash icon in the exporter list.

   

3. Configure the Prometheus scrape target

   1. Select your service, then click Operations > Exporters and click the information icon next to the exporter. You see the exporter details.

      

   2. Copy the exporter URL.

   3. In your Prometheus installation, update prometheus.yml to point to the exporter URL as a scrape target:

      scrape_configs:
       - job_name: "timescaledb-exporter"
         scheme: https
         static_configs:
           - targets: ["my-exporter-url"]
         basic_auth:
           username: "user"
           password: "pass"
      

      See the Prometheus documentation for details on configuring scrape targets.

      You can now monitor your service metrics. Use the following metrics to check the service is running correctly:

      • timescale.cloud.system.cpu.usage.millicores
      • timescale.cloud.system.cpu.total.millicores
      • timescale.cloud.system.memory.usage.bytes
      • timescale.cloud.system.memory.total.bytes
      • timescale.cloud.system.disk.usage.bytes
      • timescale.cloud.system.disk.total.bytes

      Additionally, use the following tags to filter your results.

      Tag	Example variable	Description
      host	us-east-1.timescale.cloud
      project-id
      service-id
      region	us-east-1	AWS region
      role	replica or primary	For service with replicas

To export metrics from self-hosted TimescaleDB, you import telemetry data about your database to Postgres Exporter, then configure Prometheus to scrape metrics from it. Postgres Exporter exposes metrics that you define, excluding the system metrics.

1. Create a user to access telemetry data about your database

   1. Connect to your database in psql using your connection details.

   2. Create a user named monitoring with a secure password:

      CREATE USER monitoring WITH PASSWORD '<password>';
      

   3. Grant the pg_read_all_stats permission to the monitoring user:

      GRANT pg_read_all_stats to monitoring;
      

2. Import telemetry data about your database to Postgres Exporter

   1. Connect Postgres Exporter to your database:

      Use your connection details to import telemetry data about your database. You connect as the monitoring user:

      • Local installation:

        export DATA_SOURCE_NAME="postgres://<user>:<password>@<host>:<port>/<database>?sslmode=<sslmode>"
        ./postgres_exporter
        

      • Docker:

        docker run -d \
           -e DATA_SOURCE_NAME="postgres://<user>:<password>@<host>:<port>/<database>?sslmode=<sslmode>" \
           -p 9187:9187 \
           prometheuscommunity/postgres-exporter
        

   2. Check the metrics for your database in the Prometheus format:

      • Browser:

        Navigate to http://<exporter-host>:9187/metrics.

      • Command line:

        curl http://<exporter-host>:9187/metrics
        

3. Configure Prometheus to scrape metrics

   1. In your Prometheus installation, update prometheus.yml to point to your Postgres Exporter instance as a scrape target. In the following example, you replace <exporter-host> with the hostname or IP address of the PostgreSQL Exporter.

      global:
        scrape_interval: 15s
      
      scrape_configs:
      - job_name: 'postgresql'
        static_configs:
         - targets: ['<exporter-host>:9187']
      

      If prometheus.yml has not been created during installation, create it manually. If you are using Docker, you can find the IPAddress in Inspect > Networks for the container running Postgres Exporter.

   2. Restart Prometheus.

   3. Check the Prometheus UI at http://<prometheus-host>:9090/targets and http://<prometheus-host>:9090/tsdb-status.

      You see the Postgres Exporter target and the metrics scraped from it.

You can further visualize your data with Grafana. Use the Grafana Postgres dashboard or create a custom dashboard that suits your needs.

===== PAGE: https://docs.tigerdata.com/integrations/psql/ =====

Connect to a Tiger Cloud service with psql

psql is a terminal-based frontend to Postgres that enables you to type in queries interactively, issue them to Postgres, and see the query results.

This page shows you how to use the psql command line tool to interact with your Tiger Cloud service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Check for an existing installation

On many operating systems, psql is installed by default. To use the functionality described in this page, best practice is to use the latest version of psql. To check the version running on your system:

psql --version

wmic
/output:C:\list.txt product get name, version

If you already have the latest version of psql installed, proceed to the Connect to your service section.

Install psql

If there is no existing installation, take the following steps to install psql:

Install using Homebrew. libpqxx is the official C++ client API for Postgres.

1. Install Homebrew, if you don't already have it:

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   

   For more information about Homebrew, including installation instructions, see the Homebrew documentation.

2. Make sure your Homebrew repository is up to date:

   brew doctor
   brew update
   

3. Install psql:

   brew install libpq
   

4. Update your path to include the psql tool:

   brew link --force libpq
   

On Intel chips, the symbolic link is added to /usr/local/bin. On Apple Silicon, the symbolic link is added to /opt/homebrew/bin.

Install using MacPorts. libpqxx is the official C++ client API for Postgres.

1. Install MacPorts by downloading and running the package installer.

2. Make sure MacPorts is up to date:

   sudo port selfupdate
   

3. Install the latest version of libpqxx:

   sudo port install libpqxx
   

4. View the files that were installed by libpqxx:

   port contents libpqxx
   

Install psql on Debian and Ubuntu with the apt package manager.

1. Make sure your apt repository is up to date:

      sudo apt-get update
   

2. Install the postgresql-client package:

   sudo apt-get install postgresql-client
   

psql is installed by default when you install Postgres. This procedure uses the interactive installer provided by Postgres and EnterpriseDB.

1. Download and run the Postgres installer from www.enterprisedb.com.

2. In the Select Components dialog, check Command Line Tools, along with any other components you want to install, and click Next.

3. Complete the installation wizard to install the package.

Connect to your service

To use psql to connect to your service, you need the connection details. See Find your connection details.

Connect to your service with either:

• The parameter flags:

  psql -h <HOSTNAME> -p <PORT> -U <USERNAME> -W -d <DATABASENAME>
  

• The service URL:

  psql "postgres://<USERNAME>@<HOSTNAME>:<PORT>/<DATABASENAME>?sslmode=require"
  

  You are prompted to provide the password.

• The service URL with the password already included and a stricter SSL mode enabled:

  psql "postgres://<USERNAME>:<PASSWORD>@<HOSTNAME>:<PORT>/<DATABASENAME>?sslmode=verify-full"
  

Useful psql commands

When you start using psql, these are the commands you are likely to use most frequently:

Command	Description
\c <DB_NAME>	Connect to a new database
\d	Show the details of a table
\df	List functions in the current database
\df+	List all functions with more details
\di	List all indexes from all tables
\dn	List all schemas in the current database
\dt	List available tables
\du	List Postgres database roles
\dv	List views in current schema
\dv+	List all views with more details
\dx	Show all installed extensions
ef <FUNCTION_NAME>	Edit a function
\h	Show help on syntax of SQL commands
\l	List available databases
\password <USERNAME>	Change the password for the user
\q	Quit psql
\set	Show system variables list
\timing	Show how long a query took to execute
\x	Show expanded query results
\?	List all psql slash commands

For more on psql commands, see the Tiger Data psql cheat sheet and psql documentation.

Save query results to a file

When you run queries in psql, the results are shown in the terminal by default. If you are running queries that have a lot of results, you might like to save the results into a comma-separated .csv file instead. You can do this using the COPY command. For example:

\copy (SELECT * FROM ...) TO '/tmp/output.csv' (format CSV);

This command sends the results of the query to a new file called output.csv in the /tmp/ directory. You can open the file using any spreadsheet program.

Run long queries

To run multi-line queries in psql, use the EOF delimiter. For example:

psql -d target -f -v hypertable=<hypertable> - <<'EOF'
SELECT public.alter_job(j.id, scheduled=>true)
FROM _timescaledb_config.bgw_job j
JOIN _timescaledb_catalog.hypertable h ON h.id = j.hypertable_id
WHERE j.proc_schema IN ('_timescaledb_internal', '_timescaledb_functions')
AND j.proc_name = 'policy_columnstore'
AND j.id >= 1000
AND format('%I.%I', h.schema_name, h.table_name)::text::regclass = :'hypertable'::text::regclass;
EOF

Edit queries in a text editor

Sometimes, queries can get very long, and you might make a mistake when you try typing it the first time around. If you have made a mistake in a long query, instead of retyping it, you can use a built-in text editor, which is based on Vim. Launch the query editor with the \e command. Your previous query is loaded into the editor. When you have made your changes, press Esc, then type :＋w＋q to save the changes, and return to the command prompt. Access the edited query by pressing ↑, and press Enter to run it.

===== PAGE: https://docs.tigerdata.com/integrations/google-cloud/ =====

Integrate Google Cloud with Tiger Cloud

Google Cloud is a suite of cloud computing services, offering scalable infrastructure, AI, analytics, databases, security, and developer tools to help businesses build, deploy, and manage applications.

This page explains how to integrate your Google Cloud infrastructure with Tiger Cloud using AWS Transit Gateway.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details.

• Set up AWS Transit Gateway.

Connect your Google Cloud infrastructure to your Tiger Cloud services

To connect to Tiger Cloud:

1. Connect your infrastructure to AWS Transit Gateway

   Establish connectivity between Google Cloud and AWS. See Connect HA VPN to AWS peer gateways.

2. Create a Peering VPC in Tiger Cloud Console

   1. In Security > VPC, click Create a VPC:

      

   2. Choose your region and IP range, name your VPC, then click Create VPC:

      

      Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your plan in Tiger Cloud Console.

   3. Add a peering connection:

      1. In the VPC Peering column, click Add.
      2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID.

      

      1. Click Add connection.

3. Accept and configure peering connection in your AWS account

   Once your peering connection appears as Processing, you can accept and configure it in AWS:

   1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as Connected in Tiger Cloud Console.

   2. Configure at least the following in your AWS account networking:

      • Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs.
      • Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs.
      • Security groups to allow outbound TCP 5432.

4. Attach a Tiger Cloud service to the Peering VPC In Tiger Cloud Console

   1. Select the service you want to connect to the Peering VPC.
   2. Click Operations > Security > VPC.
   3. Select the VPC, then click Attach VPC.

   You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time.

You have successfully integrated your Google Cloud infrastructure with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/troubleshooting/ =====

Troubleshooting

JDBC authentication type is not supported

When connecting to Tiger Cloud service with a Java Database Connectivity (JDBC) driver, you might get this error message:

Check that your connection definition references your JDBC database with correct URL syntax,
username, and password. The authentication type 10 is not supported.

Your Tiger Cloud authentication type doesn't match your JDBC driver's supported authentication types. The recommended approach is to upgrade your JDBC driver to a version that supports scram-sha-256 encryption. If that isn't an option, you can change the authentication type for your Tiger Cloud service to md5. Note that md5 is less secure, and is provided solely for compatibility with older clients.

For information on changing your authentication type, see the documentation on resetting your service password.

===== PAGE: https://docs.tigerdata.com/integrations/datadog/ =====

Integrate Datadog with Tiger Cloud

Datadog is a cloud-based monitoring and analytics platform that provides comprehensive visibility into applications, infrastructure, and systems through real-time monitoring, logging, and analytics.

This page explains how to:

• Monitor Tiger Cloud service metrics with Datadog

  This integration is available for Scale and Enterprise pricing plans.

• Configure Datadog Agent to collect metrics for your Tiger Cloud service

  This integration is available for all pricing plans.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details.

• Sign up for Datadog.

  You need your Datadog API key to follow this procedure.

• Install Datadog Agent.

Monitor Tiger Cloud service metrics with Datadog

Export telemetry data from your Tiger Cloud services with the time-series and analytics capability enabled to Datadog using a Tiger Cloud data exporter. The available metrics include CPU usage, RAM usage, and storage.

Create a data exporter

A Tiger Cloud data exporter sends telemetry data from a Tiger Cloud service to a third-party monitoring tool. You create an exporter on the project level, in the same AWS region as your service:

1. In Tiger Cloud Console, open Exporters

2. Click New exporter

3. Select Metrics for Data type and Datadog for provider

   

4. Choose your AWS region and provide the API key

   The AWS region must be the same for your Tiger Cloud exporter and the Datadog provider.

5. Set Site to your Datadog region, then click Create exporter

Manage a data exporter

This section shows you how to attach, monitor, edit, and delete a data exporter.

Attach a data exporter to a Tiger Cloud service

To send telemetry data to an external monitoring tool, you attach a data exporter to your Tiger Cloud service. You can attach only one exporter to a service.

To attach an exporter:

1. In Tiger Cloud Console, choose the service
2. Click Operations > Exporters
3. Select the exporter, then click Attach exporter
4. If you are attaching a first Logs data type exporter, restart the service

Monitor Tiger Cloud service metrics

You can now monitor your service metrics. Use the following metrics to check the service is running correctly:

• timescale.cloud.system.cpu.usage.millicores
• timescale.cloud.system.cpu.total.millicores
• timescale.cloud.system.memory.usage.bytes
• timescale.cloud.system.memory.total.bytes
• timescale.cloud.system.disk.usage.bytes
• timescale.cloud.system.disk.total.bytes

Additionally, use the following tags to filter your results.

Tag	Example variable	Description
host	us-east-1.timescale.cloud
project-id
service-id
region	us-east-1	AWS region
role	replica or primary	For service with replicas
node-id		For multi-node services

Edit a data exporter

To update a data exporter:

1. In Tiger Cloud Console, open Exporters
2. Next to the exporter you want to edit, click the menu > Edit
3. Edit the exporter fields and save your changes

You cannot change fields such as the provider or the AWS region.

Delete a data exporter

To remove a data exporter that you no longer need:

1. Disconnect the data exporter from your Tiger Cloud services

   1. In Tiger Cloud Console, choose the service.
   2. Click Operations > Exporters.
   3. Click the trash can icon.
   4. Repeat for every service attached to the exporter you want to remove.

   The data exporter is now unattached from all services. However, it still exists in your project.

2. Delete the exporter on the project level

   1. In Tiger Cloud Console, open Exporters
   2. Next to the exporter you want to edit, click menu > Delete
   3. Confirm that you want to delete the data exporter.

Reference

When you create the IAM OIDC provider, the URL must match the region you create the exporter in. It must be one of the following:

Region	Zone	Location	URL
ap-southeast-1	Asia Pacific	Singapore	irsa-oidc-discovery-prod-ap-southeast-1.s3.ap-southeast-1.amazonaws.com
ap-southeast-2	Asia Pacific	Sydney	irsa-oidc-discovery-prod-ap-southeast-2.s3.ap-southeast-2.amazonaws.com
ap-northeast-1	Asia Pacific	Tokyo	irsa-oidc-discovery-prod-ap-northeast-1.s3.ap-northeast-1.amazonaws.com
ca-central-1	Canada	Central	irsa-oidc-discovery-prod-ca-central-1.s3.ca-central-1.amazonaws.com
eu-central-1	Europe	Frankfurt	irsa-oidc-discovery-prod-eu-central-1.s3.eu-central-1.amazonaws.com
eu-west-1	Europe	Ireland	irsa-oidc-discovery-prod-eu-west-1.s3.eu-west-1.amazonaws.com
eu-west-2	Europe	London	irsa-oidc-discovery-prod-eu-west-2.s3.eu-west-2.amazonaws.com
sa-east-1	South America	São Paulo	irsa-oidc-discovery-prod-sa-east-1.s3.sa-east-1.amazonaws.com
us-east-1	United States	North Virginia	irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com
us-east-2	United States	Ohio	irsa-oidc-discovery-prod-us-east-2.s3.us-east-2.amazonaws.com
us-west-2	United States	Oregon	irsa-oidc-discovery-prod-us-west-2.s3.us-west-2.amazonaws.com

Configure Datadog Agent to collect metrics for your Tiger Cloud services

Datadog Agent includes a Postgres integration that you use to collect detailed Postgres database metrics about your Tiger Cloud services.

1. Connect to your Tiger Cloud service

   For Tiger Cloud, open an SQL editor in Tiger Cloud Console. For self-hosted TimescaleDB, use psql.

2. Add the datadog user to your Tiger Cloud service

   create user datadog with password '<password>';
   

   grant pg_monitor to datadog;
   

   grant SELECT ON pg_stat_database to datadog;
   

3. Test the connection and rights for the datadog user

   Update the following command with your connection details, then run it from the command line:

    psql "postgres://datadog:<datadog password>@<host>:<port>/tsdb?sslmode=require" -c \
    "select * from pg_stat_database LIMIT(1);" \
    && echo -e "\e[0;32mPostgres connection - OK\e[0m" || echo -e "\e[0;31mCannot connect to Postgres\e[0m"
   

   You see the output from the pg_stat_database table, which means you have given the correct rights to datadog.

4. Connect Datadog to your Tiger Cloud service

   1. Configure the Datadog Agent Postgres configuration file; it is usually located on the Datadog Agent host at:

      • Linux: /etc/datadog-agent/conf.d/postgres.d/conf.yaml
      • MacOS: /opt/datadog-agent/etc/conf.d/postgres.d/conf.yaml
      • Windows: C:\ProgramData\Datadog\conf.d\postgres.d\conf.yaml

   2. Integrate Datadog Agent with your Tiger Cloud service:

      Use your connection details to update the following and add it to the Datadog Agent Postgres configuration file:

      init_config:
      
      instances:
      - host: <host>
        port: <port>
        username: datadog
        password: <datadog's password>>
        dbname: tsdb
        disable_generic_tags: true
      

5. Add Tiger Cloud metrics

   Tags to make it easier for build Datadog dashboards that combine metrics from the Tiger Cloud data exporter and Datadog Agent. Use your connection details to update the following and add it to <datadog_home>/datadog.yaml:

   tags:
     - project-id:<project-id>
     - service-id:<service-id>
     - region:<region>
   

6. Restart Datadog Agent

   See how to Start, stop, and restart Datadog Agent.

Metrics for your Tiger Cloud service are now visible in Datadog. Check the Datadog Postgres integration documentation for a comprehensive list of metrics collected.

===== PAGE: https://docs.tigerdata.com/integrations/decodable/ =====

Integrate Decodable with Tiger Cloud

Decodable is a real-time data platform that allows you to build, run, and manage data pipelines effortlessly.

This page explains how to integrate Decodable with your Tiger Cloud service to enable efficient real-time streaming and analytics.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Sign up for Decodable.

  This page uses the pipeline you create using the Decodable Quickstart Guide.

Connect Decodable to your Tiger Cloud service

To stream data gathered in Decodable to a Tiger Cloud service:

1. Create the sync to pipe a Decodable data stream into your Tiger Cloud service

   1. Log in to your Decodable account.

   2. Click Connections, then click New Connection.

   3. Select a PostgreSQL sink connection type, then click Connect.

   4. Using your connection details, fill in the connection information.

      Leave schema and JDBC options empty.

   5. Select the http_events source stream, then click Next.

      Decodable creates the table in your Tiger Cloud service and starts streaming data.

2. Test the connection

   1. Connect to your Tiger Cloud service.

      For Tiger Cloud, open an SQL editor in Tiger Cloud Console. For self-hosted TimescaleDB, use psql.

   2. Check the data from Decodable is streaming into your Tiger Cloud service.

      SELECT * FROM http_events;
      

      You see something like:

      

You have successfully integrated Decodable with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/debezium/ =====

Integrate Debezium with Tiger Cloud

Debezium is an open-source distributed platform for change data capture (CDC). It enables you to capture changes in a self-hosted TimescaleDB instance and stream them to other systems in real time.

Debezium can capture events about:

• Hypertables: captured events are rerouted from their chunk-specific topics to a single logical topic named according to the following pattern: <topic.prefix>.<hypertable-schema-name>.<hypertable-name>

• Continuous aggregates: captured events are rerouted from their chunk-specific topics to a single logical topic named according to the following pattern: <topic.prefix>.<aggregate-schema-name>.<aggregate-name>

• Hypercore: If you enable hypercore, the Debezium TimescaleDB connector does not apply any special processing to data in the columnstore. Compressed chunks are forwarded unchanged to the next downstream job in the pipeline for further processing as needed. Typically, messages with compressed chunks are dropped, and are not processed by subsequent jobs in the pipeline.

  This limitation only affects changes to chunks in the columnstore. Changes to data in the rowstore work correctly.

This page explains how to capture changes in your database and stream them using Debezium on Apache Kafka.

Prerequisites

To follow the steps on this page:

• Create a target self-hosted TimescaleDB instance.

• Install Docker on your development machine.

Configure your database to work with Debezium

To set up self-hosted TimescaleDB to communicate with Debezium:

1. Configure your self-hosted Postgres deployment

   1. Open postgresql.conf.

      The Postgres configuration files are usually located in:

      • Docker: /home/postgres/pgdata/data/
      • Linux: /etc/postgresql/<version>/main/ or /var/lib/pgsql/<version>/data/
      • MacOS: /opt/homebrew/var/postgresql@<version>/
      • Windows: C:\Program Files\PostgreSQL\<version>\data\

   2. Enable logical replication.

      Modify the following settings in postgresql.conf:

      wal_level = logical
      max_replication_slots = 10
      max_wal_senders = 10
      

   3. Open pg_hba.conf and enable host replication.

      To allow replication connections, add the following:

      local replication debezium                         trust
      

      This permission is for the debezium Postgres user running on a local or Docker deployment. For more about replication permissions, see Configuring Postgres to allow replication with the Debezium connector host.

   4. Restart Postgres.

2. Connect to your self-hosted TimescaleDB instance

   Use psql.

3. Create a Debezium user in Postgres

   Create a user with the LOGIN and REPLICATION permissions:

   CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD '<debeziumpassword>';
   

4. Enable a replication spot for Debezium

   1. Create a table for Debezium to listen to:

      CREATE TABLE accounts (created_at TIMESTAMPTZ DEFAULT NOW(),
       name TEXT,
       city TEXT);
      

   2. Turn the table into a hypertable:

      SELECT create_hypertable('accounts', 'created_at');
      

      Debezium also works with continuous aggregates.

   3. Create a publication and enable a replication slot:

      CREATE PUBLICATION dbz_publication FOR ALL TABLES WITH (publish = 'insert, update');
      

Configure Debezium to work with your database

Set up Kafka Connect server, plugins, drivers, and connectors:

1. Run Zookeeper in Docker

   In another Terminal window, run the following command:

   docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 quay.io/debezium/zookeeper:3.0
   

   Check the output log to see that zookeeper is running.

2. Run Kafka in Docker

   In another Terminal window, run the following command:

   docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper quay.io/debezium/kafka:3.0
   

   Check the output log to see that Kafka is running.

3. Run Kafka Connect in Docker

   In another Terminal window, run the following command:

   docker run -it --rm --name connect \
   -p 8083:8083 \
   -e GROUP_ID=1 \
   -e CONFIG_STORAGE_TOPIC=accounts \
   -e OFFSET_STORAGE_TOPIC=offsets \
   -e STATUS_STORAGE_TOPIC=storage \
   --link kafka:kafka \
   --link timescaledb:timescaledb \
   quay.io/debezium/connect:3.0
   

   Check the output log to see that Kafka Connect is running.

4. Register the Debezium Postgres source connector

   Update the <properties> for the <debezium-user> you created in your self-hosted TimescaleDB instance in the following command. Then run the command in another Terminal window:

   curl -X POST http://localhost:8083/connectors \
   -H "Content-Type: application/json" \
   -d '{
      "name": "timescaledb-connector",
      "config": {
         "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
         "database.hostname": "timescaledb",
         "database.port": "5432",
         "database.user": "<debezium-user>",
         "database.password": "<debezium-password>",
         "database.dbname" : "postgres",
         "topic.prefix": "accounts",
         "plugin.name": "pgoutput",
         "schema.include.list": "public,_timescaledb_internal",
         "transforms": "timescaledb",
         "transforms.timescaledb.type": "io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb",
         "transforms.timescaledb.database.hostname": "timescaledb",
         "transforms.timescaledb.database.port": "5432",
         "transforms.timescaledb.database.user": "<debezium-user>",
         "transforms.timescaledb.database.password": "<debezium-password>",
         "transforms.timescaledb.database.dbname": "postgres"
      }
   }'
   

5. Verify timescaledb-source-connector is included in the connector list

   1. Check the tasks associated with timescaledb-connector:

      curl -i -X GET -H "Accept:application/json" localhost:8083/connectors/timescaledb-connector
      

      You see something like:

      {"name":"timescaledb-connector","config":
      { "connector.class":"io.debezium.connector.postgresql.PostgresConnector",
      "transforms.timescaledb.database.hostname":"timescaledb",
      "transforms.timescaledb.database.password":"debeziumpassword","database.user":"debezium",
      "database.dbname":"postgres","transforms.timescaledb.database.dbname":"postgres",
      "transforms.timescaledb.database.user":"debezium",
      "transforms.timescaledb.type":"io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb",
      "transforms.timescaledb.database.port":"5432","transforms":"timescaledb",
      "schema.include.list":"public,_timescaledb_internal","database.port":"5432","plugin.name":"pgoutput",
      "topic.prefix":"accounts","database.hostname":"timescaledb","database.password":"debeziumpassword",
      "name":"timescaledb-connector"},"tasks":[{"connector":"timescaledb-connector","task":0}],"type":"source"}
      

6. Verify timescaledb-connector is running

   1. Open the Terminal window running Kafka Connect. When the connector is active, you see something like the following:

      2025-04-30 10:40:15,168 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal._hyper_1_1_chunk' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,168 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal.bgw_job_stat' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,175 INFO   Postgres|accounts|streaming  SignalProcessor started. Scheduling it every 5000ms   [io.debezium.pipeline.signal.SignalProcessor]
      2025-04-30 10:40:15,175 INFO   Postgres|accounts|streaming  Creating thread debezium-postgresconnector-accounts-SignalProcessor   [io.debezium.util.Threads]
      2025-04-30 10:40:15,175 INFO   Postgres|accounts|streaming  Starting streaming   [io.debezium.pipeline.ChangeEventSourceCoordinator]
      2025-04-30 10:40:15,176 INFO   Postgres|accounts|streaming  Retrieved latest position from stored offset 'LSN{0/1FCE570}'   [io.debezium.connector.postgresql.PostgresStreamingChangeEventSource]
      2025-04-30 10:40:15,176 INFO   Postgres|accounts|streaming  Looking for WAL restart position for last commit LSN 'null' and last change LSN 'LSN{0/1FCE570}'   [io.debezium.connector.postgresql.connection.WalPositionLocator]
      2025-04-30 10:40:15,176 INFO   Postgres|accounts|streaming  Initializing PgOutput logical decoder publication   [io.debezium.connector.postgresql.connection.PostgresReplicationConnection]
      2025-04-30 10:40:15,189 INFO   Postgres|accounts|streaming  Obtained valid replication slot ReplicationSlot [active=false, latestFlushedLsn=LSN{0/1FCCFF0}, catalogXmin=884]   [io.debezium.connector.postgresql.connection.PostgresConnection]
      2025-04-30 10:40:15,189 INFO   Postgres|accounts|streaming  Connection gracefully closed   [io.debezium.jdbc.JdbcConnection]
      2025-04-30 10:40:15,204 INFO   Postgres|accounts|streaming  Requested thread factory for component PostgresConnector, id = accounts named = keep-alive   [io.debezium.util.Threads]
      2025-04-30 10:40:15,204 INFO   Postgres|accounts|streaming  Creating thread debezium-postgresconnector-accounts-keep-alive   [io.debezium.util.Threads]
      2025-04-30 10:40:15,216 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal.bgw_policy_chunk_stats' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,216 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for 'public.accounts' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,217 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal.bgw_job_stat_history' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,217 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal._hyper_1_1_chunk' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,217 INFO   Postgres|accounts|streaming  REPLICA IDENTITY for '_timescaledb_internal.bgw_job_stat' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns   [io.debezium.connector.postgresql.PostgresSchema]
      2025-04-30 10:40:15,219 INFO   Postgres|accounts|streaming  Processing messages   [io.debezium.connector.postgresql.PostgresStreamingChangeEventSource]
      

   2. Watch the events in the accounts topic on your self-hosted TimescaleDB instance.

      In another Terminal instance, run the following command:

      docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka quay.io/debezium/kafka:3.0 watch-topic -a -k accounts
      

      You see the topics being streamed. For example:

      status-task-timescaledb-connector-0	{"state":"RUNNING","trace":null,"worker_id":"172.17.0.5:8083","generation":31}
      status-topic-timescaledb.public.accounts:connector-timescaledb-connector	{"topic":{"name":"timescaledb.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009337985}}
      status-topic-accounts._timescaledb_internal.bgw_job_stat:connector-timescaledb-connector	{"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338118}}
      status-topic-accounts._timescaledb_internal.bgw_job_stat:connector-timescaledb-connector	{"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338120}}
      status-topic-accounts._timescaledb_internal.bgw_job_stat_history:connector-timescaledb-connector	{"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat_history","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338243}}
      status-topic-accounts._timescaledb_internal.bgw_job_stat_history:connector-timescaledb-connector	{"topic":{"name":"accounts._timescaledb_internal.bgw_job_stat_history","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338245}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338250}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}}
      status-topic-accounts.public.accounts:connector-timescaledb-connector	{"topic":{"name":"accounts.public.accounts","connector":"timescaledb-connector","task":0,"discoverTimestamp":1746009338251}}
      ["timescaledb-connector",{"server":"accounts"}]	{"last_snapshot_record":true,"lsn":33351024,"txId":893,"ts_usec":1746009337290783,"snapshot":"INITIAL","snapshot_completed":true}
      status-connector-timescaledb-connector	{"state":"UNASSIGNED","trace":null,"worker_id":"172.17.0.5:8083","generation":31}
      status-task-timescaledb-connector-0	{"state":"UNASSIGNED","trace":null,"worker_id":"172.17.0.5:8083","generation":31}
      status-connector-timescaledb-connector	{"state":"RUNNING","trace":null,"worker_id":"172.17.0.5:8083","generation":33}
      status-task-timescaledb-connector-0	{"state":"RUNNING","trace":null,"worker_id":"172.17.0.5:8083","generation":33}
      

Debezium requires logical replication to be enabled. Currently, this is not enabled by default on Tiger Cloud services. We are working on enabling this feature as you read. As soon as it is live, these docs will be updated.

And that is it, you have configured Debezium to interact with Tiger Data products.

===== PAGE: https://docs.tigerdata.com/integrations/fivetran/ =====

Integrate Fivetran with Tiger Cloud

Fivetran is a fully managed data pipeline platform that simplifies ETL (Extract, Transform, Load) processes by automatically syncing data from multiple sources to your data warehouse.

This page shows you how to inject data from data sources managed by Fivetran into a Tiger Cloud service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Sign up for Fivetran

Set your Tiger Cloud service as a destination in Fivetran

To be able to inject data into your Tiger Cloud service, set it as a destination in Fivetran:

1. In Fivetran Dashboard > Destinations, click Add destination.

2. Search for the PostgreSQL connector and click Select. Add the destination name and click Add.

3. In the PostgreSQL setup, add your Tiger Cloud service connection details, then click Save & Test.

   Fivetran validates the connection settings and sets up any security configurations.

4. Click View Destination.

   The Destination Connection Details page opens.

Set up a Fivetran connection as your data source

In a real world scenario, you can select any of the over 600 connectors available in Fivetran to sync data with your Tiger Cloud service. This section shows you how to inject the logs for your Fivetran connections into your Tiger Cloud service.

1. In Fivetran Dashboard > Connections, click Add connector.

2. Search for the Fivetran Platform connector, then click Setup.

3. Leave the default schema name, then click Save & Test.

   You see All connection tests passed!

4. Click Continue, enable Add Quickstart Data Model and click Continue.

   Your Fivetran connection is connected to your Tiger Cloud service destination.

5. Click Start Initial Sync.

   Fivetran creates the log schema in your service and syncs the data to your service.

View Fivetran data in your Tiger Cloud service

To see data injected by Fivetran into your Tiger Cloud service:

1. In data mode in Tiger Cloud Console, select your service, then run the following query:

   SELECT *
   FROM fivetran_log.account
   LIMIT 10;
   

   You see something like the following:

   

You have successfully integrated Fivetran with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/find-connection-details/ =====

Find your connection details

To connect to your Tiger Cloud service or self-hosted TimescaleDB, you need at least the following:

• Hostname
• Port
• Username
• Password
• Database name

Find the connection details based on your deployment type:

Connect to your service

Retrieve the connection details for your Tiger Cloud service:

• In <service name>-credentials.txt:

  All connection details are supplied in the configuration file you download when you create a new service.

• In Tiger Cloud Console:

  Open the Services page and select your service. The connection details, except the password, are available in Service info > Connection info > More details. If necessary, click Forgot your password? to get a new one.

  

Find your project and service ID

To retrieve the connection details for your Tiger Cloud project and Tiger Cloud service:

1. Retrieve your project ID:

   In Tiger Cloud Console, click your project name in the upper left corner, then click Copy next to the project ID.

2. Retrieve your service ID:

   Click the dots next to the service, then click Copy next to the service ID.

Create client credentials

You use client credentials to obtain access tokens outside of the user context.

To retrieve the connection details for your Tiger Cloud project for programmatic usage such as Terraform or the Tiger Cloud REST API:

1. Open the settings for your project:

   In Tiger Cloud Console, click your project name in the upper left corner, then click Project settings.

2. Create client credentials:

   1. Click Create credentials, then copy Public key and Secret key locally.

      

      This is the only time you see the Secret key. After this, only the Public key is visible in this page.

   2. Click Done.

Create client credentials

You use client credentials to obtain access tokens outside of the user context.

To retrieve the connection details for your Tiger Cloud project for programmatic usage such as Terraform or the Tiger Cloud REST API:

1. Open the settings for your project:

   In Tiger Cloud Console, click your project name in the upper left corner, then click Project settings.

2. Create client credentials:

   1. Click Create credentials, then copy Public key and Secret key locally.

      

      This is the only time you see the Secret key. After this, only the Public key is visible in this page.

   2. Click Done.

Find the connection details in the Postgres configuration file or by asking your database administrator. The postgres superuser, created during Postgres installation, has all the permissions required to run procedures in this documentation. However, it is recommended to create other users and assign permissions on the need-only basis.

In the Services page of the MST Console, click the service you want to connect to. You see the connection details:

===== PAGE: https://docs.tigerdata.com/integrations/terraform/ =====

Integrate Terraform with Tiger

Terraform is an infrastructure-as-code tool that enables you to safely and predictably provision and manage infrastructure.

This page explains how to configure Terraform to manage your Tiger Cloud service or self-hosted TimescaleDB.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Download and install Terraform.

Configure Terraform

Configure Terraform based on your deployment type:

You use the Tiger Data Terraform provider to manage Tiger Cloud services:

1. Generate client credentials for programmatic use

   1. In Tiger Cloud Console, click Projects and save your Project ID, then click Project settings.

   2. Click Create credentials, then save Public key and Secret key.

2. Configure Tiger Data Terraform provider

   1. Create a main.tf configuration file with at least the following content. Change x.y.z to the latest version of the provider.

      terraform {
        required_providers {
          timescale = {
            source  = "timescale/timescale"
            version = "x.y.z"
          }
        }
      }
      
      provider "timescale" {
       project_id = var.ts_project_id
       access_key = var.ts_access_key
       secret_key = var.ts_secret_key
      }
      
      variable "ts_project_id" {
       type = string
      }
      
      variable "ts_access_key" {
       type = string
      }
      
      variable "ts_secret_key" {
       type = string
      }
      

   2. Create a terraform.tfvars file in the same directory as your main.tf to pass in the variable values:

      export TF_VAR_ts_project_id="<your-timescale-project-id>"
      export TF_VAR_ts_access_key="<your-timescale-access-key>"
      export TF_VAR_ts_secret_key="<your-timescale-secret-key>"
      

3. Add your resources

   Add your Tiger Cloud services or VPC connections to the main.tf configuration file. For example:

   resource "timescale_service" "test" {
     name              = "test-service"
     milli_cpu         = 500
     memory_gb         = 2
     region_code       = "us-east-1"
     enable_ha_replica = false
   
     timeouts = {
       create = "30m"
     }
   }
   
   resource "timescale_vpc" "vpc" {
     cidr         = "10.10.0.0/16"
     name         = "test-vpc"
     region_code  = "us-east-1"
   }
   

You can now manage your resources with Terraform. See more about available resources and data sources.

You use the cyrilgdn/postgresql Postgres provider to connect to your self-hosted TimescaleDB instance.

Create a main.tf configuration file with the following content, using your connection details:

   terraform {
    required_providers {
     postgresql = {
      source  = "cyrilgdn/postgresql"
      version = ">= 1.15.0"
     }
    }
   }

   provider "postgresql" {
    host            = "your-timescaledb-host"
    port            = "your-timescaledb-port"
    database        = "your-database-name"
    username        = "your-username"
    password        = "your-password"
    sslmode         = "require" # Or "disable" if SSL isn't enabled
   }

You can now manage your database with Terraform.

===== PAGE: https://docs.tigerdata.com/integrations/azure-data-studio/ =====

Integrate Azure Data Studio with Tiger

Azure Data Studio is an open-source, cross-platform hybrid data analytics tool designed to simplify the data landscape.

This page explains how to integrate Azure Data Studio with Tiger Cloud.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Download and install Azure Data Studio.

• Install the Postgres extension for Azure Data Studio.

Connect to your Tiger Cloud service with Azure Data Studio

To connect to Tiger Cloud:

1. Start Azure Data Studio

2. In the SERVERS page, click New Connection

3. Configure the connection

   1. Select PostgreSQL for Connection type.

   2. Configure the server name, database, username, port, and password using your connection details.

   3. Click Advanced.

      If you configured your Tiger Cloud service to connect using stricter SSL mode, set SSL mode to the configured mode, then type the location of your SSL root CA certificate in SSL root certificate filename.

   4. In the Port field, type the port number and click OK.

4. Click Connect

You have successfully integrated Azure Data Studio with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/telegraf/ =====

Ingest data using Telegraf

Telegraf is a server-based agent that collects and sends metrics and events from databases, systems, and IoT sensors. Telegraf is an open source, plugin-driven tool for the collection and output of data.

To view metrics gathered by Telegraf and stored in a hypertable in a Tiger Cloud service.

• Link Telegraf to your Tiger Cloud service: create a Telegraf configuration
• View the metrics collected by Telegraf: connect to your service and query the metrics table

Prerequisites

Best practice is to use an Ubuntu EC2 instance hosted in the same region as your Tiger Cloud service as a migration machine. That is, the machine you run the commands on to move your data from your source database to your target Tiger Cloud service.

Before you migrate your data:

• Create a target Tiger Cloud service.

  Each Tiger Cloud service has a single database that supports the most popular extensions. Tiger Cloud services do not support tablespaces, and there is no superuser associated with a service. Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window.

• To ensure that maintenance does not run during the process, adjust the maintenance window.

• Install Telegraf

Link Telegraf to your service

To create a Telegraf configuration that exports data to a hypertable in your service:

1. Set up your service connection string

   This variable holds the connection information for the target Tiger Cloud service.

In the terminal on the source machine, set the following:

export TARGET=postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require

See where to find your connection details.

1. Generate a Telegraf configuration file

   In Terminal, run the following:

   telegraf --input-filter=cpu --output-filter=postgresql config > telegraf.conf
   

   telegraf.conf configures a CPU input plugin that samples various metrics about CPU usage, and the Postgres output plugin. telegraf.conf also includes all available input, output, processor, and aggregator plugins. These are commented out by default.

2. Test the configuration

   telegraf --config telegraf.conf --test
   

   You see an output similar to the following:

   2022-11-28T12:53:44Z I! Starting Telegraf 1.24.3
   2022-11-28T12:53:44Z I! Available plugins: 208 inputs, 9 aggregators, 26 processors, 20 parsers, 57 outputs
   2022-11-28T12:53:44Z I! Loaded inputs: cpu
   2022-11-28T12:53:44Z I! Loaded aggregators:
   2022-11-28T12:53:44Z I! Loaded processors:
   2022-11-28T12:53:44Z W! Outputs are not used in testing mode!
   2022-11-28T12:53:44Z I! Tags enabled: host=localhost
   > cpu,cpu=cpu0,host=localhost usage_guest=0,usage_guest_nice=0,usage_idle=90.00000000087311,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=6.000000000040018,usage_user=3.999999999996362 1669640025000000000
   > cpu,cpu=cpu1,host=localhost usage_guest=0,usage_guest_nice=0,usage_idle=92.15686274495818,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=5.882352941192206,usage_user=1.9607843136712912 1669640025000000000
   > cpu,cpu=cpu2,host=localhost usage_guest=0,usage_guest_nice=0,usage_idle=91.99999999982538,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=3.999999999996362,usage_user=3.999999999996362 1669640025000000000
   

3. Configure the Postgres output plugin

   1. In telegraf.conf, in the [[outputs.postgresql]] section, set connection to the value of target.

   connection = "<VALUE OF target>"
   

   1. Use hypertables when Telegraf creates a new table:

      In the section that begins with the comment ## Templated statements to execute when creating a new table, add the following template:

      ## Templated statements to execute when creating a new table.
      
      

      The by_range dimension builder was added to TimescaleDB 2.13.

View the metrics collected by Telegraf

This section shows you how to generate system metrics using Telegraf, then connect to your service and query the metrics hypertable.

1. Collect system metrics using Telegraf

   Run the following command for a 30 seconds:

   telegraf --config telegraf.conf
   

   Telegraf uses loaded inputs cpu and outputs postgresql along with global tags, the intervals when the agent collects data from the inputs, and flushes to the outputs.

2. View the metrics

   1. Connect to your Tiger Cloud service:

       psql target
      

   2. View the metrics collected in the cpu table in tsdb:

      SELECT*FROM cpu;
      

      You see something like:

      time         |    cpu    |               host               | usage_guest | usage_guest_nice |    usage_idle     | usage_iowait | usage_irq | usage_nice | usage_softirq | usage_steal |    usage_system     |     usage_user
      ---------------------+-----------+----------------------------------+-------------+------------------+-------------------+--------------+-----------+------------+---------------+-------------+---------------------+---------------------
      2022-12-05 12:25:20 | cpu0      | hostname |           0 |                0 | 83.08605341237833 |            0 |         0 |          0 |             0 |           0 |   6.824925815961274 |  10.089020771444481
      2022-12-05 12:25:20 | cpu1      | hostname |           0 |                0 | 84.27299703278959 |            0 |         0 |          0 |             0 |           0 |   5.934718100814769 |   9.792284866395647
      2022-12-05 12:25:20 | cpu2      | hostname |           0 |                0 | 87.53709198848934 |            0 |         0 |          0 |             0 |           0 |   4.747774480755411 |   7.715133531241037
      2022-12-05 12:25:20 | cpu3      | hostname|           0 |                0 | 86.68639053296472 |            0 |         0 |          0 |             0 |           0 |    4.43786982253345 |   8.875739645039992
      2022-12-05 12:25:20 | cpu4      | hostname |           0 |                0 | 96.15384615371369 |            0 |         0 |          0 |             0 |           0 |  1.1834319526667423 |  2.6627218934917614
      

      To view the average usage per CPU core, use SELECT cpu, avg(usage_user) FROM cpu GROUP BY cpu;.

For more information about the options that you can configure in Telegraf, see the PostgreQL output plugin.

===== PAGE: https://docs.tigerdata.com/integrations/supabase/ =====

Integrate Supabase with Tiger

Supabase is an open source Firebase alternative. This page shows how to run real-time analytical queries against a Tiger Cloud service through Supabase using a foreign data wrapper (fdw) to bring aggregated data from your Tiger Cloud service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Create a Supabase project

Set up your Tiger Cloud service

To set up a Tiger Cloud service optimized for analytics to receive data from Supabase:

1. Optimize time-series data in hypertables

   Time-series data represents how a system, process, or behavior changes over time. Hypertables are Postgres tables that help you improve insert and query performance by automatically partitioning your data by time.

   1. Connect to your Tiger Cloud service and create a table that will point to a Supabase database:

      CREATE TABLE signs (
          time timestamptz NOT NULL DEFAULT now(),
          origin_time timestamptz NOT NULL,
          name TEXT
      ) WITH (
        tsdb.hypertable,
        tsdb.partition_column='time'
      );
      

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

2. Optimize cooling data for analytics

   Hypercore is the hybrid row-columnar storage engine in TimescaleDB, designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage. This flexibility enables TimescaleDB to deliver the best of both worlds, solving the key challenges in real-time analytics.

   ALTER TABLE signs SET (
     timescaledb.enable_columnstore = true,
     timescaledb.segmentby = 'name');
   

3. Create optimized analytical queries

   Continuous aggregates are designed to make queries on very large datasets run faster. Continuous aggregates in Tiger Cloud use Postgres materialized views to continuously, and incrementally refresh a query in the background, so that when you run the query, only the data that has changed needs to be computed, not the entire dataset.

   1. Create a continuous aggregate pointing to the Supabase database.

      CREATE MATERIALIZED VIEW IF NOT EXISTS signs_per_minute
      WITH (timescaledb.continuous)
      AS
      SELECT time_bucket('1 minute', time) as ts,
       name,
       count(*) as total
      FROM signs
      GROUP BY 1, 2
      WITH NO DATA;
      

   2. Setup a delay stats comparing origin_time to time.

      CREATE MATERIALIZED VIEW IF NOT EXISTS _signs_per_minute_delay
      WITH (timescaledb.continuous)
      AS
      SELECT time_bucket('1 minute', time) as ts,
        stats_agg(extract(epoch from origin_time - time)::float8) as delay_agg,
        candlestick_agg(time, extract(epoch from origin_time - time)::float8, 1) as delay_candlestick
      FROM signs GROUP BY 1
      WITH NO DATA;
      

   3. Setup a view to recieve the data from Supabase.

      CREATE VIEW signs_per_minute_delay
      AS
        SELECT ts,
        average(delay_agg) as avg_delay,
        stddev(delay_agg) as stddev_delay,
        open(delay_candlestick) as open,
        high(delay_candlestick) as high,
        low(delay_candlestick) as low,
        close(delay_candlestick) as close
      FROM _signs_per_minute_delay
      

4. Add refresh policies for your analytical queries

   You use start_offset and end_offset to define the time range that the continuous aggregate will cover. Assuming that the data is being inserted without any delay, set the start_offset to 5 minutes and the end_offset to 1 minute. This means that the continuous aggregate is refreshed every minute, and the refresh covers the last 5 minutes. You set schedule_interval to INTERVAL '1 minute' so the continuous aggregate refreshes on your Tiger Cloud service every minute. The data is accessed from Supabase, and the continuous aggregate is refreshed every minute in the other side.

   SELECT add_continuous_aggregate_policy('signs_per_minute',
    start_offset => INTERVAL '5 minutes',
    end_offset => INTERVAL '1 minute',
    schedule_interval => INTERVAL '1 minute');
   

   Do the same thing for data inserted with a delay:

   SELECT add_continuous_aggregate_policy('_signs_per_minute_delay',
    start_offset => INTERVAL '5 minutes',
    end_offset => INTERVAL '1 minute',
    schedule_interval => INTERVAL '1 minute');
   

Set up a Supabase database

To set up a Supabase database that injects data into your Tiger Cloud service:

1. Connect a foreign server in Supabase to your Tiger Cloud service

   1. Connect to your Supabase project using Supabase dashboard or psql.

   2. Enable the postgres_fdw extension.

      CREATE EXTENSION postgres_fdw;
      

   3. Create a foreign server that points to your Tiger Cloud service.

      Update the following command with your connection details, then run it in the Supabase database:

      CREATE SERVER timescale
      FOREIGN DATA WRAPPER postgres_fdw
      OPTIONS (
          host '<value of host>',
          port '<value of port>',
          dbname '<value of dbname>',
          sslmode 'require',
          extensions 'timescaledb'
      );
      

2. Create the user mapping for the foreign server

   Update the following command with your connection details, the run it in the Supabase database:

   CREATE USER MAPPING FOR CURRENT_USER
   SERVER timescale
   OPTIONS (
      user '<value of user>',
      password '<value of password>'
   );
   

3. Create a foreign table that points to a table in your Tiger Cloud service.

   This query introduced the following columns:

   • time: with a default value of now(). This is because the time column is used by Tiger Cloud to optimize data in the columnstore.
   • origin_time: store the original timestamp of the data.

   Using both columns, you understand the delay between Supabase (origin_time) and the time the data is inserted into your Tiger Cloud service (time).

   CREATE FOREIGN TABLE signs (
     TIME timestamptz NOT NULL DEFAULT now(),
     origin_time timestamptz NOT NULL,
     NAME TEXT)
   SERVER timescale OPTIONS (
     schema_name 'public',
     table_name 'signs'
   );
   

4. Create a foreign table in Supabase

   1. Create a foreign table that matches the signs_per_minute view in your Tiger Cloud service. It represents a top level view of the data.

      CREATE FOREIGN TABLE signs_per_minute (
       ts timestamptz,
       name text,
       total int
      )
      SERVER timescale OPTIONS (schema_name 'public', table_name 'signs_per_minute');
      

   2. Create a foreign table that matches the signs_per_minute_delay view in your Tiger Cloud service.

      CREATE FOREIGN TABLE signs_per_minute_delay (
         ts timestamptz,
         avg_delay float8,
         stddev_delay float8,
         open float8,
         high float8,
         low float8,
         close float8
      ) SERVER timescale OPTIONS (schema_name 'public', table_name 'signs_per_minute_delay');
      

Test the integration

To inject data into your Tiger Cloud service from a Supabase database using a foreign table:

1. Insert data into your Supabase database

   Connect to Supabase and run the following query:

   INSERT INTO signs (origin_time, name) VALUES (now(), 'test')
   

2. Check the data in your Tiger Cloud service

   Connect to your Tiger Cloud service and run the following query:

   SELECT * from signs;
   

   You see something like:

   origin_time	time	name
   2025-02-27 16:30:04.682391+00	2025-02-27 16:30:04.682391+00	test

You have successfully integrated Supabase with your Tiger Cloud service.

===== PAGE: https://docs.tigerdata.com/integrations/index/ =====

Integrations

You can integrate your Tiger Cloud service with third-party solutions to expand and extend what you can do with your data.

Integrates with Postgres? Integrates with your service!

A Tiger Cloud service is a Postgres database instance extended by Tiger Data with custom capabilities. This means that any third-party solution that you can integrate with Postgres, you can also integrate with Tiger Cloud. See the full list of Postgres integrations here.

Some of the most in-demand integrations are listed below.

Authentication and security

Name	Description
Auth.js	Implement authentication and authorization for web applications.
Auth0	Securely manage user authentication and access controls for applications.
Okta	Secure authentication and user identity management for applications.

Business intelligence and data visualization

Name	Description
Cube.js	Build and optimize data APIs for analytics applications.
Looker	Explore, analyze, and share business insights with a BI platform.
Metabase	Create dashboards and visualize business data without SQL expertise.
Power BI	Visualize data, build interactive dashboards, and share insights.
Superset	Create and explore data visualizations and dashboards.

Configuration and deployment

Name	Description
Azure Functions	Run event-driven serverless code in the cloud without managing infrastructure.
Deno Deploy	Deploy and run JavaScript and TypeScript applications at the edge.
Flyway	Manage and automate database migrations using version control.
Liquibase	Track, version, and automate database schema changes.
Pulumi	Define and manage cloud infrastructure using code in multiple languages.
Render	Deploy and scale web applications, databases, and services easily.
Terraform	Safely and predictably provision and manage infrastructure in any cloud.
Kubernetes	Deploy, scale, and manage containerized applications automatically.

Data engineering and extract, transform, load

Name	Description
Airbyte	Sync data between various sources and destinations.
Amazon SageMaker	Build, train, and deploy ML models into a production-ready hosted environment.
Apache Airflow	Programmatically author, schedule, and monitor workflows.
Apache Beam	Build and execute batch and streaming data pipelines across multiple processing engines.
Apache Kafka	Stream high-performance data pipelines, analytics, and data integration.
AWS Lambda	Run code without provisioning or managing servers, scaling automatically as needed.
dbt	Transform and model data in your warehouse using SQL-based workflows.
Debezium	Capture and stream real-time changes from databases.
Decodable	Build, run, and manage data pipelines effortlessly.
DeltaLake	Enhance data lakes with ACID transactions and schema enforcement.
Firebase Wrapper	Simplify interactions with Firebase services through an abstraction layer.
Stitch	Extract, load, and transform data from various sources to data warehouses.

Data ingestion and streaming

Name	Description
Apache Spark	Process large-scale data workloads quickly using distributed computing.
Confluent	Manage and scale Apache Kafka-based event streaming applications. You can also set up Postgres as a source.
ElectricSQL	Enable real-time synchronization between databases and frontend applications.
EMQX	Deploy an enterprise-grade MQTT broker for IoT messaging.
Estuary	Stream and synchronize data in real time between different systems.
Flink	Process real-time data streams with fault-tolerant distributed computing.
Fivetran	Sync data from multiple sources to your data warehouse.
HighByte	Connect operational technology sources, model the data, and stream it into Postgres.
Redpanda	Stream and process real-time data as a Kafka-compatible platform.
Striim	Ingest, process, and analyze real-time data streams.

Development tools

Name	Description
Deepnote	Collaborate on data science projects with a cloud-based notebook platform.
Django	Develop scalable and secure web applications using a Python framework.
LangChain	Build applications that integrate with language models like GPT.
Rust	Build high-performance, memory-safe applications with a modern programming language.
Streamlit	Create interactive data applications and dashboards using Python.

Language-specific integrations

Name	Description
Golang	Integrate Tiger Cloud with a Golang application.
Java	Integrate Tiger Cloud with a Java application.
Node.js	Integrate Tiger Cloud with a Node.js application.
Python	Integrate Tiger Cloud with a Python application.
Ruby	Integrate Tiger Cloud with a Ruby application.

Logging and system administration

Name	Description
RSyslog	Collect, filter, and forward system logs for centralized logging.
SchemaSpy	Generate database schema documentation and visualization.

Observability and alerting

Name	Description
Amazon Cloudwatch	Collect, analyze, and act on data from applications, infrastructure, and services running in AWS and on-premises environments.
Apache SkyWalking	Monitor, trace, and diagnose distributed applications for improved observability. You can also set up Postgres as storage.
Azure Monitor	Collect and analyze telemetry data from cloud and on-premises environments.
Dash0	OpenTelemetry Native Observability, built on CNCF Open Standards like PromQL, Perses, and OTLP, and offering full cost control.
Datadog	Gain comprehensive visibility into applications, infrastructure, and systems through real-time monitoring, logging, and analytics.
Grafana	Query, visualize, alert on, and explore your metrics and logs.
IBM Instana	Monitor application performance and detect issues in real-time.
Jaeger	Trace and diagnose distributed transactions for observability.
New Relic	Monitor applications, infrastructure, and logs for performance insights.
OpenTelemetry Beta	Collect and analyze telemetry data for observability across systems.
Prometheus	Track the performance and health of systems, applications, and infrastructure.
SigNoz	Monitor application performance with an open-source observability tool.
Tableau	Connect to data sources, analyze data, and create interactive visualizations and dashboards.
Telegraf	Collect, process, and ship metrics and events into databases or monitoring platforms.

Query and administration

Name	Description
Azure Data Studio	Query, manage, visualize, and develop databases across SQL Server, Azure SQL, and Postgres.
DBeaver	Connect to, manage, query, and analyze multiple database in a single interface with SQL editing, visualization, and administration tools.
Forest Admin	Create admin panels and dashboards for business applications.
Hasura	Instantly generate GraphQL APIs from databases with access control.
Mode Analytics	Analyze data, create reports, and share insights with teams.
Neon	Run a cloud-native, serverless Postgres database with automatic scaling.
pgAdmin	Manage, query, and administer Postgres databases through a graphical interface.
Postgres	Access and query data from external sources as if they were regular Postgres tables.
Prisma	Simplify database access with an open-source ORM for Node.js.
psql	Run SQL queries, manage databases, automate tasks, and interact directly with Postgres.
Qlik Replicate	Move and synchronize data across multiple database platforms. You an also set up Postgres as a source.
qStudio	Write and execute SQL queries, manage database objects, and analyze data in a user-friendly interface.
Redash	Query, visualize, and share data from multiple sources.
SQLalchemy	Manage database operations using a Python SQL toolkit and ORM.
Sequelize	Interact with SQL databases in Node.js using an ORM.
StepZen	Build and deploy GraphQL APIs with data from multiple sources.
TypeORM	Work with databases in TypeScript and JavaScript using an ORM.

Secure connectivity to Tiger Cloud

Name	Description
Amazon Web Services	Connect your other services and applications running in AWS to Tiger Cloud.
Corporate data center	Connect your on-premise data center to Tiger Cloud.
Google Cloud	Connect your Google Cloud infrastructure to Tiger Cloud.
Microsoft Azure	Connect your Microsoft Azure infrastructure to Tiger Cloud.

Workflow automation and no-code tools

Name	Description
Appsmith	Create internal business applications with a low-code platform.
n8n	Automate workflows and integrate services with a no-code platform.
Retool	Build custom internal tools quickly using a drag-and-drop interface.
Tooljet	Develop internal tools and business applications with a low-code builder.
Zapier	Automate workflows by connecting different applications and services.

===== PAGE: https://docs.tigerdata.com/integrations/aws-lambda/ =====

Integrate AWS Lambda with Tiger Cloud

AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS) that allows you to run code without provisioning or managing servers, scaling automatically as needed.

This page shows you how to integrate AWS Lambda with Tiger Cloud service to process and store time-series data efficiently.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Set up an AWS Account.

• Install and configure AWS CLI.

• Install NodeJS v18.x or later.

Prepare your Tiger Cloud service to ingest data from AWS Lambda

Create a table in Tiger Cloud service to store time-series data.

1. Connect to your Tiger Cloud service

   For Tiger Cloud, open an SQL editor in Tiger Cloud Console. For self-hosted TimescaleDB, use psql.

2. Create a hypertable to store sensor data

   Hypertables are Postgres tables that automatically partition your data by time. You interact with hypertables in the same way as regular Postgres tables, but with extra features that make managing your time-series data much easier.

   CREATE TABLE sensor_data (
     time TIMESTAMPTZ NOT NULL,
     sensor_id TEXT NOT NULL,
     value DOUBLE PRECISION NOT NULL
   ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='time'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

Create the code to inject data into a Tiger Cloud service

Write an AWS Lambda function in a Node.js project that processes and inserts time-series data into a Tiger Cloud service.

1. Initialize a new Node.js project to hold your Lambda function

   mkdir lambda-timescale && cd lambda-timescale
   npm init -y
   

2. Install the Postgres client library in your project

   npm install pg
   

3. Write a Lambda Function that inserts data into your Tiger Cloud service

   Create a file named index.js, then add the following code:

   const {
       Client
   } = require('pg');
   
   exports.handler = async (event) => {
       const client = new Client({
           host: process.env.TIMESCALE_HOST,
           port: process.env.TIMESCALE_PORT,
           user: process.env.TIMESCALE_USER,
           password: process.env.TIMESCALE_PASSWORD,
           database: process.env.TIMESCALE_DB,
       });
   
       try {
           await client.connect();
            //
           const query = `
               INSERT INTO sensor_data (time, sensor_id, value)
               VALUES ($1, $2, $3);
               `;
   
           const data = JSON.parse(event.body);
           const values = [new Date(), data.sensor_id, data.value];
   
           await client.query(query, values);
   
           return {
               statusCode: 200,
               body: JSON.stringify({
                   message: 'Data inserted successfully!'
               }),
           };
       } catch (error) {
           console.error('Error inserting data:', error);
           return {
               statusCode: 500,
               body: JSON.stringify({
                   error: 'Failed to insert data.'
               }),
           };
       } finally {
           await client.end();
       }
   
   };
   

Deploy your Node project to AWS Lambda

To create an AWS Lambda function that injects data into your Tiger Cloud service:

1. Compress your code into a .zip

   zip -r lambda-timescale.zip .
   

2. Deploy to AWS Lambda

   In the following example, replace <IAM_ROLE_ARN> with your AWS IAM credentials, then use AWS CLI to create a Lambda function for your project:

   aws lambda create-function \
      --function-name TimescaleIntegration \
      --runtime nodejs14.x \
      --role <IAM_ROLE_ARN> \
      --handler index.handler \
      --zip-file fileb://lambda-timescale.zip
   

3. Set up environment variables

   In the following example, use your connection details to add your Tiger Cloud service connection settings to your Lambda function:

   aws lambda update-function-configuration \
   --function-name TimescaleIntegration \
   --environment "Variables={TIMESCALE_HOST=<host>,TIMESCALE_PORT=<port>, \
                  TIMESCALE_USER=<Username>,TIMESCALE_PASSWORD=<Password>, \
                  TIMESCALE_DB=<Database name>}"
   

4. Test your AWS Lambda function

   1. Invoke the Lambda function and send some data to your Tiger Cloud service:

      aws lambda invoke \
         --function-name TimescaleIntegration \
         --payload '{"body": "{\"sensor_id\": \"sensor-123\", \"value\": 42.5}"}' \
         --cli-binary-format raw-in-base64-out \
         response.json
      

   2. Verify that the data is in your service.

      Open an SQL editor and check the sensor_data table:

      SELECT * FROM sensor_data;
      

      You see something like:

      time	sensor_id	value
      2025-02-10 10:58:45.134912+00	sensor-123	42.5

You can now seamlessly ingest time-series data from AWS Lambda into Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/postgresql/ =====

Integrate with PostgreSQL

You use Postgres foreign data wrappers (FDWs) to query external data sources from a Tiger Cloud service. These external data sources can be one of the following:

• Other Tiger Cloud services
• Postgres databases outside of Tiger Cloud

If you are using VPC peering, you can create FDWs in your Customer VPC to query a service in your Tiger Cloud project. However, you can't create FDWs in your Tiger Cloud services to query a data source in your Customer VPC. This is because Tiger Cloud VPC peering uses AWS PrivateLink for increased security. See VPC peering documentation for additional details.

Postgres FDWs are particularly useful if you manage multiple Tiger Cloud services with different capabilities, and need to seamlessly access and merge regular and time-series data.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Query another data source

To query another data source:

You create Postgres FDWs with the postgres_fdw extension, which is enabled by default in Tiger Cloud.

1. Connect to your service

   See how to connect.

2. Create a server

   Run the following command using your connection details:

   CREATE SERVER myserver
   FOREIGN DATA WRAPPER postgres_fdw
   OPTIONS (host '<host>', dbname 'tsdb', port '<port>');
   

3. Create user mapping

   Run the following command using your connection details:

   CREATE USER MAPPING FOR tsdbadmin
   SERVER myserver
   OPTIONS (user 'tsdbadmin', password '<password>');
   

4. Import a foreign schema (recommended) or create a foreign table

   • Import the whole schema:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     FROM SERVER myserver
     INTO foreign_stuff ;
     

   • Alternatively, import a limited number of tables:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     LIMIT TO (table1, table2)
     FROM SERVER myserver
     INTO foreign_stuff;
     

   • Create a foreign table. Skip if you are importing a schema:

     CREATE FOREIGN TABLE films (
         code        char(5) NOT NULL,
         title       varchar(40) NOT NULL,
         did         integer NOT NULL,
         date_prod   date,
         kind        varchar(10),
         len         interval hour to minute
     )
     SERVER film_server;
     

A user with the tsdbadmin role assigned already has the required USAGE permission to create Postgres FDWs. You can enable another user, without the tsdbadmin role assigned, to query foreign data. To do so, explicitly grant the permission. For example, for a new grafana user:

CREATE USER grafana;

GRANT grafana TO tsdbadmin;

CREATE SCHEMA fdw AUTHORIZATION grafana;

CREATE SERVER db1 FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host '<host>', dbname 'tsdb', port '<port>');

CREATE USER MAPPING FOR grafana SERVER db1
OPTIONS (user 'tsdbadmin', password '<password>');

GRANT USAGE ON FOREIGN SERVER db1 TO grafana;

SET ROLE grafana;

IMPORT FOREIGN SCHEMA public
       FROM SERVER db1
       INTO fdw;

You create Postgres FDWs with the postgres_fdw extension. See documenation on how to enable it.

1. Connect to your database

   Use psql to connect to your database.

2. Create a server

   Run the following command using your connection details:

   CREATE SERVER myserver
   FOREIGN DATA WRAPPER postgres_fdw
   OPTIONS (host '<host>', dbname '<database_name>', port '<port>');
   

3. Create user mapping

   Run the following command using your connection details:

   CREATE USER MAPPING FOR postgres
   SERVER myserver
   OPTIONS (user 'postgres', password '<password>');
   

4. Import a foreign schema (recommended) or create a foreign table

   • Import the whole schema:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     FROM SERVER myserver
     INTO foreign_stuff ;
     

   • Alternatively, import a limited number of tables:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     LIMIT TO (table1, table2)
     FROM SERVER myserver
     INTO foreign_stuff;
     

   • Create a foreign table. Skip if you are importing a schema:

     CREATE FOREIGN TABLE films (
         code        char(5) NOT NULL,
         title       varchar(40) NOT NULL,
         did         integer NOT NULL,
         date_prod   date,
         kind        varchar(10),
         len         interval hour to minute
     )
     SERVER film_server;
     

===== PAGE: https://docs.tigerdata.com/integrations/power-bi/ =====

Integrate Power BI with Tiger

Power BI is a business analytics tool for visualizing data, creating interactive reports, and sharing insights across an organization.

This page explains how to integrate Power BI with Tiger Cloud using the Postgres ODBC driver, so that you can build interactive reports based on the data in your Tiger Cloud service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Download Power BI Desktop on your Microsoft Windows machine.
• Install the PostgreSQL ODBC driver.

Add your Tiger Cloud service as an ODBC data source

Use the PostgreSQL ODBC driver to connect Power BI to Tiger Cloud.

1. Open the ODBC data sources

   On your Windows machine, search for and select ODBC Data Sources.

2. Connect to your Tiger Cloud service

   1. Under User DSN, click Add.
   2. Choose PostgreSQL Unicode and click Finish.
   3. Use your connection details to configure the data source.
   4. Click Test to ensure the connection works, then click Save.

Import the data from your your Tiger Cloud service into Power BI

Establish a connection and import data from your Tiger Cloud service into Power BI:

1. Connect Power BI to your Tiger Cloud service

   1. Open Power BI, then click Get data from other sources.
   2. Search for and select ODBC, then click Connect.
   3. In Data source name (DSN), select the Tiger Cloud data source and click OK.
   4. Use your connection details to enter your User Name and Password, then click Connect.

   After connecting, Navigator displays the available tables and schemas.

2. Import your data into Power BI

   1. Select the tables to import and click Load.

      The Data pane shows your imported tables.

   2. To visualize your data and build reports, drag fields from the tables onto the canvas.

You have successfully integrated Power BI with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/tableau/ =====

Integrate Tableau and Tiger

Tableau is a popular analytics platform that helps you gain greater intelligence about your business. You can use it to visualize data stored in Tiger Cloud.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install Tableau Server or sign up for Tableau Cloud.

Add your Tiger Cloud service as a virtual connection

To connect the data in your Tiger Cloud service to Tableau:

1. Log in to Tableau

   • Tableau Cloud: sign in, then click Explore and select a project.
   • Tableau Desktop: sign in, then open a workbook.

2. Configure Tableau to connect to your Tiger Cloud service

   1. Add a new data source:

      • Tableau Cloud: click New > Virtual Connection.
      • Tableau Desktop: click Data > New Data Source.

   2. Search for and select PostgreSQL.

      For Tableau Desktop download the driver and restart Tableau.

   3. Configure the connection:

      • Server, Port, Database, Username, Password: configure using your connection details.
      • Require SSL: tick the checkbox.

3. Click Sign In and connect Tableau to your service

You have successfully integrated Tableau with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/apache-kafka/ =====

Integrate Apache Kafka with Tiger Cloud

Apache Kafka is a distributed event streaming platform used for high-performance data pipelines, streaming analytics, and data integration. Apache Kafka Connect is a tool to scalably and reliably stream data between Apache Kafka® and other data systems. Kafka Connect is an ecosystem of pre-written and maintained Kafka Producers (source connectors) and Kafka Consumers (sink connectors) for data products and platforms like databases and message brokers.

This guide explains how to set up Kafka and Kafka Connect to stream data from a Kafka topic into your Tiger Cloud service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Java8 or higher to run Apache Kafka

Install and configure Apache Kafka

To install and configure Apache Kafka:

1. Extract the Kafka binaries to a local folder

   curl https://dlcdn.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz | tar -xzf -
   cd kafka_2.13-3.9.0
   

   From now on, the folder where you extracted the Kafka binaries is called <KAFKA_HOME>.

2. Configure and run Apache Kafka

   KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
   ./bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/kraft/reconfig-server.properties
   ./bin/kafka-server-start.sh config/kraft/reconfig-server.properties
   

   Use the -daemon flag to run this process in the background.

3. Create Kafka topics

   In another Terminal window, navigate to <KAFKA_HOME>, then call kafka-topics.sh and create the following topics:

   • accounts: publishes JSON messages that are consumed by the timescale-sink connector and inserted into your Tiger Cloud service.
   • deadletter: stores messages that cause errors and that Kafka Connect workers cannot process.

   ./bin/kafka-topics.sh \
        --create \
        --topic accounts \
        --bootstrap-server localhost:9092 \
        --partitions 10
   
   ./bin/kafka-topics.sh \
        --create \
        --topic deadletter \
        --bootstrap-server localhost:9092 \
        --partitions 10
   

4. Test that your topics are working correctly

   1. Run kafka-console-producer to send messages to the accounts topic:

      bin/kafka-console-producer.sh --topic accounts --bootstrap-server localhost:9092
      

   2. Send some events. For example, type the following:

      >Tiger
      >How Cool
      

   3. In another Terminal window, navigate to <KAFKA_HOME>, then run kafka-console-consumer to consume the events you just sent:

      bin/kafka-console-consumer.sh --topic accounts --from-beginning --bootstrap-server localhost:9092
      

      You see

      Tiger
      How Cool
      

Keep these terminals open, you use them to test the integration later.

Install the sink connector to communicate with Tiger Cloud

To set up Kafka Connect server, plugins, drivers, and connectors:

1. Install the Postgres connector

   In another Terminal window, navigate to <KAFKA_HOME>, then download and configure the Postgres sink and driver.

   mkdir -p "plugins/camel-postgresql-sink-kafka-connector"
   curl https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-postgresql-sink-kafka-connector/3.21.0/camel-postgresql-sink-kafka-connector-3.21.0-package.tar.gz \
   | tar -xzf - -C "plugins/camel-postgresql-sink-kafka-connector" --strip-components=1
   curl  -H "Accept: application/zip" https://jdbc.postgresql.org/download/postgresql-42.7.5.jar -o  "plugins/camel-postgresql-sink-kafka-connector/postgresql-42.7.5.jar"
   echo "plugin.path=`pwd`/plugins/camel-postgresql-sink-kafka-connector" >> "config/connect-distributed.properties"
   echo "plugin.path=`pwd`/plugins/camel-postgresql-sink-kafka-connector" >> "config/connect-standalone.properties"
   

2. Start Kafka Connect

   export CLASSPATH=`pwd`/plugins/camel-postgresql-sink-kafka-connector/*
   ./bin/connect-standalone.sh config/connect-standalone.properties
   

   Use the -daemon flag to run this process in the background.

3. Verify Kafka Connect is running

   In yet another another Terminal window, run the following command:

   curl http://localhost:8083
   

   You see something like:

   {"version":"3.9.0","commit":"a60e31147e6b01ee","kafka_cluster_id":"J-iy4IGXTbmiALHwPZEZ-A"}
   

Create a table in your Tiger Cloud service to ingest Kafka events

To prepare your Tiger Cloud service for Kafka integration:

1. Connect to your Tiger Cloud service

2. Create a hypertable to ingest Kafka events

   CREATE TABLE accounts (
    created_at TIMESTAMPTZ DEFAULT NOW(),
    name TEXT,
    city TEXT
   ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='created_at'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

Create the Tiger Cloud sink

To create a Tiger Cloud sink in Apache Kafka:

1. Create the connection configuration

   1. In the terminal running Kafka Connect, stop the process by pressing Ctrl+C.

   2. Write the following configuration to <KAFKA_HOME>/config/timescale-standalone-sink.properties, then update the <properties> with your connection details.

      name=timescale-standalone-sink
      connector.class=org.apache.camel.kafkaconnector.postgresqlsink.CamelPostgresqlsinkSinkConnector
      errors.tolerance=all
      errors.deadletterqueue.topic.name=deadletter
      tasks.max=10
      value.converter=org.apache.kafka.connect.storage.StringConverter
      key.converter=org.apache.kafka.connect.storage.StringConverter
      topics=accounts
      camel.kamelet.postgresql-sink.databaseName=<dbname>
      camel.kamelet.postgresql-sink.username=<user>
      camel.kamelet.postgresql-sink.password=<password>
      camel.kamelet.postgresql-sink.serverName=<host>
      camel.kamelet.postgresql-sink.serverPort=<port>
      camel.kamelet.postgresql-sink.query=INSERT INTO accounts (name,city) VALUES (:#name,:#city)
      

   3. Restart Kafka Connect with the new configuration:

      export CLASSPATH=`pwd`/plugins/camel-postgresql-sink-kafka-connector/*
      ./bin/connect-standalone.sh config/connect-standalone.properties config/timescale-standalone-sink.properties
      

2. Test the connection

   To see your sink, query the /connectors route in a GET request:

   curl -X GET http://localhost:8083/connectors
   

   You see:

   #["timescale-standalone-sink"]
   

Test the integration with Tiger Cloud

To test this integration, send some messages onto the accounts topic. You can do this using the kafkacat or kcat utility.

1. In the terminal running kafka-console-producer.sh enter the following json strings

   {"name":"Lola","city":"Copacabana"}
   {"name":"Holly","city":"Miami"}
   {"name":"Jolene","city":"Tennessee"}
   {"name":"Barbara Ann ","city":"California"}
   

   Look in your terminal running kafka-console-consumer to see the messages being processed.

2. Query your Tiger Cloud service for all rows in the accounts table

   SELECT * FROM accounts;
   

   You see something like:

   created_at	name	city
   2025-02-18 13:55:05.147261+00	Lola	Copacabana
   2025-02-18 13:55:05.216673+00	Holly	Miami
   2025-02-18 13:55:05.283549+00	Jolene	Tennessee
   2025-02-18 13:55:05.35226+00	Barbara Ann	California

You have successfully integrated Apache Kafka with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/apache-airflow/ =====

Integrate Apache Airflow with Tiger

Apache Airflow® is a platform created by the community to programmatically author, schedule, and monitor workflows.

A DAG (Directed Acyclic Graph) is the core concept of Airflow, collecting Tasks together, organized with dependencies and relationships to say how they should run. You declare a DAG in a Python file in the $AIRFLOW_HOME/dags folder of your Airflow instance.

This page shows you how to use a Python connector in a DAG to integrate Apache Airflow with a Tiger Cloud service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install Python3 and pip3

• Install Apache Airflow

  Ensure that your Airflow instance has network access to Tiger Cloud.

This example DAG uses the company table you create in Optimize time-series data in hypertables

Install python connectivity libraries

To install the Python libraries required to connect to Tiger Cloud:

1. Enable Postgres connections between Airflow and Tiger Cloud

   pip install psycopg2-binary
   

2. Enable Postgres connection types in the Airflow UI

   pip install apache-airflow-providers-postgres
   

Create a connection between Airflow and your Tiger Cloud service

In your Airflow instance, securely connect to your Tiger Cloud service:

1. Run Airflow

   On your development machine, run the following command:

   airflow standalone
   

   The username and password for Airflow UI are displayed in the standalone | Login with username line in the output.

2. Add a connection from Airflow to your Tiger Cloud service

   1. In your browser, navigate to localhost:8080, then select Admin > Connections.
   2. Click + (Add a new record), then use your connection info to fill in the form. The Connection Type is Postgres.

Exchange data between Airflow and your Tiger Cloud service

To exchange data between Airflow and your Tiger Cloud service:

1. Create and execute a DAG

   To insert data in your Tiger Cloud service from Airflow:

   1. In $AIRFLOW_HOME/dags/timescale_dag.py, add the following code:

      from airflow import DAG
      from airflow.operators.python_operator import PythonOperator
      from airflow.hooks.postgres_hook import PostgresHook
      from datetime import datetime
      
      def insert_data_to_timescale():
          hook = PostgresHook(postgres_conn_id='the ID of the connenction you created')
          conn = hook.get_conn()
          cursor = conn.cursor()
          """
            This could be any query. This example inserts data into the table
            you create in:
      
            https://docs.tigerdata.com/getting-started/latest/try-key-features-timescale-products/#optimize-time-series-data-in-hypertables
           """
          cursor.execute("INSERT INTO crypto_assets (symbol, name) VALUES (%s, %s)",
           ('NEW/Asset','New Asset Name'))
          conn.commit()
          cursor.close()
          conn.close()
      
      default_args = {
          'owner': 'airflow',
          'start_date': datetime(2023, 1, 1),
          'retries': 1,
      }
      
      dag = DAG('timescale_dag', default_args=default_args, schedule_interval='@daily')
      
      insert_task = PythonOperator(
          task_id='insert_data',
          python_callable=insert_data_to_timescale,
          dag=dag,
      )
      

      This DAG uses the company table created in Create regular Postgres tables for relational data.

   2. In your browser, refresh the Airflow UI.

   3. In Search DAGS, type timescale_dag and press ENTER.

   4. Press the play icon and trigger the DAG:

2. Verify that the data appears in Tiger Cloud

   1. In Tiger Cloud Console, navigate to your service and click SQL editor.

   2. Run a query to view your data. For example: SELECT symbol, name FROM company;.

      You see the new rows inserted in the table.

You have successfully integrated Apache Airflow with Tiger Cloud and created a data pipeline.

===== PAGE: https://docs.tigerdata.com/integrations/amazon-sagemaker/ =====

Integrate Amazon Sagemaker with Tiger

Amazon SageMaker AI is a fully managed machine learning (ML) service. With SageMaker AI, data scientists and developers can quickly and confidently build, train, and deploy ML models into a production-ready hosted environment.

This page shows you how to integrate Amazon Sagemaker with a Tiger Cloud service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Set up an AWS Account

Prepare your Tiger Cloud service to ingest data from SageMaker

Create a table in Tiger Cloud service to store model predictions generated by SageMaker.

1. Connect to your Tiger Cloud service

   For Tiger Cloud, open an SQL editor in Tiger Cloud Console. For self-hosted TimescaleDB, use psql.

2. For better performance and easier real-time analytics, create a hypertable

   Hypertables are Postgres tables that automatically partition your data by time. You interact with hypertables in the same way as regular Postgres tables, but with extra features that makes managing your time-series data much easier.

   CREATE TABLE model_predictions (
     time TIMESTAMPTZ NOT NULL,
     model_name TEXT NOT NULL,
     prediction DOUBLE PRECISION NOT NULL
   ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='time'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

Create the code to inject data into a Tiger Cloud service

1. Create a SageMaker Notebook instance

   1. In Amazon SageMaker > Notebooks and Git repos, click Create Notebook instance.
   2. Follow the wizard to create a default Notebook instance.

2. Write a Notebook script that inserts data into your Tiger Cloud service

   1. When your Notebook instance is inService, click Open JupyterLab and click conda_python3.

   2. Update the following script with your connection details, then paste it in the Notebook.

      import psycopg2
      from datetime import datetime
      
      def insert_prediction(model_name, prediction, host, port, user, password, dbname):
            conn = psycopg2.connect(
               host=host,
               port=port,
               user=user,
               password=password,
               dbname=dbname
            )
            cursor = conn.cursor()
      
            query = """
               INSERT INTO model_predictions (time, model_name, prediction)
               VALUES (%s, %s, %s);
            """
      
            values = (datetime.utcnow(), model_name, prediction)
            cursor.execute(query, values)
            conn.commit()
      
            cursor.close()
            conn.close()
      
      insert_prediction(
            model_name="example_model",
            prediction=0.95,
            host="<host>",
            port="<port>",
            user="<user>",
            password="<password>",
            dbname="<dbname>"
      )
      

3. Test your SageMaker script

   1. Run the script in your SageMaker notebook.

   2. Verify that the data is in your service

      Open an SQL editor and check the sensor_data table:

      SELECT * FROM model_predictions;
      

      You see something like:

      time	model_name	prediction
      2025-02-06 16:56:34.370316+00	timescale-cloud-model	0.95

Now you can seamlessly integrate Amazon SageMaker with Tiger Cloud to store and analyze time-series data generated by machine learning models. You can also untegrate visualization tools like Grafana or Tableau with Tiger Cloud to create real-time dashboards of your model predictions.

===== PAGE: https://docs.tigerdata.com/integrations/aws/ =====

Integrate Amazon Web Services with Tiger Cloud

Amazon Web Services (AWS) is a comprehensive cloud computing platform that provides on-demand infrastructure, storage, databases, AI, analytics, and security services to help businesses build, deploy, and scale applications in the cloud.

This page explains how to integrate your AWS infrastructure with Tiger Cloud using AWS Transit Gateway.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details.

• Set up AWS Transit Gateway.

Connect your AWS infrastructure to your Tiger Cloud services

To connect to Tiger Cloud:

1. Create a Peering VPC in Tiger Cloud Console

   1. In Security > VPC, click Create a VPC:

      

   2. Choose your region and IP range, name your VPC, then click Create VPC:

      

      Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your plan in Tiger Cloud Console.

   3. Add a peering connection:

      1. In the VPC Peering column, click Add.
      2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID.

      

      1. Click Add connection.

2. Accept and configure peering connection in your AWS account

   Once your peering connection appears as Processing, you can accept and configure it in AWS:

   1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as Connected in Tiger Cloud Console.

   2. Configure at least the following in your AWS account networking:

      • Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs.
      • Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs.
      • Security groups to allow outbound TCP 5432.

3. Attach a Tiger Cloud service to the Peering VPC In Tiger Cloud Console

   1. Select the service you want to connect to the Peering VPC.
   2. Click Operations > Security > VPC.
   3. Select the VPC, then click Attach VPC.

   You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time.

You have successfully integrated your AWS infrastructure with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/grafana/ =====

Integrate Grafana and Tiger

Grafana enables you to query, visualize, alert on, and explore your metrics, logs, and traces wherever they’re stored.

This page shows you how to integrate Grafana with a Tiger Cloud service, create a dashboard and panel, then visualize geospatial data.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Install self-managed Grafana or sign up for Grafana Cloud.

Connect Grafana to Tiger Cloud

To visualize the results of your queries, enable Grafana to read the data in your service:

1. Log in to Grafana

   In your browser, log in to either:

   • Self-hosted Grafana: at http://localhost:3000/. The default credentials are admin, admin.
   • Grafana Cloud: use the URL and credentials you set when you created your account.

2. Add your service as a data source

   1. Open Connections > Data sources, then click Add new data source.

   2. Select PostgreSQL from the list.

   3. Configure the connection:

      • Host URL, Database name, Username, and Password

        Configure using your connection details. Host URL is in the format <host>:<port>.

      • TLS/SSL Mode: select require.

      • PostgreSQL options: enable TimescaleDB.

      • Leave the default setting for all other fields.

   4. Click Save & test.

   Grafana checks that your details are set correctly.

Create a Grafana dashboard and panel

Grafana is organized into dashboards and panels. A dashboard represents a view into the performance of a system, and each dashboard consists of one or more panels, which represent information about a specific metric related to that system.

To create a new dashboard:

1. On the Dashboards page, click New and select New dashboard

2. Click Add visualization

3. Select the data source

   Select your service from the list of pre-configured data sources or configure a new one.

4. Configure your panel

   Select the visualization type. The type defines specific fields to configure in addition to standard ones, such as the panel name.

5. Run your queries

   You can edit the queries directly or use the built-in query editor. If you are visualizing time-series data, select Time series in the Format drop-down.

6. Click Save dashboard

   You now have a dashboard with one panel. Add more panels to a dashboard by clicking Add at the top right and selecting Visualization from the drop-down.

Use the time filter function

Grafana time-series panels include a time filter:

1. Call _timefilter() to link the user interface construct in a Grafana panel with the query

   For example, to set the pickup_datetime column as the filtering range for your visualizations:

   SELECT
     --1--
     time_bucket('1 day', pickup_datetime) AS "time",
     --2--
     COUNT(*)
   FROM rides
   WHERE _timeFilter(pickup_datetime)
   

2. Group your visualizations and order the results by time buckets

   In this case, the GROUP BY and ORDER BY statements reference time.

   For example:

   SELECT
     --1--
     time_bucket('1 day', pickup_datetime) AS time,
     --2--
     COUNT(*)
   FROM rides
   WHERE _timeFilter(pickup_datetime)
   GROUP BY time
   ORDER BY time
   

   When you visualize this query in Grafana, you see this:

   

   You can adjust the time_bucket function and compare the graphs:

   SELECT
     --1--
     time_bucket('5m', pickup_datetime) AS time,
     --2--
     COUNT(*)
   FROM rides
   WHERE _timeFilter(pickup_datetime)
   GROUP BY time
   ORDER BY time
   

   When you visualize this query, it looks like this:

   

Visualize geospatial data

Grafana includes a Geomap panel so you can see geospatial data overlaid on a map. This can be helpful to understand how data changes based on its location.

This section visualizes taxi rides in Manhattan, where the distance traveled was greater than 5 miles. It uses the same query as the NYC Taxi Cab tutorial as a starting point.

1. Add a geospatial visualization

   1. In your Grafana dashboard, click Add > Visualization.

   2. Select Geomap in the visualization type drop-down at the top right.

2. Configure the data format

   1. In the Queries tab below, select your data source.

   2. In the Format drop-down, select Table.

   3. In the mode switcher, toggle Code and enter the query, then click Run.

      For example:

      SELECT time_bucket('5m', rides.pickup_datetime) AS time,
             rides.trip_distance AS value,
             rides.pickup_latitude AS latitude,
             rides.pickup_longitude AS longitude
      FROM rides
      WHERE rides.trip_distance > 5
      GROUP BY time,
               rides.trip_distance,
               rides.pickup_latitude,
               rides.pickup_longitude
      ORDER BY time
      LIMIT 500;
      

3. Customize the Geomap settings

   With default settings, the visualization uses green circles of the fixed size. Configure at least the following for a more representative view:

   • Map layers > Styles > Size > value.

     This changes the size of the circle depending on the value, with bigger circles representing bigger values.

   • Map layers > Styles > Color > value.

   • Thresholds > Add threshold.

     Add thresholds for 7 and 10, to mark rides over 7 and 10 miles in different colors, respectively.

   You now have a visualization that looks like this:

   

===== PAGE: https://docs.tigerdata.com/integrations/dbeaver/ =====

Integrate DBeaver with Tiger

DBeaver is a free cross-platform database tool for developers, database administrators, analysts, and everyone working with data. DBeaver provides an SQL editor, administration features, data and schema migration, and the ability to monitor database connection sessions.

This page explains how to integrate DBeaver with your Tiger Cloud service.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Download and install DBeaver.

Connect DBeaver to your Tiger Cloud service

To connect to Tiger Cloud:

1. Start DBeaver

2. In the toolbar, click the plug+ icon

3. In Connect to a database search for TimescaleDB

4. Select TimescaleDB, then click Next

5. Configure the connection

   Use your connection details to add your connection settings.

   If you configured your service to connect using a stricter SSL mode, in the SSL tab check Use SSL and set SSL mode to the configured mode. Then, in the CA Certificate field type the location of the SSL root CA certificate.

6. Click Test Connection. When the connection is successful, click Finish

   Your connection is listed in the Database Navigator.

You have successfully integrated DBeaver with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/qstudio/ =====

Integrate qStudio with Tiger

qStudio is a modern free SQL editor that provides syntax highlighting, code-completion, excel export, charting, and much more. You can use it to run queries, browse tables, and create charts for your Tiger Cloud service.

This page explains how to integrate qStudio with Tiger Cloud.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

• Download and install qStudio.

Connect qStudio to your Tiger Cloud service

To connect to Tiger Cloud:

1. Start qStudio

2. Click Server > Add Server

3. Configure the connection

   • For Server Type, select Postgres.
   • For Connect By, select Host.
   • For Host, Port, Database, Username, and Password, use your connection details.

1. Click Test

   qStudio indicates whether the connection works.

2. Click Add

   The server is listed in the Server Tree.

You have successfully integrated qStudio with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/integrations/microsoft-azure/ =====

Integrate Microsoft Azure with Tiger Cloud

Microsoft Azure is a cloud computing platform and services suite, offering infrastructure, AI, analytics, security, and developer tools to help businesses build, deploy, and manage applications.

This page explains how to integrate your Microsoft Azure infrastructure with Tiger Cloud using AWS Transit Gateway.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details.

• Set up AWS Transit Gateway.

Connect your Microsoft Azure infrastructure to your Tiger Cloud services

To connect to Tiger Cloud:

1. Connect your infrastructure to AWS Transit Gateway

   Establish connectivity between Azure and AWS. See the AWS architectural documentation for details.

2. Create a Peering VPC in Tiger Cloud Console

   1. In Security > VPC, click Create a VPC:

      

   2. Choose your region and IP range, name your VPC, then click Create VPC:

      

      Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your plan in Tiger Cloud Console.

   3. Add a peering connection:

      1. In the VPC Peering column, click Add.
      2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID.

      

      1. Click Add connection.

3. Accept and configure peering connection in your AWS account

   Once your peering connection appears as Processing, you can accept and configure it in AWS:

   1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as Connected in Tiger Cloud Console.

   2. Configure at least the following in your AWS account networking:

      • Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs.
      • Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs.
      • Security groups to allow outbound TCP 5432.

4. Attach a Tiger Cloud service to the Peering VPC In Tiger Cloud Console

   1. Select the service you want to connect to the Peering VPC.
   2. Click Operations > Security > VPC.
   3. Select the VPC, then click Attach VPC.

   You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time.

You have successfully integrated your Microsoft Azure infrastructure with Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/migrate/index/ =====

Sync, import, and migrate your data to Tiger

In Tiger Cloud, you can easily add and sync data to your service from other sources.

This includes:

• Sync or stream directly, so data from another source is continuously updated in your service.
• Import individual files using Tiger Cloud Console or the command line.
• Migrate data from other databases.

Sync from Postgres or S3

Tiger Cloud provides source connectors for Postgres, S3, and Kafka. You use them to synchronize all or some of your data to your Tiger Cloud service in real time. You run the connectors continuously, using your data as a primary database and your Tiger Cloud service as a logical replica. This enables you to leverage Tiger Cloud’s real-time analytics capabilities on your replica data.

Connector options	Downtime requirements
Source Postgres connector	None
Source S3 connector	None
Source Kafka connector	None

Import individual files

You can import individual files using Console, from your local machine or S3. This includes CSV, Parquet, TXT, and MD files. Alternatively, import files using the terminal.

Migrate your data

Depending on the amount of data you need to migrate, and the amount of downtime you can afford, Tiger Data offers the following migration options:

Migration strategy	Use when	Downtime requirements
Migrate with downtime	Use pg_dump and pg_restore to migrate when you can afford downtime.	Some downtime
Live migration	Simplified end-to-end migration with almost zero downtime.	Minimal downtime
Dual-write and backfill	Append-only data, heavy insert workload (~20,000 inserts per second) when modifying your ingestion pipeline is not an issue.	Minimal downtime

All strategies work to migrate from Postgres, TimescaleDB, AWS RDS, and Managed Service for TimescaleDB. Migration assistance is included with Tiger Cloud support. If you encounter any difficulties while migrating your data, consult the troubleshooting page, open a support request, or take your issue to the #migration channel in the community slack, the developers of this migration method are there to help.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

If you're migrating your data from another source database type, best practice is export the data from your source database as a CSV file, then import to your Tiger Cloud service using timescaledb-parallel-copy.

===== PAGE: https://docs.tigerdata.com/migrate/dual-write-and-backfill/ =====

Low-downtime migrations with dual-write and backfill

Dual-write and backfill is a migration strategy to move a large amount of time-series data (100 GB-10 TB+) with low downtime (on the order of minutes of downtime). It is significantly more complicated to execute than a migration with downtime using pg_dump/restore, and has some prerequisites on the data ingest patterns of your application, so it may not be universally applicable.

Dual-write and backfill can be used for any source database type, as long as it can provide data in csv format. It can be used to move data from a PostgresSQL source, and from TimescaleDB to TimescaleDB.

Dual-write and backfill works well when:

1. The bulk of the (on-disk) data is in time-series tables.
2. Writes by the application do not reference historical time-series data.
3. Writes to time-series data are append-only.
4. No UPDATE or DELETE queries will be run on time-series data in the source database during the migration process (or if they are, it happens in a controlled manner, such that it's possible to either ignore, or re-backfill).
5. Either the relational (non-time-series) data is small enough to be copied from source to target in an acceptable amount of time for this to be done with downtime, or the relational data can be copied asynchronously while the application continues to run (that is, changes relatively infrequently).

Prerequisites

Best practice is to use an Ubuntu EC2 instance hosted in the same region as your Tiger Cloud service to move data. That is, the machine you run the commands on to move your data from your source database to your target Tiger Cloud service.

Before you move your data:

• Create a target Tiger Cloud service.

  Each Tiger Cloud service has a single Postgres instance that supports the most popular extensions. Tiger Cloud services do not support tablespaces, and there is no superuser associated with a service. Best practice is to create a Tiger Cloud service with at least 8 CPUs for a smoother experience. A higher-spec instance can significantly reduce the overall migration window.

• To ensure that maintenance does not run while migration is in progress, best practice is to adjust the maintenance window.

Migrate to Tiger Cloud

To move your data from a self-hosted database to a Tiger Cloud service:

===== PAGE: https://docs.tigerdata.com/getting-started/index/ =====

Get started with Tiger Data

A Tiger Cloud service is a single optimised Postgres instance extended with innovations in the database engine such as TimescaleDB, in a cloud infrastructure that delivers speed without sacrifice.

A Tiger Cloud service is a radically faster Postgres database for transactional, analytical, and agentic workloads at scale.

It’s not a fork. It’s not a wrapper. It is Postgres—extended with innovations in the database engine and cloud infrastructure to deliver speed (10-1000x faster at scale) without sacrifice. A Tiger Cloud service brings together the familiarity and reliability of Postgres with the performance of purpose-built engines.

Tiger Cloud is the fastest Postgres cloud. It includes everything you need to run Postgres in a production-reliable, scalable, observable environment.

This section shows you how to:

• Create and connect to a Tiger Cloud service: choose the capabilities that match your business and engineering needs on Tiger Data's cloud-based Postgres platform.
• Try the main features in Tiger Data products: rapidly implement the features in Tiger Cloud that enable you to ingest and query data faster while keeping the costs low.
• Start coding with Tiger Data: quickly integrate Tiger Cloud and TimescaleDB into your apps using your favorite programming language.
• Run queries from Tiger Cloud Console: securely interact with your data in the Tiger Cloud Console UI.

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/ai/index/ =====

Integrate AI with Tiger Data

You can build and deploy AI Assistants that understand, analyze, and act on your organizational data using Tiger Data. Whether you're building semantic search applications, recommendation systems, or intelligent agents that answer complex business questions, Tiger Data provides the tools and infrastructure you need.

Tiger Data's AI ecosystem combines Postgres with advanced vector capabilities, intelligent agents, and seamless integrations. Your AI Assistants can:

• Access organizational knowledge from Slack, GitHub, Linear, and other data sources
• Understand context using advanced vector search and embeddings across large datasets
• Execute tasks, generate reports, and interact with your Tiger Cloud services through natural language
• Scale reliably with enterprise-grade performance for concurrent conversations

Tiger Eon for complete organizational AI

Tiger Eon automatically integrates Tiger Agents for Work with your organizational data. You can:

• Get instant access to company knowledge from Slack, GitHub, and Linear
• Process data in real-time as conversations and updates happen
• Store data efficiently with time-series partitioning and compression
• Deploy quickly with Docker and an interactive setup wizard

Use Eon when you want to unlock knowledge from your communication and development tools.

Tiger Agents for Work for enterprise Slack AI

Tiger Agents for Work provides enterprise-grade Slack-native AI agents. You get:

• Durable event handling with Postgres-backed processing
• Horizontal scalability across multiple Tiger Agent instances
• Flexibility to choose AI models and customize prompts
• Integration with specialized data sources through MCP servers
• Complete observability and monitoring with Logfire

Use Tiger Agents for Work when you need reliable, customizable AI agents for high-volume conversations.

Tiger MCP Server for direct AI Assistant integration

The Tiger Model Context Protocol Server integrates directly with popular AI Assistants. You can:

• Work with Claude Code, Cursor, VS Code, and other editors
• Manage services and optimize queries through natural language
• Access comprehensive Tiger Data documentation during development
• Use secure authentication and access control

Use the Tiger MCP Server when you want to manage Tiger Data resources from your AI Assistant.

pgvectorscale and️ pgvector

Pgvector is a popular open source extension for vector storage and similarity search in Postgres and pgvectorscale adds advanced indexing capabilities to pgvector. pgai on Tiger Cloud offers both extensions so you can use all the capabilities already available in pgvector (like HNSW and ivfflat indexes) and also make use of the StreamingDiskANN index in pgvectorscale to speed up vector search.

This makes it easy to migrate your existing pgvector deployment and take advantage of the additional performance features in pgvectorscale. You also have the flexibility to create different index types suited to your needs. See the vector search indexing section for more information.

Embeddings offer a way to represent the semantic essence of data and to allow comparing data according to how closely related it is in terms of meaning. In the database context, this is extremely powerful: think of this as full-text search on steroids. Vector databases allow storing embeddings associated with data and then searching for embeddings that are similar to a given query.

• Semantic search: transcend the limitations of traditional keyword-driven search methods by creating systems that understand the intent and contextual meaning of a query, thereby returning more relevant results. Semantic search doesn't just seek exact word matches; it grasps the deeper intent behind a user's query. The result? Even if search terms differ in phrasing, relevant results are surfaced. Taking advantage of hybrid search, which marries lexical and semantic search methodologies, offers users a search experience that's both rich and accurate. It's not just about finding direct matches anymore; it's about tapping into contextually and conceptually similar content to meet user needs.

• Recommendation systems: imagine a user who has shown interest in several articles on a singular topic. With embeddings, the recommendation engine can delve deep into the semantic essence of those articles, surfacing other database items that resonate with the same theme. Recommendations, thus, move beyond just the superficial layers like tags or categories and dive into the very heart of the content.

• Retrieval augmented generation (RAG): supercharge generative AI by providing additional context to Large Language Models (LLMs) like OpenAI's GPT-4, Anthropic's Claude 2, and open source modes like Llama 2. When a user poses a query, relevant database content is fetched and used to supplement the query as additional information for the LLM. This helps reduce LLM hallucinations, as it ensures the model's output is more grounded in specific and relevant information, even if it wasn't part of the model's original training data.

• Clustering: embeddings also offer a robust solution for clustering data. Transforming data into these vectorized forms allows for nuanced comparisons between data points in a high-dimensional space. Through algorithms like K-means or hierarchical clustering, data can be categorized into semantic categories, offering insights that surface-level attributes might miss. This surfaces inherent data patterns, enriching both exploration and decision-making processes.

Vector similarity search: How does it work

On a high level, embeddings help a database to look for data that is similar to a given piece of information (similarity search). This process includes a few steps:

• First, embeddings are created for data and inserted into the database. This can take place either in an application or in the database itself.
• Second, when a user has a search query (for example, a question in chat), that query is then transformed into an embedding.
• Third, the database takes the query embedding and searches for the closest matching (most similar) embeddings it has stored.

Under the hood, embeddings are represented as a vector (a list of numbers) that capture the essence of the data. To determine the similarity of two pieces of data, the database uses mathematical operations on vectors to get a distance measure (commonly Euclidean or cosine distance). During a search, the database should return those stored items where the distance between the query embedding and the stored embedding is as small as possible, suggesting the items are most similar.

Embedding models

pgai on Tiger Cloud works with the most popular embedding models that have output vectors of 2,000 dimensions or less.:

• OpenAI embedding models: text-embedding-ada-002 is OpenAI's recommended embedding generation model.
• Cohere representation models: Cohere offers many models that can be used to generate embeddings from text in English or multiple languages.

And here are some popular choices for image embeddings:

• OpenAI CLIP: Useful for applications involving text and images.
• VGG
• Vision Transformer (ViT)

===== PAGE: https://docs.tigerdata.com/api/hyperfunctions/ =====

Hyperfunctions

Hyperfunctions in TimescaleDB are a specialized set of functions that allow you to analyze time-series data. You can use hyperfunctions to analyze anything you have stored as time-series data, including IoT devices, IT systems, marketing analytics, user behavior, financial metrics, and cryptocurrency.

Some hyperfunctions are included by default in TimescaleDB. For additional hyperfunctions, you need to install the TimescaleDB Toolkit Postgres extension.

For more information, see the hyperfunctions documentation.

===== PAGE: https://docs.tigerdata.com/api/time-weighted-averages/ =====

Time-weighted average functions

This section contains functions related to time-weighted averages and integrals. Time weighted averages and integrals are commonly used in cases where a time series is not evenly sampled, so a traditional average gives misleading results. For more information about these functions, see the hyperfunctions documentation.

Some hyperfunctions are included in the default TimescaleDB product. For additional hyperfunctions, you need to install the TimescaleDB Toolkit Postgres extension.

===== PAGE: https://docs.tigerdata.com/api/counter_aggs/ =====

Counter and gauge aggregation

This section contains functions related to counter and gauge aggregation. Counter aggregation functions are used to accumulate monotonically increasing data by treating any decrements as resets. Gauge aggregates are similar, but are used to track data which can decrease as well as increase. For more information about counter aggregation functions, see the hyperfunctions documentation.

Some hyperfunctions are included in the default TimescaleDB product. For additional hyperfunctions, you need to install the TimescaleDB Toolkit Postgres extension.

All accessors can be used with CounterSummary, and all but num_resets with GaugeSummary.

===== PAGE: https://docs.tigerdata.com/api/gapfilling-interpolation/ =====

Gapfilling and interpolation

This section contains functions related to gapfilling and interpolation. You can use a gapfilling function to create additional rows of data in any gaps, ensuring that the returned rows are in chronological order, and contiguous. For more information about gapfilling and interpolation functions, see the hyperfunctions documentation.

Some hyperfunctions are included in the default TimescaleDB product. For additional hyperfunctions, you need to install the TimescaleDB Toolkit Postgres extension.

===== PAGE: https://docs.tigerdata.com/api/state-aggregates/ =====

State aggregates

This section includes functions used to measure the time spent in a relatively small number of states.

For these hyperfunctions, you need to install the TimescaleDB Toolkit Postgres extension.

Notes on compact_state_agg and state_agg

state_agg supports all hyperfunctions that operate on CompactStateAggs, in addition to some additional functions that need a full state timeline.

All compact_state_agg and state_agg hyperfunctions support both string (TEXT) and integer (BIGINT) states. You can't mix different types of states within a single aggregate. Integer states are useful when the state value is a foreign key representing a row in another table that stores all possible states.

Hyperfunctions

===== PAGE: https://docs.tigerdata.com/api/index/ =====

TimescaleDB API reference

TimescaleDB provides many SQL functions and views to help you interact with and manage your data. See a full list below or search by keyword to find reference documentation for a specific API.

APIReference

Refer to the installation documentation for detailed setup instructions.

===== PAGE: https://docs.tigerdata.com/api/rollup/ =====

rollup()

Combines multiple OpenHighLowClose aggregates. Using rollup, you can reaggregate a continuous aggregate into larger time buckets.

rollup(
    ohlc OpenHighLowClose
) RETURNS OpenHighLowClose

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

Required arguments

Name	Type	Description
ohlc	OpenHighLowClose	The aggregate to roll up

Returns

Column	Type	Description
ohlc	OpenHighLowClose	A new aggregate, which is an object storing (timestamp, value) pairs for each of the opening, high, low, and closing prices.

Sample usage

Roll up your by-minute continuous aggregate into hourly buckets and return the OHLC prices:

SELECT time_bucket('1 hour'::interval, ts) AS hourly_bucket,
    symbol,
    toolkit_experimental.open(toolkit_experimental.rollup(ohlc)),
    toolkit_experimental.high(toolkit_experimental.rollup(ohlc)),
    toolkit_experimental.low(toolkit_experimental.rollup(ohlc)),
    toolkit_experimental.close(toolkit_experimental.rollup(ohlc)),
  FROM ohlc
 GROUP BY hourly_bucket, symbol
;

Roll up your by-minute continuous aggregate into a daily aggregate and return the OHLC prices:

WITH ohlc AS (
    SELECT time_bucket('1 minute'::interval, ts) AS minute_bucket,
      symbol,
      toolkit_experimental.ohlc(ts, price)
    FROM crypto_ticks
    GROUP BY minute_bucket, symbol
)
SELECT time_bucket('1 day'::interval , bucket) AS daily_bucket
  symbol,
  toolkit_experimental.open(toolkit_experimental.rollup(ohlc)),
  toolkit_experimental.high(toolkit_experimental.rollup(ohlc)),
  toolkit_experimental.low(toolkit_experimental.rollup(ohlc)),
  toolkit_experimental.close(toolkit_experimental.rollup(ohlc))
FROM ohlc
GROUP BY daily_bucket, symbol
;

===== PAGE: https://docs.tigerdata.com/api/to_epoch/ =====

to_epoch()

Given a timestamptz, returns the number of seconds since January 1, 1970 (the Unix epoch).

Required arguments

Name	Type	Description
date	TIMESTAMPTZ	Timestamp to use to calculate epoch

Sample usage

Convert a date to a Unix epoch time:

SELECT to_epoch('2021-01-01 00:00:00+03'::timestamptz);

The output looks like this:

  to_epoch
------------
 1609448400

===== PAGE: https://docs.tigerdata.com/tutorials/ingest-real-time-websocket-data/ =====

Ingest real-time financial data using WebSocket

This tutorial shows you how to ingest real-time time-series data into TimescaleDB using a websocket connection. The tutorial sets up a data pipeline to ingest real-time data from our data partner, Twelve Data. Twelve Data provides a number of different financial APIs, including stock, cryptocurrencies, foreign exchanges, and ETFs. It also supports websocket connections in case you want to update your database frequently. With websockets, you need to connect to the server, subscribe to symbols, and you can start receiving data in real-time during market hours.

When you complete this tutorial, you'll have a data pipeline set up that ingests real-time financial data into your Tiger Cloud.

This tutorial uses Python and the API wrapper library provided by Twelve Data.

Prerequisites

Before you begin, make sure you have:

• Signed up for a free Tiger Data account.
• Downloaded the file that contains your Tiger Cloud service credentials such as <HOST>, <PORT>, and <PASSWORD>. Alternatively, you can find these details in the Connection Info section for your service.
• Installed Python 3
• Signed up for Twelve Data. The free tier is perfect for this tutorial.
• Made a note of your Twelve Data API key.

When you connect to the Twelve Data API through a websocket, you create a persistent connection between your computer and the websocket server. You set up a Python environment, and pass two arguments to create a websocket object and establish the connection.

Set up a new Python environment

Create a new Python virtual environment for this project and activate it. All the packages you need to complete for this tutorial are installed in this environment.

Setting up a new Python environment

1. Create and activate a Python virtual environment:

   virtualenv env
   source env/bin/activate
   

2. Install the Twelve Data Python wrapper library with websocket support. This library allows you to make requests to the API and maintain a stable websocket connection.

   pip install twelvedata websocket-client
   

3. Install Psycopg2 so that you can connect the TimescaleDB from your Python script:

   pip install psycopg2-binary
   

Create the websocket connection

A persistent connection between your computer and the websocket server is used to receive data for as long as the connection is maintained. You need to pass two arguments to create a websocket object and establish connection.

Websocket arguments

• on_event

  This argument needs to be a function that is invoked whenever there's a new data record is received from the websocket:

  def on_event(event):
      print(event) # prints out the data record (dictionary)
  

  This is where you want to implement the ingestion logic so whenever there's new data available you insert it into the database.

• symbols

  This argument needs to be a list of stock ticker symbols (for example, MSFT) or crypto trading pairs (for example, BTC/USD). When using a websocket connection you always need to subscribe to the events you want to receive. You can do this by using the symbols argument or if your connection is already created you can also use the subscribe() function to get data for additional symbols.

Connecting to the websocket server

1. Create a new Python file called websocket_test.py and connect to the Twelve Data servers using the <YOUR_API_KEY>:

      import time
      from twelvedata import TDClient
   
       messages_history = []
   
       def on_event(event):
        print(event) # prints out the data record (dictionary)
        messages_history.append(event)
   
      td = TDClient(apikey="<YOUR_API_KEY>")
      ws = td.websocket(symbols=["BTC/USD", "ETH/USD"], on_event=on_event)
      ws.subscribe(['ETH/BTC', 'AAPL'])
      ws.connect()
      while True:
      print('messages received: ', len(messages_history))
      ws.heartbeat()
      time.sleep(10)
   

2. Run the Python script:

   python websocket_test.py
   

3. When you run the script, you receive a response from the server about the status of your connection:

   {'event': 'subscribe-status',
    'status': 'ok',
    'success': [
           {'symbol': 'BTC/USD', 'exchange': 'Coinbase Pro', 'mic_code': 'Coinbase Pro', 'country': '', 'type': 'Digital Currency'},
           {'symbol': 'ETH/USD', 'exchange': 'Huobi', 'mic_code': 'Huobi', 'country': '', 'type': 'Digital Currency'}
       ],
    'fails': None
   }
   

   When you have established a connection to the websocket server, wait a few seconds, and you can see data records, like this:

   {'event': 'price', 'symbol': 'BTC/USD', 'currency_base': 'Bitcoin', 'currency_quote': 'US Dollar', 'exchange': 'Coinbase Pro', 'type': 'Digital Currency', 'timestamp': 1652438893, 'price': 30361.2, 'bid': 30361.2, 'ask': 30361.2, 'day_volume': 49153}
   {'event': 'price', 'symbol': 'BTC/USD', 'currency_base': 'Bitcoin', 'currency_quote': 'US Dollar', 'exchange': 'Coinbase Pro', 'type': 'Digital Currency', 'timestamp': 1652438896, 'price': 30380.6, 'bid': 30380.6, 'ask': 30380.6, 'day_volume': 49157}
   {'event': 'heartbeat', 'status': 'ok'}
   {'event': 'price', 'symbol': 'ETH/USD', 'currency_base': 'Ethereum', 'currency_quote': 'US Dollar', 'exchange': 'Huobi', 'type': 'Digital Currency', 'timestamp': 1652438899, 'price': 2089.07, 'bid': 2089.02, 'ask': 2089.03, 'day_volume': 193818}
   {'event': 'price', 'symbol': 'BTC/USD', 'currency_base': 'Bitcoin', 'currency_quote': 'US Dollar', 'exchange': 'Coinbase Pro', 'type': 'Digital Currency', 'timestamp': 1652438900, 'price': 30346.0, 'bid': 30346.0, 'ask': 30346.0, 'day_volume': 49167}
   

   Each price event gives you multiple data points about the given trading pair such as the name of the exchange, and the current price. You can also occasionally see heartbeat events in the response; these events signal the health of the connection over time. At this point the websocket connection is working successfully to pass data.

To ingest the data into your Tiger Cloud service, you need to implement the on_event function.

After the websocket connection is set up, you can use the on_event function to ingest data into the database. This is a data pipeline that ingests real-time financial data into your Tiger Cloud service.

Stock trades are ingested in real-time Monday through Friday, typically during normal trading hours of the New York Stock Exchange (9:30 AM to 4:00 PM EST).

Optimize time-series data in hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Create a hypertable to store the real-time stock data

   CREATE TABLE stocks_real_time (
     time TIMESTAMPTZ NOT NULL,
     symbol TEXT NOT NULL,
     price DOUBLE PRECISION NULL,
     day_volume INT NULL
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

3. Create an index to support efficient queries

   Index on the symbol and time columns:

   CREATE INDEX ix_symbol_time ON stocks_real_time (symbol, time DESC);
   

Create standard Postgres tables for relational data

When you have other relational data that enhances your time-series data, you can create standard Postgres tables just as you would normally. For this dataset, there is one other table of data called company.

1. Add a table to store the company data

   CREATE TABLE company (
     symbol TEXT NOT NULL,
     name TEXT NOT NULL
   );
   

You now have two tables in your Tiger Cloud service. One hypertable named stocks_real_time, and one regular Postgres table named company.

When you ingest data into a transactional database like Timescale, it is more efficient to insert data in batches rather than inserting data row-by-row. Using one transaction to insert multiple rows can significantly increase the overall ingest capacity and speed of your Tiger Cloud service.

Batching in memory

A common practice to implement batching is to store new records in memory first, then after the batch reaches a certain size, insert all the records from memory into the database in one transaction. The perfect batch size isn't universal, but you can experiment with different batch sizes (for example, 100, 1000, 10000, and so on) and see which one fits your use case better. Using batching is a fairly common pattern when ingesting data into TimescaleDB from Kafka, Kinesis, or websocket connections.

You can implement a batching solution in Python with Psycopg2. You can implement the ingestion logic within the on_event function that you can then pass over to the websocket object.

This function needs to:

1. Check if the item is a data item, and not websocket metadata.
2. Adjust the data so that it fits the database schema, including the data types, and order of columns.
3. Add it to the in-memory batch, which is a list in Python.
4. If the batch reaches a certain size, insert the data, and reset or empty the list.

Ingesting data in real-time

1. Update the Python script that prints out the current batch size, so you can follow when data gets ingested from memory into your database. Use the <HOST>, <PASSWORD>, and <PORT> details for the Tiger Cloud service where you want to ingest the data and your API key from Twelve Data:

   import time
   import psycopg2
   
   from twelvedata import TDClient
   from psycopg2.extras import execute_values
   from datetime import datetime
   
   class WebsocketPipeline():
       DB_TABLE = "stocks_real_time"
   
       DB_COLUMNS=["time", "symbol", "price", "day_volume"]
   
       MAX_BATCH_SIZE=100
   
       def __init__(self, conn):
           """Connect to the Twelve Data web socket server and stream
           data into the database.
   
           Args:
               conn: psycopg2 connection object
           """
           self.conn = conn
           self.current_batch = []
           self.insert_counter = 0
   
       def _insert_values(self, data):
           if self.conn is not None:
               cursor = self.conn.cursor()
               sql = f"""
               INSERT INTO {self.DB_TABLE} ({','.join(self.DB_COLUMNS)})
               VALUES %s;"""
               execute_values(cursor, sql, data)
               self.conn.commit()
   
       def _on_event(self, event):
           """This function gets called whenever there's a new data record coming
           back from the server.
   
           Args:
               event (dict): data record
           """
           if event["event"] == "price":
               timestamp = datetime.utcfromtimestamp(event["timestamp"])
               data = (timestamp, event["symbol"], event["price"], event.get("day_volume"))
   
               self.current_batch.append(data)
               print(f"Current batch size: {len(self.current_batch)}")
   
               if len(self.current_batch) == self.MAX_BATCH_SIZE:
                   self._insert_values(self.current_batch)
                   self.insert_counter += 1
                   print(f"Batch insert #{self.insert_counter}")
                   self.current_batch = []
           def start(self, symbols):
               """Connect to the web socket server and start streaming real-time data
               into the database.
   
               Args:
                   symbols (list of symbols): List of stock/crypto symbols
               """
               td = TDClient(apikey="<YOUR_API_KEY")
               ws = td.websocket(on_event=self._on_event)
               ws.subscribe(symbols)
               ws.connect()
               while True:
                  ws.heartbeat()
                  time.sleep(10)
       onn = psycopg2.connect(database="tsdb",
                           host="<HOST>",
                           user="tsdbadmin",
                           password="<PASSWORD>",
                           port="<PORT>")
   
       symbols = ["BTC/USD", "ETH/USD", "MSFT", "AAPL"]
       websocket = WebsocketPipeline(conn)
       websocket.start(symbols=symbols)
       ```
   
   

2. Run the script:

   python websocket_test.py
   

You can even create separate Python scripts to start multiple websocket connections for different types of symbols, for example, one for stock, and another one for cryptocurrency prices.

Troubleshooting

If you see an error message similar to this:

2022-05-13 18:51:41,976 - ws-twelvedata - ERROR - TDWebSocket ERROR: Handshake status 200 OK

Then check that you use a proper API key received from Twelve Data.

To look at OHLCV values, the most effective way is to create a continuous aggregate. You can create a continuous aggregate to aggregate data for each hour, then set the aggregate to refresh every hour, and aggregate the last two hours' worth of data.

Creating a continuous aggregate

1. Connect to the Tiger Cloud service tsdb that contains the Twelve Data stocks dataset.

2. At the psql prompt, create the continuous aggregate to aggregate data every minute:

   CREATE MATERIALIZED VIEW one_hour_candle
   WITH (timescaledb.continuous) AS
       SELECT
           time_bucket('1 hour', time) AS bucket,
           symbol,
           FIRST(price, time) AS "open",
           MAX(price) AS high,
           MIN(price) AS low,
           LAST(price, time) AS "close",
           LAST(day_volume, time) AS day_volume
       FROM stocks_real_time
       GROUP BY bucket, symbol;
   

   When you create the continuous aggregate, it refreshes by default.

3. Set a refresh policy to update the continuous aggregate every hour, if there is new data available in the hypertable for the last two hours:

   SELECT add_continuous_aggregate_policy('one_hour_candle',
       start_offset => INTERVAL '3 hours',
       end_offset => INTERVAL '1 hour',
       schedule_interval => INTERVAL '1 hour');
   

Query the continuous aggregate

When you have your continuous aggregate set up, you can query it to get the OHLCV values.

Querying the continuous aggregate

1. Connect to the Tiger Cloud service that contains the Twelve Data stocks dataset.

2. At the psql prompt, use this query to select all AAPL OHLCV data for the past 5 hours, by time bucket:

   SELECT * FROM one_hour_candle
   WHERE symbol = 'AAPL' AND bucket >= NOW() - INTERVAL '5 hours'
   ORDER BY bucket;
   

   The result of the query looks like this:

            bucket         | symbol  |  open   |  high   |   low   |  close  | day_volume
   ------------------------+---------+---------+---------+---------+---------+------------
    2023-05-30 08:00:00+00 | AAPL   | 176.31 | 176.31 |    176 | 176.01 |
    2023-05-30 08:01:00+00 | AAPL   | 176.27 | 176.27 | 176.02 |  176.2 |
    2023-05-30 08:06:00+00 | AAPL   | 176.03 | 176.04 | 175.95 |    176 |
    2023-05-30 08:07:00+00 | AAPL   | 175.95 |    176 | 175.82 | 175.91 |
    2023-05-30 08:08:00+00 | AAPL   | 175.92 | 176.02 |  175.8 | 176.02 |
    2023-05-30 08:09:00+00 | AAPL   | 176.02 | 176.02 |  175.9 | 175.98 |
    2023-05-30 08:10:00+00 | AAPL   | 175.98 | 175.98 | 175.94 | 175.94 |
    2023-05-30 08:11:00+00 | AAPL   | 175.94 | 175.94 | 175.91 | 175.91 |
    2023-05-30 08:12:00+00 | AAPL   |  175.9 | 175.94 |  175.9 | 175.94 |
   

You can visualize the OHLCV data that you created using the queries in Grafana.

Graph OHLCV data

When you have extracted the raw OHLCV data, you can use it to graph the result in a candlestick chart, using Grafana. To do this, you need to have Grafana set up to connect to your self-hosted TimescaleDB instance.

Graphing OHLCV data

1. Ensure you have Grafana installed, and you are using the TimescaleDB database that contains the Twelve Data dataset set up as a data source.

2. In Grafana, from the Dashboards menu, click New Dashboard. In the New Dashboard page, click Add a new panel.

3. In the Visualizations menu in the top right corner, select Candlestick from the list. Ensure you have set the Twelve Data dataset as your data source.

4. Click Edit SQL and paste in the query you used to get the OHLCV values.

5. In the Format as section, select Table.

6. Adjust elements of the table as required, and click Apply to save your graph to the dashboard.

   

===== PAGE: https://docs.tigerdata.com/tutorials/index/ =====

Tutorials

Tiger Data tutorials are designed to help you get up and running with Tiger Data products. They walk you through a variety of scenarios using example datasets, to teach you how to construct interesting queries, find out what information your database has hidden in it, and even give you options for visualizing and graphing your results.

• Real-time analytics
  • Analytics on energy consumption: make data-driven decisions using energy consumption data.
  • Analytics on transport and geospatial data: optimize profits using geospatial transport data.
• Cryptocurrency
  • Query the Bitcoin blockchain: do your own research on the Bitcoin blockchain.
  • Analyze the Bitcoin blockchain: discover the relationship between transactions, blocks, fees, and miner revenue.
• Finance
  • Analyze financial tick data: chart the trading highs and lows for your favorite stock.
  • Ingest real-time financial data using WebSocket: use a websocket connection to visualize the trading highs and lows for your favorite stock.
• IoT
  • Simulate an IoT sensor dataset: simulate an IoT sensor dataset and run simple queries on it.
• Cookbooks
  • Tiger community cookbook: get suggestions from the Tiger community about how to resolve common issues.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/compression-dml-tuple-limit/ =====

Tuple decompression limit exceeded by operation

When inserting, updating, or deleting tuples from chunks in the columnstore, it might be necessary to convert tuples to the rowstore. This happens either when you are updating existing tuples or have constraints that need to be verified during insert time. If you happen to trigger a lot of rowstore conversion with a single command, you may end up running out of storage space. For this reason, a limit has been put in place on the number of tuples you can decompress into the rowstore for a single command.

The limit can be increased or turned off (set to 0) like so:

-- set limit to a milion tuples
SET timescaledb.max_tuples_decompressed_per_dml_transaction TO 1000000;
-- disable limit by setting to 0
SET timescaledb.max_tuples_decompressed_per_dml_transaction TO 0;

===== PAGE: https://docs.tigerdata.com/_troubleshooting/caggs-queries-fail/ =====

Queries fail when defining continuous aggregates but work on regular tables

Continuous aggregates do not work on all queries. For example, TimescaleDB does not support window functions on continuous aggregates. If you use an unsupported function, you see the following error:

      ERROR:  invalid continuous aggregate view
      SQL state: 0A000

The following table summarizes the aggregate functions supported in continuous aggregates:

Function, clause, or feature	TimescaleDB 2.6 and earlier	TimescaleDB 2.7, 2.8, and 2.9	TimescaleDB 2.10 and later
Parallelizable aggregate functions	✅	✅	✅
Non-parallelizable SQL aggregates	❌	✅	✅
ORDER BY	❌	✅	✅
Ordered-set aggregates	❌	✅	✅
Hypothetical-set aggregates	❌	✅	✅
DISTINCT in aggregate functions	❌	✅	✅
FILTER in aggregate functions	❌	✅	✅
FROM clause supports JOINS	❌	❌	✅

DISTINCT works in aggregate functions, not in the query definition. For example, for the table:

CREATE TABLE public.candle(
symbol_id uuid                     NOT NULL,
symbol    text                     NOT NULL,
"time"    timestamp with time zone NOT NULL,
open      double precision         NOT NULL,
high      double precision         NOT NULL,
low       double precision         NOT NULL,
close     double precision         NOT NULL,
volume    double precision         NOT NULL
);

• The following works:

  CREATE MATERIALIZED VIEW candles_start_end
  WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 hour', "time"), COUNT(DISTINCT symbol), first(time, time) as first_candle, last(time, time) as last_candle
  FROM candle
  GROUP BY 1;
  

• This does not:

  CREATE MATERIALIZED VIEW candles_start_end
  WITH (timescaledb.continuous) AS
  SELECT DISTINCT ON (symbol)
  symbol,symbol_id, first(time, time) as first_candle, last(time, time) as last_candle
  FROM candle
  GROUP BY symbol_id;
  

===== PAGE: https://docs.tigerdata.com/_troubleshooting/caggs-real-time-previously-materialized-not-shown/ =====

Updates to previously materialized regions aren't shown in real-time aggregates

Real-time aggregates automatically add the most recent data when you query your continuous aggregate. In other words, they include data more recent than your last materialized bucket.

If you add new historical data to an already-materialized bucket, it won't be reflected in a real-time aggregate. You should wait for the next scheduled refresh, or manually refresh by calling refresh_continuous_aggregate. You can think of real-time aggregates as being eventually consistent for historical data.

The following example shows how this works:

1. Create the hypertable:

   CREATE TABLE conditions(
     day DATE NOT NULL,
     city text NOT NULL,
     temperature INT NOT NULL
   )
   WITH (
      tsdb.hypertable,
      tsdb.partition_column='day',
      tsdb.chunk_interval='1 day'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

2. Add data to your hypertable:

   INSERT INTO conditions (day, city, temperature) VALUES
     ('2021-06-14', 'Moscow', 26),
     ('2021-06-15', 'Moscow', 22),
     ('2021-06-16', 'Moscow', 24),
     ('2021-06-17', 'Moscow', 24),
     ('2021-06-18', 'Moscow', 27),
     ('2021-06-19', 'Moscow', 28),
     ('2021-06-20', 'Moscow', 30),
     ('2021-06-21', 'Moscow', 31),
     ('2021-06-22', 'Moscow', 34),
     ('2021-06-23', 'Moscow', 34),
     ('2021-06-24', 'Moscow', 34),
     ('2021-06-25', 'Moscow', 32),
     ('2021-06-26', 'Moscow', 32),
     ('2021-06-27', 'Moscow', 31);
   

3. Create a continuous aggregate but do not materialize any data:

   1. Create the continuous aggregate:

      CREATE MATERIALIZED VIEW conditions_summary
      WITH (timescaledb.continuous) AS
      SELECT city,
         time_bucket('7 days', day) AS bucket,
         MIN(temperature),
         MAX(temperature)
      FROM conditions
      GROUP BY city, bucket
      WITH NO DATA;
      

   2. Check your data:

       SELECT * FROM conditions_summary ORDER BY bucket;
      

      The query on the continuous aggregate fetches data directly from the hypertable:

      city	bucket	min	max
      Moscow	2021-06-14	22	30
      Moscow	2021-06-21	31	34

4. Materialize data into the continuous aggregate:

   1. Add a refresh policy:

      CALL refresh_continuous_aggregate('conditions_summary', '2021-06-14', '2021-06-21');
      

   2. Check your data:

      SELECT * FROM conditions_summary ORDER BY bucket;
      

      The select query returns the same data, as expected, but this time the data is fetched from the underlying materialized table

      city	bucket	min	max
      Moscow	2021-06-14	22	30
      Moscow	2021-06-21	31	34

5. Update the data in the previously materialized bucket:

   1. Update the data in your hypertable:

      UPDATE conditions
      SET temperature = 35
      WHERE day = '2021-06-14' and city = 'Moscow';
      

   2. Check your data:

      SELECT * FROM conditions_summary ORDER BY bucket;
      

      The updated data is not yet visible when you query the continuous aggregate. This is because these changes have not been materialized. (Similarly, any INSERTs or DELETEs would also not be visible).

      city	bucket	min	max
      Moscow	2021-06-14	22	30
      Moscow	2021-06-21	31	34

6. Refresh the data again to update the previously materialized region:

   1. Refresh the data:

      CALL refresh_continuous_aggregate('conditions_summary', '2021-06-14', '2021-06-21');
      

7. Check your data:

   SELECT * FROM conditions_summary ORDER BY bucket;
   

   You see something like:

   city	bucket	min	max
   Moscow	2021-06-14	22	35
   Moscow	2021-06-21	31	34

===== PAGE: https://docs.tigerdata.com/_troubleshooting/caggs-hierarchical-buckets/ =====

Hierarchical continuous aggregate fails with incompatible bucket width

If you attempt to create a hierarchical continuous aggregate, you must use compatible time buckets. You can't create a continuous aggregate with a fixed-width time bucket on top of a continuous aggregate with a variable-width time bucket. For more information, see the restrictions section in hierarchical continuous aggregates.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/caggs-migrate-permissions/ =====

Permissions error when migrating a continuous aggregate

You might get a permissions error when migrating a continuous aggregate from old to new format using cagg_migrate. The user performing the migration must have the following permissions:

• Select, insert, and update permissions on the tables _timescale_catalog.continuous_agg_migrate_plan and _timescale_catalog.continuous_agg_migrate_plan_step
• Usage permissions on the sequence _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq

To solve the problem, change to a user capable of granting permissions, and grant the following permissions to the user performing the migration:

GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan TO <USER>;
GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan_step TO <USER>;
GRANT USAGE ON SEQUENCE _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq TO <USER>;

===== PAGE: https://docs.tigerdata.com/_troubleshooting/compression-high-cardinality/ =====

Low compression rate

Low compression rates are often caused by high cardinality of the segment key. This means that the column you selected for grouping the rows during compression has too many unique values. This makes it impossible to group a lot of rows in a batch. To achieve better compression results, choose a segment key with lower cardinality.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/dropping-chunks-times-out/ =====

Dropping chunks times out

When you drop a chunk, it requires an exclusive lock. If a chunk is being accessed by another session, you cannot drop the chunk at the same time. If a drop chunk operation can't get the lock on the chunk, then it times out and the process fails. To resolve this problem, check what is locking the chunk. In some cases, this could be caused by a continuous aggregate or other process accessing the chunk. When the drop chunk operation can get an exclusive lock on the chunk, it completes as expected.

For more information about locks, see the Postgres lock monitoring documentation.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/hypertables-unique-index-partitioning/ =====

Can't create unique index on hypertable, or can't create hypertable with unique index

You might get a unique index and partitioning column error in 2 situations:

• When creating a primary key or unique index on a hypertable
• When creating a hypertable from a table that already has a unique index or primary key

For more information on how to fix this problem, see the section on creating unique indexes on hypertables.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/explain/ =====

A particular query executes more slowly than expected

To troubleshoot a query, you can examine its EXPLAIN plan.

Postgres's EXPLAIN feature allows users to understand the underlying query plan that Postgres uses to execute a query. There are multiple ways that Postgres can execute a query: for example, a query might be fulfilled using a slow sequence scan or a much more efficient index scan. The choice of plan depends on what indexes are created on the table, the statistics that Postgres has about your data, and various planner settings. The EXPLAIN output let's you know which plan Postgres is choosing for a particular query. Postgres has a in-depth explanation of this feature.

To understand the query performance on a hypertable, we suggest first making sure that the planner statistics and table maintenance is up-to-date on the hypertable by running VACUUM ANALYZE <your-hypertable>;. Then, we suggest running the following version of EXPLAIN:

EXPLAIN (ANALYZE on, BUFFERS on) <original query>;

If you suspect that your performance issues are due to slow IOs from disk, you can get even more information by enabling the track_io_timing variable with SET track_io_timing = 'on'; before running the above EXPLAIN.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/caggs-hypertable-retention-policy-not-applying/ =====

Hypertable retention policy isn't applying to continuous aggregates

A retention policy set on a hypertable does not apply to any continuous aggregates made from the hypertable. This allows you to set different retention periods for raw and summarized data. To apply a retention policy to a continuous aggregate, set the policy on the continuous aggregate itself.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/columnstore-backlog-ooms/ =====

Out of memory errors after enabling the columnstore

By default, columnstore policies move all uncompressed chunks to the columnstore. However, before converting a large backlog of chunks from the rowstore to the columnstore, best practice is to set maxchunks_to_compress and limit to amount of chunks to be converted. For example:

SELECT alter_job(job_id, config.maxchunks_to_compress => 10);

When all chunks have been converted to the columnstore, set maxchunks_to_compress to 0, unlimited.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/cloud-singledb/ =====

Cannot create another database

Each Tiger Cloud service hosts a single Postgres instance called tsdb. You see this error when you try to create an additional database in a service. If you need another database, create a new service.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/caggs-inserted-historic-data-no-refresh/ =====

Continuous aggregate doesn't refresh with newly inserted historical data

Materialized views are generally used with ordered data. If you insert historic data, or data that is not related to the current time, you need to refresh policies and reevaluate the values that are dragging from past to present.

You can set up an after insert rule for your hypertable or upsert to trigger something that can validate what needs to be refreshed as the data is merged.

Let's say you inserted ordered timeframes named A, B, D, and F, and you already have a continuous aggregation looking for this data. If you now insert E, you need to refresh E and F. However, if you insert C we'll need to refresh C, D, E and F.

For example:

1. A, B, D, and F are already materialized in a view with all data.
2. To insert C, split the data into AB and DEF subsets.
3. AB are consistent and the materialized data is too; you only need to reuse it.
4. Insert C, DEF, and refresh policies after C.

This can use a lot of resources to process, especially if you have any important data in the past that also needs to be brought to the present.

Consider an example where you have 300 columns on a single hypertable and use, for example, five of them in a continuous aggregation. In this case, it could be hard to refresh and would make more sense to isolate these columns in another hypertable. Alternatively, you might create one hypertable per metric and refresh them independently.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/locf-queries-null-values-not-missing/ =====

Queries using locf() don't treat NULL values as missing

When you have a query that uses a last observation carried forward (locf) function, the query carries forward NULL values by default. If you want the function to ignore NULL values instead, you can set treat_null_as_missing=TRUE as the second parameter in the query. For example:

dev=# select * FROM (select time_bucket_gapfill(4, time,-5,13), locf(avg(v)::int,treat_null_as_missing:=true) FROM (VALUES (0,0),(8,NULL)) v(time, v) WHERE time BETWEEN 0 AND 10 GROUP BY 1) i ORDER BY 1 DESC;
 time_bucket_gapfill | locf
---------------------+------
                  12 |    0
                   8 |    0
                   4 |    0
                   0 |    0
                  -4 |
                  -8 |
(6 rows)

===== PAGE: https://docs.tigerdata.com/_troubleshooting/cagg-watermark-in-future/ =====

Continuous aggregate watermark is in the future

Continuous aggregates use a watermark to indicate which time buckets have already been materialized. When you query a continuous aggregate, your query returns materialized data from before the watermark. It returns real-time, non-materialized data from after the watermark.

In certain cases, the watermark might be in the future. If this happens, all buckets, including the most recent bucket, are materialized and below the watermark. No real-time data is returned.

This might happen if you refresh your continuous aggregate over the time window <START_TIME>, NULL, which materializes all recent data. It might also happen if you create a continuous aggregate using the WITH DATA option. This also implicitly refreshes your continuous aggregate with a window of NULL, NULL.

To fix this, create a new continuous aggregate using the WITH NO DATA option. Then use a policy to refresh this continuous aggregate over an explicit time window.

Creating a new continuous aggregate with an explicit refresh window

1. Create a continuous aggregate using the WITH NO DATA option:

   CREATE MATERIALIZED VIEW <continuous_aggregate_name>
       WITH (timescaledb.continuous)
       AS SELECT time_bucket('<interval>', <partition_column>),
       <other_columns_to_select>,
       ...
       FROM <hypertable>
       GROUP BY bucket,
       WITH NO DATA;
   

2. Refresh the continuous aggregate using a policy with an explicit end_offset. For example:

   SELECT add_continuous_aggregate_policy('<continuous_aggregate_name>',
       start_offset => INTERVAL '30 day',
       end_offset => INTERVAL '1 hour',
       schedule_interval => INTERVAL '1 hour');
   

3. Check your new continuous aggregate's watermark to make sure it is in the past, not the future.

   Get the ID for the materialization hypertable that contains the actual continuous aggregate data:

   SELECT id FROM _timescaledb_catalog.hypertable
       WHERE table_name=(
           SELECT materialization_hypertable_name
               FROM timescaledb_information.continuous_aggregates
               WHERE view_name='<continuous_aggregate_name>'
       );
   

4. Use the returned ID to query for the watermark's timestamp:

   For TimescaleDB >= 2.12:

   SELECT COALESCE(
       _timescaledb_functions.to_timestamp(_timescaledb_functions.cagg_watermark(<ID>)),
       '-infinity'::timestamp with time zone
   );
   

   For TimescaleDB < 2.12:

   SELECT COALESCE(
       _timescaledb_internal.to_timestamp(_timescaledb_internal.cagg_watermark(<ID>)),
       '-infinity'::timestamp with time zone
   );
   

If you choose to delete your old continuous aggregate after creating a new one, beware of historical data loss. If your old continuous aggregate contained data that you dropped from your original hypertable, for example through a data retention policy, the dropped data is not included in your new continuous aggregate.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/scheduled-jobs-stop-running/ =====

Scheduled jobs stop running

Your scheduled jobs might stop running for various reasons. On self-hosted TimescaleDB, you can fix this by restarting background workers:

= 2.12">

SELECT _timescaledb_functions.start_background_workers();

SELECT _timescaledb_internal.start_background_workers();

On Tiger Cloud and Managed Service for TimescaleDB, restart background workers by doing one of the following:

• Run SELECT timescaledb_pre_restore(), followed by SELECT timescaledb_post_restore().
• Power the service off and on again. This might cause a downtime of a few minutes while the service restores from backup and replays the write-ahead log.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/invalid-attribute-reindex-hypertable/ =====

Reindex hypertables to fix large indexes

You might see this error if your hypertable indexes have become very large. To resolve the problem, reindex your hypertables with this command:

reindex table _timescaledb_internal._hyper_2_1523284_chunk

For more information, see the hypertable documentation.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/compression-userperms/ =====

User permissions do not allow chunks to be converted to columnstore or rowstore

You might get this error if you attempt to compress a chunk into the columnstore, or decompress it back into rowstore with a non-privileged user account. To compress or decompress a chunk, your user account must have permissions that allow it to perform CREATE INDEX on the chunk. You can check the permissions of the current user with this command at the psql command prompt:

\dn+ <USERNAME>

To resolve this problem, grant your user account the appropriate privileges with this command:

GRANT PRIVILEGES
    ON TABLE
    TO <ROLE_TYPE>;

For more information about the GRANT command, see the Postgres documentation.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/compression-inefficient-chunk-interval/ =====

Inefficient compress_chunk_time_interval configuration

When you configure compress_chunk_time_interval but do not set the primary dimension as the first column in compress_orderby, TimescaleDB decompresses chunks before merging. This makes merging less efficient. Set the primary dimension of the chunk as the first column in compress_orderby to improve efficiency.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/cloud-jdbc-authentication-support/ =====

JDBC authentication type is not supported

When connecting to Tiger Cloud with a Java Database Connectivity (JDBC) driver, you might get this error message.

Your Tiger Cloud authentication type doesn't match your JDBC driver's supported authentication types. The recommended approach is to upgrade your JDBC driver to a version that supports scram-sha-256 encryption. If that isn't an option, you can change the authentication type for your Tiger Cloud service to md5. Note that md5 is less secure, and is provided solely for compatibility with older clients.

For information on changing your authentication type, see the documentation on resetting your service password.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/chunk-temp-file-limit/ =====

Temporary file size limit exceeded when converting chunks to the columnstore

When you try to convert a chunk to the columnstore, especially if the chunk is very large, you could get this error. Compression operations write files to a new compressed chunk table, which is written in temporary memory. The maximum amount of temporary memory available is determined by the temp_file_limit parameter. You can work around this problem by adjusting the temp_file_limit and maintenance_work_mem parameters.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/slow-tiering-chunks/ =====

Slow tiering of chunks

Chunks are tiered asynchronously. Chunks are selected to be tiered to the object storage tier one at a time ordered by their enqueue time.

To see the chunks waiting to be tiered query the timescaledb_osm.chunks_queued_for_tiering view

select count(*) from timescaledb_osm.chunks_queued_for_tiering

Processing all the chunks in the queue may take considerable time if a large quantity of data is being migrated to the object storage tier.

===== PAGE: https://docs.tigerdata.com/self-hosted/index/ =====

Self-hosted TimescaleDB

TimescaleDB is an extension for Postgres that enables time-series workloads, increasing ingest, query, storage and analytics performance.

Best practice is to run TimescaleDB in a Tiger Cloud service, but if you want to self-host you can run TimescaleDB yourself. Deploy a Tiger Cloud service. We tune your database for performance and handle scalability, high availability, backups and management so you can relax.

Self-hosted TimescaleDB is community supported. For additional help check out the friendly Tiger Data community.

If you'd prefer to pay for support then check out our self-managed support.

===== PAGE: https://docs.tigerdata.com/self-hosted/configuration/about-configuration/ =====

About configuration in TimescaleDB

By default, TimescaleDB uses the default Postgres server configuration settings. However, in some cases, these settings are not appropriate, especially if you have larger servers that use more hardware resources such as CPU, memory, and storage. This section explains some of the settings you are most likely to need to adjust.

Some of these settings are Postgres settings, and some are TimescaleDB specific settings. For most changes, you can use the tuning tool to adjust your configuration. For more advanced configuration settings, or to change settings that aren't included in the timescaledb-tune tool, you can manually adjust the postgresql.conf configuration file.

Memory

Settings:

• shared_buffers
• effective_cache_size
• work_mem
• maintenance_work_mem
• max_connections

You can adjust each of these to match the machine's available memory. To make it easier, you can use the PgTune site to work out what settings to use: enter your machine details, and select the data warehouse DB type to see the suggested parameters.

You can adjust these settings with timescaledb-tune.

Workers

Settings:

• timescaledb.max_background_workers
• max_parallel_workers
• max_worker_processes

Postgres uses worker pools to provide workers for live queries and background jobs. If you do not configure these settings, your queries and background jobs could run more slowly.

TimescaleDB background workers are configured with timescaledb.max_background_workers. Each database needs a background worker allocated to schedule jobs. Additional workers run background jobs as required. This setting should be the sum of the total number of databases and the total number of concurrent background workers you want running at any one time. By default, timescaledb-tune sets timescaledb.max_background_workers to 16. You can change this setting directly, use the --max-bg-workers flag, or adjust the TS_TUNE_MAX_BG_WORKERS Docker environment variable.

TimescaleDB parallel workers are configured with max_parallel_workers. For larger queries, Postgres automatically uses parallel workers if they are available. Increasing this setting can improve query performance for large queries that trigger the use of parallel workers. By default, this setting corresponds to the number of CPUs available. You can change this parameter directly, by adjusting the --cpus flag, or by using the TS_TUNE_NUM_CPUS Docker environment variable.

The max_worker_processes setting defines the total pool of workers available to both background and parallel workers, as well a small number of built-in Postgres workers. It should be at least the sum of timescaledb.max_background_workers and max_parallel_workers.

You can adjust these settings with timescaledb-tune.

Disk writes

Settings:

• synchronous_commit

By default, disk writes are performed synchronously, so each transaction must be completed and a success message sent, before the next transaction can begin. You can change this to asynchronous to increase write throughput by setting synchronous_commit = 'off'. Note that disabling synchronous commits could result in some committed transactions being lost. To help reduce the risk, do not also change fsync setting. For more information about asynchronous commits and disk write speed, see the Postgres documentation.

You can adjust these settings in the postgresql.conf configuration file.

Transaction locks

Settings:

• max_locks_per_transaction

TimescaleDB relies on table partitioning to scale time-series workloads. A hypertable needs to acquire locks on many chunks during queries, which can exhaust the default limits for the number of allowed locks held. In some cases, you might see a warning like this:

psql: FATAL:  out of shared memory
HINT:  You might need to increase max_locks_per_transaction.

To avoid this issue, you can increase the max_locks_per_transaction setting from the default value, which is usually 64. This parameter limits the average number of object locks used by each transaction; individual transactions can lock more objects as long as the locks of all transactions fit in the lock table.

For most workloads, choose a number equal to double the maximum number of chunks you expect to have in a hypertable divided by max_connections. This takes into account that the number of locks used by a hypertable query is roughly equal to the number of chunks in the hypertable if you need to access all chunks in a query, or double that number if the query uses an index. You can see how many chunks you currently have using the timescaledb_information.hypertables view. Changing this parameter requires a database restart, so make sure you pick a larger number to allow for some growth. For more information about lock management, see the Postgres documentation.

You can adjust these settings in the postgresql.conf configuration file.

===== PAGE: https://docs.tigerdata.com/self-hosted/configuration/timescaledb-config/ =====

TimescaleDB configuration and tuning

Just as you can tune settings in Postgres, TimescaleDB provides a number of configuration settings that may be useful to your specific installation and performance needs. These can also be set within the postgresql.conf file or as command-line parameters when starting Postgres. when starting Postgres.

Just as you can tune settings in Postgres, TimescaleDB provides a number of configuration settings that may be useful to your specific installation and performance needs. These can also be set within the postgresql.conf file or as command-line parameters when starting Postgres.

Query Planning and Execution

timescaledb.enable_chunkwise_aggregation (bool)

If enabled, aggregations are converted into partial aggregations during query planning. The first part of the aggregation is executed on a per-chunk basis. Then, these partial results are combined and finalized. Splitting aggregations decreases the size of the created hash tables and increases data locality, which speeds up queries.

timescaledb.vectorized_aggregation (bool)

Enables or disables the vectorized optimizations in the query executor. For example, the sum() aggregation function on compressed chunks can be optimized in this way.

timescaledb.enable_merge_on_cagg_refresh (bool)

Set to ON to dramatically decrease the amount of data written on a continuous aggregate in the presence of a small number of changes, reduce the i/o cost of refreshing a continuous aggregate, and generate fewer Write-Ahead Logs (WAL). Only works for continuous aggregates that don't have compression enabled.

Please refer to the Grand Unified Configuration (GUC) parameters for a complete list.

Policies

timescaledb.max_background_workers (int)

Max background worker processes allocated to TimescaleDB. Set to at least 1 + the number of databases loaded with the TimescaleDB extension in a Postgres instance. Default value is 16.

Tiger Cloud service tuning

timescaledb.disable_load (bool)

Disable the loading of the actual extension

Administration

timescaledb.restoring (bool)

Set TimescaleDB in restoring mode. It is disabled by default.

timescaledb.license (string)

Change access to features based on the TimescaleDB license in use. For example, setting timescaledb.license to apache limits TimescaleDB to features that are implemented under the Apache 2 license. The default value is timescale, which allows access to all features.

timescaledb.telemetry_level (enum)

Telemetry settings level. Level used to determine which telemetry to send. Can be set to off or basic. Defaults to basic.

timescaledb.last_tuned (string)

Records last time timescaledb-tune ran.

timescaledb.last_tuned_version (string)

Version of timescaledb-tune used to tune when it runs.

Distributed hypertables

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

timescaledb.enable_2pc (bool)

Enables two-phase commit for distributed hypertables. If disabled, it uses a one-phase commit instead, which is faster but can result in inconsistent data. It is by default enabled.

timescaledb.enable_per_data_node_queries

If enabled, TimescaleDB combines different chunks belonging to the same hypertable into a single query per data node. It is by default enabled.

timescaledb.max_insert_batch_size (int)

When acting as a access node, TimescaleDB splits batches of inserted tuples across multiple data nodes. It batches up to max_insert_batch_size tuples per data node before flushing. Setting this to 0 disables batching, reverting to tuple-by-tuple inserts. The default value is 1000.

timescaledb.enable_connection_binary_data (bool)

Enables binary format for data exchanged between nodes in the cluster. It is by default enabled.

timescaledb.enable_client_ddl_on_data_nodes (bool)

Enables DDL operations on data nodes by a client and do not restrict execution of DDL operations only by access node. It is by default disabled.

timescaledb.enable_async_append (bool)

Enables optimization that runs remote queries asynchronously across data nodes. It is by default enabled.

timescaledb.enable_remote_explain (bool)

Enable getting and showing EXPLAIN output from remote nodes. This requires sending the query to the data node, so it can be affected by the network connection and availability of data nodes. It is by default disabled.

timescaledb.remote_data_fetcher (enum)

Pick data fetcher type based on type of queries you plan to run, which can be either copy, cursor, or auto. The default is auto.

timescaledb.ssl_dir (string)

Specifies the path used to search user certificates and keys when connecting to data nodes using certificate authentication. Defaults to timescaledb/certs under the Postgres data directory.

timescaledb.passfile (string) [

Specifies the name of the file where passwords are stored and when connecting to data nodes using password authentication.

===== PAGE: https://docs.tigerdata.com/self-hosted/configuration/docker-config/ =====

Configuration with Docker

If you are running TimescaleDB in a Docker container, there are two different ways to modify your Postgres configuration. You can edit the Postgres configuration file inside the Docker container, or you can set parameters at the command prompt.

Edit the Postgres configuration file inside Docker

You can start the Dockert container, and then use a text editor to edit the Postgres configuration file directly. The configuration file requires one parameter per line. Blank lines are ignored, and you can use a # symbol at the beginning of a line to denote a comment.

Editing the Postgres configuration file inside Docker

1. Start your Docker instance:

   docker start timescaledb
   

2. Open a shell:

   docker exec -i -t timescaledb /bin/bash
   

3. Open the configuration file in Vi editor or your preferred text editor.

   vi /var/lib/postgresql/data/postgresql.conf
   

4. Restart the container to reload the configuration:

   docker restart timescaledb
   

Setting parameters at the command prompt

If you don't want to open the configuration file to make changes, you can also set parameters directly from the command prompt inside your Docker container, using the -c option. For example:

docker run -i -t timescale/timescaledb:latest-pg10 postgres -c max_wal_size=2GB

===== PAGE: https://docs.tigerdata.com/self-hosted/configuration/configuration/ =====

Configuring TimescaleDB

TimescaleDB works with the default Postgres server configuration settings. However, we find that these settings are typically too conservative and can be limiting when using larger servers with more resources (CPU, memory, disk, etc). Adjusting these settings, either automatically with our tool timescaledb-tune or manually editing your machine's postgresql.conf, can improve performance.

You can determine the location of postgresql.conf by running SHOW config_file; from your Postgres client (for example, psql).

In addition, other TimescaleDB specific settings can be modified through the postgresql.conf file as covered in the TimescaleDB settings section.

Using timescaledb-tune

To streamline the configuration process, use timescaledb-tune that handles setting the most common parameters to appropriate values based on your system, accounting for memory, CPU, and Postgres version. timescaledb-tune is packaged along with the binary releases as a dependency, so if you installed one of the binary releases (including Docker), you should have access to the tool. Alternatively, with a standard Go environment, you can also go get the repository to install it.

timescaledb-tune reads your system's postgresql.conf file and offers interactive suggestions for updating your settings:

Using postgresql.conf at this path:
/usr/local/var/postgres/postgresql.conf

Is this correct? [(y)es/(n)o]: y
Writing backup to:
/var/folders/cr/zpgdkv194vz1g5smxl_5tggm0000gn/T/timescaledb_tune.backup201901071520

shared_preload_libraries needs to be updated
Current:
#shared_preload_libraries = 'timescaledb'
Recommended:
shared_preload_libraries = 'timescaledb'
Is this okay? [(y)es/(n)o]: y
success: shared_preload_libraries will be updated

Tune memory/parallelism/WAL and other settings? [(y)es/(n)o]: y
Recommendations based on 8.00 GB of available memory and 4 CPUs for PostgreSQL 11

Memory settings recommendations
Current:
shared_buffers = 128MB
#effective_cache_size = 4GB
#maintenance_work_mem = 64MB
#work_mem = 4MB
Recommended:
shared_buffers = 2GB
effective_cache_size = 6GB
maintenance_work_mem = 1GB
work_mem = 26214kB
Is this okay? [(y)es/(s)kip/(q)uit]:

These changes are then written to your postgresql.conf and take effect on the next (re)start. If you are starting on fresh instance and don't feel the need to approve each group of changes, you can also automatically accept and append the suggestions to the end of your postgresql.conf like so:

timescaledb-tune --quiet --yes --dry-run >> /path/to/postgresql.conf

Postgres configuration and tuning

If you prefer to tune the settings yourself, or are curious about the suggestions that timescaledb-tune makes, then check these. However, timescaledb-tune does not cover all settings that you need to adjust.

Memory settings

All of these settings are handled by timescaledb-tune.

The settings shared_buffers, effective_cache_size, work_mem, and maintenance_work_mem need to be adjusted to match the machine's available memory. Get the configuration values from the PgTune website (suggested DB Type: Data warehouse). You should also adjust the max_connections setting to match the ones given by PgTune since there is a connection between max_connections and memory settings. Other settings from PgTune may also be helpful.

Worker settings

All of these settings are handled by timescaledb-tune.

Postgres utilizes worker pools to provide the required workers needed to support both live queries and background jobs. If you do not configure these settings, you may observe performance degradation on both queries and background jobs.

TimescaleDB background workers are configured using the timescaledb.max_background_workers setting. You should configure this setting to the sum of your total number of databases and the total number of concurrent background workers you want running at any given point in time. You need a background worker allocated to each database to run a lightweight scheduler that schedules jobs. On top of that, any additional workers you allocate here run background jobs when needed.

For larger queries, Postgres automatically uses parallel workers if they are available. To configure this use the max_parallel_workers setting. Increasing this setting improves query performance for larger queries. Smaller queries may not trigger parallel workers. By default, this setting corresponds to the number of CPUs available. Use the --cpus flag or the TS_TUNE_NUM_CPUS docker environment variable to change it.

Finally, you must configure max_worker_processes to be at least the sum of timescaledb.max_background_workers and max_parallel_workers. max_worker_processes is the total pool of workers available to both background and parallel workers (as well as a handful of built-in Postgres workers).

By default, timescaledb-tune sets timescaledb.max_background_workers to 16. In order to change this setting, use the --max-bg-workers flag or the TS_TUNE_MAX_BG_WORKERS docker environment variable. The max_worker_processes setting is automatically adjusted as well.

Disk-write settings

In order to increase write throughput, there are multiple settings to adjust the behavior that Postgres uses to write data to disk. In tests, performance is good with the default, or safest, settings. If you want a bit of additional performance, you can set synchronous_commit = 'off'(Postgres docs). Please note that when disabling synchronous_commit in this way, an operating system or database crash might result in some recent allegedly committed transactions being lost. We actively discourage changing the fsync setting.

Lock settings

TimescaleDB relies heavily on table partitioning for scaling time-series workloads, which has implications for lock management. A hypertable needs to acquire locks on many chunks (sub-tables) during queries, which can exhaust the default limits for the number of allowed locks held. This might result in a warning like the following:

psql: FATAL:  out of shared memory
HINT:  You might need to increase max_locks_per_transaction.

To avoid this issue, it is necessary to increase the max_locks_per_transaction setting from the default value (which is typically 64). Since changing this parameter requires a database restart, it is advisable to estimate a good setting that also allows some growth. For most use cases we recommend the following setting:

max_locks_per_transaction = 2 * num_chunks / max_connections

where num_chunks is the maximum number of chunks you expect to have in a hypertable and max_connections is the number of connections configured for Postgres. This takes into account that the number of locks used by a hypertable query is roughly equal to the number of chunks in the hypertable if you need to access all chunks in a query, or double that number if the query uses an index. You can see how many chunks you currently have using the timescaledb_information.hypertables view. Changing this parameter requires a database restart, so make sure you pick a larger number to allow for some growth. For more information about lock management, see the Postgres documentation.

TimescaleDB configuration and tuning

Just as you can tune settings in Postgres, TimescaleDB provides a number of configuration settings that may be useful to your specific installation and performance needs. These can also be set within the postgresql.conf file or as command-line parameters when starting Postgres.

Policies

timescaledb.max_background_workers (int)

Max background worker processes allocated to TimescaleDB. Set to at least 1 + number of databases in Postgres instance to use background workers. Default value is 8.

Distributed hypertables

timescaledb.hypertable_distributed_default (enum)

Set default policy to create local or distributed hypertables for create_hypertable() command, when the distributed argument is not provided. Supported values are auto, local or distributed.

timescaledb.hypertable_replication_factor_default (int)

Global default value for replication factor to use with hypertables when the replication_factor argument is not provided. Defaults to 1.

timescaledb.enable_2pc (bool)

Enables two-phase commit for distributed hypertables. If disabled, it uses a one-phase commit instead, which is faster but can result in inconsistent data. It is by default enabled.

timescaledb.enable_per_data_node_queries (bool)

If enabled, TimescaleDB combines different chunks belonging to the same hypertable into a single query per data node. It is by default enabled.

timescaledb.max_insert_batch_size (int)

When acting as a access node, TimescaleDB splits batches of inserted tuples across multiple data nodes. It batches up to max_insert_batch_size tuples per data node before flushing. Setting this to 0 disables batching, reverting to tuple-by-tuple inserts. The default value is 1000.

timescaledb.enable_connection_binary_data (bool)

Enables binary format for data exchanged between nodes in the cluster. It is by default enabled.

timescaledb.enable_client_ddl_on_data_nodes (bool)

Enables DDL operations on data nodes by a client and do not restrict execution of DDL operations only by access node. It is by default disabled.

timescaledb.enable_async_append (bool)

Enables optimization that runs remote queries asynchronously across data nodes. It is by default enabled.

timescaledb.enable_remote_explain (bool)

Enable getting and showing EXPLAIN output from remote nodes. This requires sending the query to the data node, so it can be affected by the network connection and availability of data nodes. It is by default disabled.

timescaledb.remote_data_fetcher (enum)

Pick data fetcher type based on type of queries you plan to run, which can be either rowbyrow or cursor. The default is rowbyrow.

timescaledb.ssl_dir (string)

Specifies the path used to search user certificates and keys when connecting to data nodes using certificate authentication. Defaults to timescaledb/certs under the Postgres data directory.

timescaledb.passfile (string)

Specifies the name of the file where passwords are stored and when connecting to data nodes using password authentication.

Administration

timescaledb.restoring (bool)

Set TimescaleDB in restoring mode. It is by default disabled.

timescaledb.license (string)

TimescaleDB license type. Determines which features are enabled. The variable can be set to timescale or apache. Defaults to timescale.

timescaledb.telemetry_level (enum)

Telemetry settings level. Level used to determine which telemetry to send. Can be set to off or basic. Defaults to basic.

timescaledb.last_tuned (string)

Records last time timescaledb-tune ran.

timescaledb.last_tuned_version (string)

Version of timescaledb-tune used to tune when it ran.

Changing configuration with Docker

When running TimescaleDB in a Docker container, there are two approaches to modifying your Postgres configuration. In the following example, we modify the size of the database instance's write-ahead-log (WAL) from 1 GB to 2 GB in a Docker container named timescaledb.

Modifying postgres.conf inside Docker

1. Open a shell in Docker to change the configuration on a running container.

docker start timescaledb
docker exec -i -t timescaledb /bin/bash

1. Edit and then save the config file, modifying the setting for the desired configuration parameter (for example, max_wal_size).

vi /var/lib/postgresql/data/postgresql.conf

1. Restart the container so the config gets reloaded.

docker restart timescaledb

1. Test to see if the change worked.

    docker exec -it timescaledb psql -U postgres

    postgres=# show max_wal_size;
     max_wal_size
    --------------
    2GB

Specify configuration parameters as boot options

Alternatively, one or more parameters can be passed in to the docker run command via a -c option, as in the following.

docker run -i -t timescale/timescaledb:latest-pg10 postgres -cmax_wal_size=2GB

Additional examples of passing in arguments at boot can be found in our discussion about using WAL-E for incremental backup.

===== PAGE: https://docs.tigerdata.com/self-hosted/configuration/telemetry/ =====

Telemetry and version checking

TimescaleDB collects anonymous usage data to help us better understand and assist our users. It also helps us provide some services, such as automated version checking. Your privacy is the most important thing to us, so we do not collect any personally identifying information. In particular, the UUID (user ID) fields contain no identifying information, but are randomly generated by appropriately seeded random number generators.

This is an example of the JSON data file that is sent for a specific deployment:

{
  "db_uuid": "860c2be4-59a3-43b5-b895-5d9e0dd44551",
  "license": {
    "edition": "community"
  },
  "os_name": "Linux",
  "relations": {
    "views": {
      "num_relations": 0
    },
    "tables": {
      "heap_size": 32768,
      "toast_size": 16384,
      "indexes_size": 98304,
      "num_relations": 4,
      "num_reltuples": 12
    },
    "hypertables": {
      "heap_size": 3522560,
      "toast_size": 23379968,
      "compression": {
        "compressed_heap_size": 3522560,
        "compressed_row_count": 4392,
        "compressed_toast_size": 20365312,
        "num_compressed_chunks": 366,
        "uncompressed_heap_size": 41951232,
        "uncompressed_row_count": 421368,
        "compressed_indexes_size": 11993088,
        "uncompressed_toast_size": 2998272,
        "uncompressed_indexes_size": 42696704,
        "num_compressed_hypertables": 1
      },
      "indexes_size": 18022400,
      "num_children": 366,
      "num_relations": 2,
      "num_reltuples": 421368
    },
    "materialized_views": {
      "heap_size": 0,
      "toast_size": 0,
      "indexes_size": 0,
      "num_relations": 0,
      "num_reltuples": 0
    },
    "partitioned_tables": {
      "heap_size": 0,
      "toast_size": 0,
      "indexes_size": 0,
      "num_children": 0,
      "num_relations": 0,
      "num_reltuples": 0
    },
    "continuous_aggregates": {
      "heap_size": 122404864,
      "toast_size": 6225920,
      "compression": {
        "compressed_heap_size": 0,
        "compressed_row_count": 0,
        "num_compressed_caggs": 0,
        "compressed_toast_size": 0,
        "num_compressed_chunks": 0,
        "uncompressed_heap_size": 0,
        "uncompressed_row_count": 0,
        "compressed_indexes_size": 0,
        "uncompressed_toast_size": 0,
        "uncompressed_indexes_size": 0
      },
      "indexes_size": 165044224,
      "num_children": 760,
      "num_relations": 24,
      "num_reltuples": 914704,
      "num_caggs_on_distributed_hypertables": 0,
      "num_caggs_using_real_time_aggregation": 24
    },
    "distributed_hypertables_data_node": {
      "heap_size": 0,
      "toast_size": 0,
      "compression": {
        "compressed_heap_size": 0,
        "compressed_row_count": 0,
        "compressed_toast_size": 0,
        "num_compressed_chunks": 0,
        "uncompressed_heap_size": 0,
        "uncompressed_row_count": 0,
        "compressed_indexes_size": 0,
        "uncompressed_toast_size": 0,
        "uncompressed_indexes_size": 0,
        "num_compressed_hypertables": 0
      },
      "indexes_size": 0,
      "num_children": 0,
      "num_relations": 0,
      "num_reltuples": 0
    },
    "distributed_hypertables_access_node": {
      "heap_size": 0,
      "toast_size": 0,
      "compression": {
        "compressed_heap_size": 0,
        "compressed_row_count": 0,
        "compressed_toast_size": 0,
        "num_compressed_chunks": 0,
        "uncompressed_heap_size": 0,
        "uncompressed_row_count": 0,
        "compressed_indexes_size": 0,
        "uncompressed_toast_size": 0,
        "uncompressed_indexes_size": 0,
        "num_compressed_hypertables": 0
      },
      "indexes_size": 0,
      "num_children": 0,
      "num_relations": 0,
      "num_reltuples": 0,
      "num_replica_chunks": 0,
      "num_replicated_distributed_hypertables": 0
    }
  },
  "os_release": "5.10.47-linuxkit",
  "os_version": "#1 SMP Sat Jul 3 21:51:47 UTC 2021",
  "data_volume": 381903727,
  "db_metadata": {},
  "build_os_name": "Linux",
  "functions_used": {
    "pg_catalog.int8(integer)": 8,
    "pg_catalog.count(pg_catalog.\"any\")": 20,
    "pg_catalog.int4eq(integer,integer)": 7,
    "pg_catalog.textcat(pg_catalog.text,pg_catalog.text)": 10,
    "pg_catalog.chareq(pg_catalog.\"char\",pg_catalog.\"char\")": 6,
  },
  "install_method": "docker",
  "installed_time": "2022-02-17T19:55:14+00",
  "os_name_pretty": "Alpine Linux v3.15",
  "last_tuned_time": "2022-02-17T19:55:14Z",
  "build_os_version": "5.11.0-1028-azure",
  "exported_db_uuid": "5730161f-0d18-42fb-a800-45df33494c21",
  "telemetry_version": 2,
  "build_architecture": "x86_64",
  "distributed_member": "none",
  "last_tuned_version": "0.12.0",
  "postgresql_version": "12.10",
  "related_extensions": {
    "postgis": false,
    "pg_prometheus": false,
    "timescale_analytics": false,
    "timescaledb_toolkit": false
  },
  "timescaledb_version": "2.6.0",
  "num_reorder_policies": 0,
  "num_retention_policies": 0,
  "num_compression_policies": 1,
  "num_user_defined_actions": 1,
  "build_architecture_bit_size": 64,
  "num_continuous_aggs_policies": 24
}

If you want to see the exact JSON data file that is sent, use the get_telemetry_report API call.

Telemetry reports are different if you are using an open source or community version of TimescaleDB. For these versions, the report includes an edition field, with a value of either apache_only or community.

Change what is included the telemetry report

If you want to adjust which metadata is included or excluded from the telemetry report, you can do so in the _timescaledb_catalog.metadata table. Metadata which has include_in_telemetry set to true, and a value of timescaledb_telemetry.cloud, is included in the telemetry report.

Version checking

Telemetry reports are sent periodically in the background. In response to the telemetry report, the database receives the most recent version of TimescaleDB available for installation. This version is recorded in your server logs, along with any applicable out-of-date version warnings. You do not have to update immediately to the newest release, but we highly recommend that you do so, to take advantage of performance improvements and bug fixes.

Disable telemetry

It is highly recommend that you leave telemetry enabled, as it provides useful features for you, and helps to keep improving Timescale. However, you can turn off telemetry if you need to for a specific database, or for an entire instance.

If you turn off telemetry, the version checking feature is also turned off.

Disabling telemetry

1. Open your Postgres configuration file, and locate the timescaledb.telemetry_level parameter. See the Postgres configuration file instructions for locating and opening the file.

2. Change the parameter setting to off:

   timescaledb.telemetry_level=off
   

3. Reload the configuration file:

   pg_ctl
   

4. Alternatively, you can use this command at the psql prompt, as the root user:

   ALTER [SYSTEM | DATABASE | USER] { *db_name* | *role_specification* } SET timescaledb.telemetry_level=off
   

   This command disables telemetry for the specified system, database, or user.

Enabling telemetry

1. Open your Postgres configuration file, and locate the 'timescaledb.telemetry_level' parameter. See the Postgres configuration file instructions for locating and opening the file.

2. Change the parameter setting to 'off':

   timescaledb.telemetry_level=basic
   

3. Reload the configuration file:

   pg_ctl
   

4. Alternatively, you can use this command at the psql prompt, as the root user:

   ALTER [SYSTEM | DATABASE | USER] { *db_name* | *role_specification* } SET timescaledb.telemetry_level=basic
   

   This command enables telemetry for the specified system, database, or user.

===== PAGE: https://docs.tigerdata.com/self-hosted/configuration/timescaledb-tune/ =====

TimescaleDB tuning tool

To help make configuring TimescaleDB a little easier, you can use the timescaledb-tune tool. This tool handles setting the most common parameters to good values based on your system. It accounts for memory, CPU, and Postgres version. timescaledb-tune is packaged with the TimescaleDB binary releases as a dependency, so if you installed TimescaleDB from a binary release (including Docker), you should already have access to the tool. Alternatively, you can use the go install command to install it:

go install github.com/timescale/timescaledb-tune/cmd/timescaledb-tune@latest

The timescaledb-tune tool reads your system's postgresql.conf file and offers interactive suggestions for your settings. Here is an example of the tool running:

Using postgresql.conf at this path:
/usr/local/var/postgres/postgresql.conf

Is this correct? [(y)es/(n)o]: y
Writing backup to:
/var/folders/cr/example/T/timescaledb_tune.backup202101071520

shared_preload_libraries needs to be updated
Current:
#shared_preload_libraries = 'timescaledb'
Recommended:
shared_preload_libraries = 'timescaledb'
Is this okay? [(y)es/(n)o]: y
success: shared_preload_libraries will be updated

Tune memory/parallelism/WAL and other settings? [(y)es/(n)o]: y
Recommendations based on 8.00 GB of available memory and 4 CPUs for PostgreSQL 12

Memory settings recommendations
Current:
shared_buffers = 128MB
#effective_cache_size = 4GB
#maintenance_work_mem = 64MB
#work_mem = 4MB
Recommended:
shared_buffers = 2GB
effective_cache_size = 6GB
maintenance_work_mem = 1GB
work_mem = 26214kB
Is this okay? [(y)es/(s)kip/(q)uit]:

When you have answered the questions, the changes are written to your postgresql.conf and take effect when you next restart.

If you are starting on a fresh instance and don't want to approve each group of changes, you can automatically accept and append the suggestions to the end of your postgresql.conf by using some additional flags when you run the tool:

timescaledb-tune --quiet --yes --dry-run >> /path/to/postgresql.conf

===== PAGE: https://docs.tigerdata.com/self-hosted/configuration/postgres-config/ =====

Manual Postgres configuration and tuning

If you prefer to tune settings yourself, or for settings not covered by timescaledb-tune, you can manually configure your installation using the Postgres configuration file.

For some common configuration settings you might want to adjust, see the about-configuration page.

For more information about the Postgres configuration page, see the Postgres documentation.

Edit the Postgres configuration file

The location of the Postgres configuration file depends on your operating system and installation.

1. Find the location of the config file for your Postgres instance

   1. Connect to your database:

      psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>"
      

   2. Retrieve the database file location from the database internal configuration.

      SHOW config_file;
      

      Postgres returns the path to your configuration file. For example:

      --------------------------------------------
      /home/postgres/pgdata/data/postgresql.conf
      (1 row)
      

2. Open the config file, then edit your Postgres configuration

   vi /home/postgres/pgdata/data/postgresql.conf
   

3. Save your updated configuration

   When you have saved the changes you make to the configuration file, the new configuration is not applied immediately. The configuration file is automatically reloaded when the server receives a SIGHUP signal. To manually reload the file, use the pg_ctl command.

Setting parameters at the command prompt

If you don't want to open the configuration file to make changes, you can also set parameters directly from the command prompt, using the postgres command. For example:

postgres -c log_connections=yes -c log_destination='syslog'

===== PAGE: https://docs.tigerdata.com/self-hosted/tooling/install-toolkit/ =====

Install and update TimescaleDB Toolkit

Some hyperfunctions are included by default in TimescaleDB. For additional hyperfunctions, you need to install the TimescaleDB Toolkit Postgres extension.

If you're using Tiger Cloud, the TimescaleDB Toolkit is already installed. If you're hosting the TimescaleDB extension on your self-hosted database, you can install Toolkit by:

• Using the TimescaleDB high-availability Docker image
• Using a package manager such as yum, apt, or brew on platforms where pre-built binaries are available
• Building from source. For more information, see the Toolkit developer documentation

Prerequisites

To follow this procedure:

• Install TimescaleDB.
• Add the TimescaleDB repository and the GPG key.

Install TimescaleDB Toolkit

These instructions use the apt package manager.

1. Update your local repository list:

   sudo apt update
   

2. Install TimescaleDB Toolkit:

   sudo apt install timescaledb-toolkit-postgresql-17
   

3. Connect to the database where you want to use Toolkit.

4. Create the Toolkit extension in the database:

   CREATE EXTENSION timescaledb_toolkit;
   

Update TimescaleDB Toolkit

Update Toolkit by installing the latest version and running ALTER EXTENSION.

1. Update your local repository list:

   apt update
   

2. Install the latest version of TimescaleDB Toolkit:

   apt install timescaledb-toolkit-postgresql-17
   

3. Connect to the database where you want to use the new version of Toolkit.

4. Update the Toolkit extension in the database:

   ALTER EXTENSION timescaledb_toolkit UPDATE;
   

   For some Toolkit versions, you might need to disconnect and reconnect active sessions.

Prerequisites

To follow this procedure:

• Install TimescaleDB.
• Add the TimescaleDB repository and the GPG key.

Install TimescaleDB Toolkit

These instructions use the apt package manager.

1. Update your local repository list:

   sudo apt update
   

2. Install TimescaleDB Toolkit:

   sudo apt install timescaledb-toolkit-postgresql-17
   

3. Connect to the database where you want to use Toolkit.

4. Create the Toolkit extension in the database:

   CREATE EXTENSION timescaledb_toolkit;
   

Update TimescaleDB Toolkit

Update Toolkit by installing the latest version and running ALTER EXTENSION.

1. Update your local repository list:

   apt update
   

2. Install the latest version of TimescaleDB Toolkit:

   apt install timescaledb-toolkit-postgresql-17
   

3. Connect to the database where you want to use the new version of Toolkit.

4. Update the Toolkit extension in the database:

   ALTER EXTENSION timescaledb_toolkit UPDATE;
   

   For some Toolkit versions, you might need to disconnect and reconnect active sessions.

Prerequisites

To follow this procedure:

• Install TimescaleDB.
• Create a TimescaleDB repository in your yum repo.d directory.

Install TimescaleDB Toolkit

These instructions use the yum package manager.

1. Set up the repository:

   curl -s https://packagecloud.io/install/repositories/timescale/timescaledb/script.deb.sh | sudo bash
   

2. Update your local repository list:

   yum update
   

3. Install TimescaleDB Toolkit:

   yum install timescaledb-toolkit-postgresql-17
   

4. Connect to the database where you want to use Toolkit.

5. Create the Toolkit extension in the database:

   CREATE EXTENSION timescaledb_toolkit;
   

Update TimescaleDB Toolkit

Update Toolkit by installing the latest version and running ALTER EXTENSION.

1. Update your local repository list:

   yum update
   

2. Install the latest version of TimescaleDB Toolkit:

   yum install timescaledb-toolkit-postgresql-17
   

3. Connect to the database where you want to use the new version of Toolkit.

4. Update the Toolkit extension in the database:

   ALTER EXTENSION timescaledb_toolkit UPDATE;
   

   For some Toolkit versions, you might need to disconnect and reconnect active sessions.

Prerequisites

To follow this procedure:

• Install TimescaleDB.
• Create a TimescaleDB repository in your yum repo.d directory.

Install TimescaleDB Toolkit

These instructions use the yum package manager.

1. Set up the repository:

   curl -s https://packagecloud.io/install/repositories/timescale/timescaledb/script.deb.sh | sudo bash
   

2. Update your local repository list:

   yum update
   

3. Install TimescaleDB Toolkit:

   yum install timescaledb-toolkit-postgresql-17
   

4. Connect to the database where you want to use Toolkit.

5. Create the Toolkit extension in the database:

   CREATE EXTENSION timescaledb_toolkit;
   

Update TimescaleDB Toolkit

Update Toolkit by installing the latest version and running ALTER EXTENSION.

1. Update your local repository list:

   yum update
   

2. Install the latest version of TimescaleDB Toolkit:

   yum install timescaledb-toolkit-postgresql-17
   

3. Connect to the database where you want to use the new version of Toolkit.

4. Update the Toolkit extension in the database:

   ALTER EXTENSION timescaledb_toolkit UPDATE;
   

   For some Toolkit versions, you might need to disconnect and reconnect active sessions.

Install TimescaleDB Toolkit

Best practice for Toolkit installation is to use the TimescaleDB Docker image. To get Toolkit, use the high availability image, timescaledb-ha:

docker pull timescale/timescaledb-ha:pg17

For more information on running TimescaleDB using Docker, see Install TimescaleDB from a Docker container.

Update TimescaleDB Toolkit

To get the latest version of Toolkit, update the TimescaleDB HA docker image.

Prerequisites

To follow this procedure:

• Install TimescaleDB.

Install TimescaleDB Toolkit

These instructions use the brew package manager. For more information on installing or using Homebrew, see the brew homepage.

1. Tap the Tiger Data formula repository, which also contains formulae for TimescaleDB and timescaledb-tune.

   brew tap timescale/tap
   

2. Update your local brew installation:

   brew update
   

3. Install TimescaleDB Toolkit:

   brew install timescaledb-toolkit
   

4. Connect to the database where you want to use Toolkit.

5. Create the Toolkit extension in the database:

   CREATE EXTENSION timescaledb_toolkit;
   

Update TimescaleDB Toolkit

Update Toolkit by installing the latest version and running ALTER EXTENSION.

1. Update your local repository list:

   brew update
   

2. Install the latest version of TimescaleDB Toolkit:

   brew upgrade timescaledb-toolkit
   

3. Connect to the database where you want to use the new version of Toolkit.

4. Update the Toolkit extension in the database:

   ALTER EXTENSION timescaledb_toolkit UPDATE;
   

   For some Toolkit versions, you might need to disconnect and reconnect active sessions.

===== PAGE: https://docs.tigerdata.com/self-hosted/tooling/about-timescaledb-tune/ =====

About timescaledb-tune

Get better performance by tuning your TimescaleDB database to match your system resources and Postgres version. timescaledb-tune is an open source command line tool that analyzes and adjusts your database settings.

Install timescaledb-tune

timescaledb-tune is packaged with binary releases of TimescaleDB. If you installed TimescaleDB from any binary release, including Docker, you already have access. For more install instructions, see the GitHub repository.

Tune your database with timescaledb-tune

Run timescaledb-tune from the command line. The tool analyzes your postgresql.conf file to provide recommendations for memory, parallelism, write-ahead log, and other settings. These changes are written to your postgresql.conf. They take effect on the next restart.

1. At the command line, run timescaledb-tune. To accept all recommendations automatically, include the --yes flag.

   timescaledb-tune
   

2. If you didn't use the --yes flag, respond to each prompt to accept or reject the recommendations.

3. The changes are written to your postgresql.conf.

For detailed instructions and other options, see the documentation in the Github repository.

===== PAGE: https://docs.tigerdata.com/self-hosted/install/installation-windows/ =====

Install TimescaleDB on Windows

TimescaleDB is a Postgres extension for time series and demanding workloads that ingest and query high volumes of data.

This section shows you how to:

• Install and configure TimescaleDB on Postgres: set up a self-hosted Postgres instance to efficiently run TimescaleDB.
• Add the TimescaleDB extension to your database: enable TimescaleDB features and performance improvements on a database.

The following instructions are for development and testing installations. For a production environment, we strongly recommend that you implement the following, many of which you can achieve using Postgres tooling:

• Incremental backup and database snapshots, with efficient point-in-time recovery.
• High availability replication, ideally with nodes across multiple availability zones.
• Automatic failure detection with fast restarts, for both non-replicated and replicated deployments.
• Asynchronous replicas for scaling reads when needed.
• Connection poolers for scaling client connections.
• Zero-down-time minor version and extension upgrades.
• Forking workflows for major version upgrades and other feature testing.
• Monitoring and observability.

Deploying for production? With a Tiger Cloud service we tune your database for performance and handle scalability, high availability, backups, and management, so you can relax.

Prerequisites

To install TimescaleDB on your Windows device, you need:

• OpenSSL v3.x

  For TimescaleDB v2.14.1 only, you need to install OpenSSL v1.1.1.

• Visual C++ Redistributable for Visual Studio 2015

Install and configure TimescaleDB on Postgres

This section shows you how to install the latest version of Postgres and TimescaleDB on a supported platform using the packages supplied by Tiger Data.

If you have previously installed Postgres without a package manager, you may encounter errors following these install instructions. Best practice is to full remove any existing Postgres installations before you begin.

To keep your current Postgres installation, Install from source.

1. Install the latest version of Postgres and psql

   1. Download Postgres, then run the installer.

      1. In the Select Components dialog, check Command Line Tools, along with any other components you want to install, and click Next.

      2. Complete the installation wizard.

   2. Check that you can run pg_config. If you cannot run pg_config from the command line, in the Windows Search tool, enter system environment variables. The path should be C:\Program Files\PostgreSQL\<version>\bin.

2. Install TimescaleDB

   1. Unzip the TimescaleDB installer to <install_dir>, that is, your selected directory.

      Best practice is to use the latest version.

   2. In <install_dir>\timescaledb, right-click setup.exe, then choose Run as Administrator.

   3. Complete the installation wizard.

      If you see an error like could not load library "C:/Program Files/PostgreSQL/17/lib/timescaledb-2.17.2.dll": The specified module could not be found., use Dependencies to ensure that your system can find the compatible DLLs for this release of TimescaleDB.

3. Tune your Postgres instance for TimescaleDB

   Run the timescaledb-tune script included in the timescaledb-tools package with TimescaleDB. For more information, see configuration.

4. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

5. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

Add the TimescaleDB extension to your database

For improved performance, you enable TimescaleDB on each database on your self-hosted Postgres instance. This section shows you how to enable TimescaleDB for a new database in Postgres using psql from the command line.

1. Connect to a database on your Postgres instance

   In Postgres, the default user and database are both postgres. To use a different database, set <database-name> to the name of that database:

   psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>"
   

2. Add TimescaleDB to the database

   CREATE EXTENSION IF NOT EXISTS timescaledb;
   

3. Check that TimescaleDB is installed

   \dx
   

   You see the list of installed extensions:

   List of installed extensions
   Name     | Version |   Schema   |                                      Description
   -------------+---------+------------+---------------------------------------------------------------------------------------
   plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
   timescaledb | 2.17.2  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
   

   Press q to exit the list of extensions.

And that is it! You have TimescaleDB running on a database on a self-hosted instance of Postgres.

Supported platforms

The latest TimescaleDB releases for Postgres are:

 [Postgres 17: TimescaleDB release](https://github.com/timescale/timescaledb/releases/download/2.21.2/timescaledb-postgresql-17-windows-amd64.zip)

[Postgres 16: TimescaleDB release](https://github.com/timescale/timescaledb/releases/download/2.21.2/timescaledb-postgresql-16-windows-amd64.zip)

[Postgres 15: TimescaleDB release](https://github.com/timescale/timescaledb/releases/download/2.21.2/timescaledb-postgresql-15-windows-amd64.zip)

You can deploy TimescaleDB on the following systems:

Operation system	Version
Microsoft Windows	10, 11
Microsoft Windows Server	2019, 2020

For release information, see the GitHub releases page and the release notes.

Where to next

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/self-hosted/install/installation-cloud-image/ =====

Install TimescaleDB from cloud image

You can install TimescaleDB on a cloud hosting provider, from a pre-built, publicly available machine image. These instructions show you how to use a pre-built Amazon machine image (AMI), on Amazon Web Services (AWS).

The currently available pre-built cloud image is:

• Ubuntu 20.04 Amazon EBS-backed AMI

The TimescaleDB AMI uses Elastic Block Store (EBS) attached volumes. This allows you to store image snapshots, dynamic IOPS configuration, and provides some protection of your data if the EC2 instance goes down. Choose an EC2 instance type that is optimized for EBS attached volumes. For information on choosing the right EBS optimized EC2 instance type, see the AWS instance configuration documentation.

This section shows how to use the AMI from within the AWS EC2 dashboard. However, you can also use the AMI to build an instance using tools like Cloudformation, Terraform, the AWS CLI, or any other AWS deployment tool that supports public AMIs.

Installing TimescaleDB from a pre-build cloud image

1. Make sure you have an Amazon Web Services account, and are signed in to your EC2 dashboard.
2. Navigate to Images → AMIs.
3. In the search bar, change the search to Public images and type Timescale search term to find all available TimescaleDB images.
4. Select the image you want to use, and click Launch instance from image.

After you have completed the installation, connect to your instance and configure your database. For information about connecting to the instance, see the AWS accessing instance documentation. The easiest way to configure your database is to run the timescaledb-tune script, which is included with the timescaledb-tools package. For more information, see the configuration section.

After running the timescaledb-tune script, you need to restart the Postgres service for the configuration changes to take effect. To restart the service, run sudo systemctl restart postgresql.service.

Set up the TimescaleDB extension

When you have Postgres and TimescaleDB installed, connect to your instance and set up the TimescaleDB extension.

1. On your instance, at the command prompt, connect to the Postgres instance as the postgres superuser:

   sudo -u postgres psql
   

2. At the prompt, create an empty database. For example, to create a database called tsdb:

   CREATE database tsdb;
   

3. Connect to the database you created:

   \c tsdb
   

4. Add the TimescaleDB extension:

   CREATE EXTENSION IF NOT EXISTS timescaledb;
   

You can check that the TimescaleDB extension is installed by using the \dx command at the command prompt. It looks like this:

tsdb=# \dx

                                      List of installed extensions
    Name     | Version |   Schema   |                            Description
-------------+---------+------------+-------------------------------------------------------------------
 plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
 timescaledb | 2.1.1   | public     | Enables scalable inserts and complex queries for time-series data
(2 rows)

(END)

Where to next

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/self-hosted/install/installation-macos/ =====

Install TimescaleDB on macOS

TimescaleDB is a Postgres extension for time series and demanding workloads that ingest and query high volumes of data. You can host TimescaleDB on macOS device.

This section shows you how to:

• Install and configure TimescaleDB on Postgres - set up a self-hosted Postgres instance to efficiently run TimescaleDB.
• Add the TimescaleDB extension to your database - enable TimescaleDB features and performance improvements on a database.

The following instructions are for development and testing installations. For a production environment, we strongly recommend that you implement the following, many of which you can achieve using Postgres tooling:

• Incremental backup and database snapshots, with efficient point-in-time recovery.
• High availability replication, ideally with nodes across multiple availability zones.
• Automatic failure detection with fast restarts, for both non-replicated and replicated deployments.
• Asynchronous replicas for scaling reads when needed.
• Connection poolers for scaling client connections.
• Zero-down-time minor version and extension upgrades.
• Forking workflows for major version upgrades and other feature testing.
• Monitoring and observability.

Deploying for production? With a Tiger Cloud service we tune your database for performance and handle scalability, high availability, backups, and management, so you can relax.

Prerequisites

To install TimescaleDB on your MacOS device, you need:

• Postgres: for the latest functionality, install Postgres v16

If you have already installed Postgres using a method other than Homebrew or MacPorts, you may encounter errors following these install instructions. Best practice is to full remove any existing Postgres installations before you begin.

To keep your current Postgres installation, Install from source.

Install and configure TimescaleDB on Postgres

This section shows you how to install the latest version of Postgres and TimescaleDB on a supported platform using the packages supplied by Tiger Data.

1. Install Homebrew, if you don't already have it:

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   

   For more information about Homebrew, including installation instructions, see the Homebrew documentation.

2. At the command prompt, add the TimescaleDB Homebrew tap:

   brew tap timescale/tap
   

3. Install TimescaleDB and psql:

   brew install timescaledb libpq
   

4. Update your path to include psql.

   brew link --force libpq
   

   On Intel chips, the symbolic link is added to /usr/local/bin. On Apple Silicon, the symbolic link is added to /opt/homebrew/bin.

5. Run the timescaledb-tune script to configure your database:

timescaledb-tune --quiet --yes

1. Change to the directory where the setup script is located. It is typically, located at /opt/homebrew/Cellar/timescaledb/<VERSION>/bin/, where <VERSION> is the version of timescaledb that you installed:

cd /opt/homebrew/Cellar/timescaledb/<VERSION>/bin/

1. Run the setup script to complete installation.

   ./timescaledb_move.sh
   

2. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

3. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

4. Install MacPorts by downloading and running the package installer.

   For more information about MacPorts, including installation instructions, see the MacPorts documentation.

5. Install TimescaleDB and psql:

   sudo port install timescaledb libpqxx
   

   To view the files installed, run:

   port contents timescaledb libpqxx
   

   MacPorts does not install the timescaledb-tools package or run the timescaledb-tune script. For more information about tuning your database, see the TimescaleDB tuning tool.

6. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

7. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

Add the TimescaleDB extension to your database

For improved performance, you enable TimescaleDB on each database on your self-hosted Postgres instance. This section shows you how to enable TimescaleDB for a new database in Postgres using psql from the command line.

1. Connect to a database on your Postgres instance

   In Postgres, the default user and database are both postgres. To use a different database, set <database-name> to the name of that database:

   psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>"
   

2. Add TimescaleDB to the database

   CREATE EXTENSION IF NOT EXISTS timescaledb;
   

3. Check that TimescaleDB is installed

   \dx
   

   You see the list of installed extensions:

   List of installed extensions
   Name     | Version |   Schema   |                                      Description
   -------------+---------+------------+---------------------------------------------------------------------------------------
   plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
   timescaledb | 2.17.2  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
   

   Press q to exit the list of extensions.

And that is it! You have TimescaleDB running on a database on a self-hosted instance of Postgres.

Supported platforms

You can deploy TimescaleDB on the following systems:

Operation system	Version
macOS	From 10.15 Catalina to 14 Sonoma

For the latest functionality, install MacOS 14 Sonoma.

Where to next

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/self-hosted/install/installation-kubernetes/ =====

Install TimescaleDB on Kubernetes

You can run TimescaleDB inside Kubernetes using the TimescaleDB Docker container images.

The following instructions are for development and testing installations. For a production environment, we strongly recommend that you implement the following, many of which you can achieve using Postgres tooling:

• Incremental backup and database snapshots, with efficient point-in-time recovery.
• High availability replication, ideally with nodes across multiple availability zones.
• Automatic failure detection with fast restarts, for both non-replicated and replicated deployments.
• Asynchronous replicas for scaling reads when needed.
• Connection poolers for scaling client connections.
• Zero-down-time minor version and extension upgrades.
• Forking workflows for major version upgrades and other feature testing.
• Monitoring and observability.

Deploying for production? With a Tiger Cloud service we tune your database for performance and handle scalability, high availability, backups, and management, so you can relax.

Prerequisites

To follow the steps on this page:

• Install self-managed Kubernetes or sign up for a Kubernetes Turnkey Cloud Solution.
• Install kubectl for command-line interaction with your cluster.

Integrate TimescaleDB in a Kubernetes cluster

Running TimescaleDB on Kubernetes is similar to running Postgres. This procedure outlines the steps for a non-distributed system.

To connect your Kubernetes cluster to self-hosted TimescaleDB running in the cluster:

1. Create a default namespace for Tiger Data components

   1. Create the Tiger Data namespace:

      kubectl create namespace timescale
      

   2. Set this namespace as the default for your session:

      kubectl config set-context --current --namespace=timescale
      

   For more information, see Kubernetes Namespaces.

2. Set up a persistent volume claim (PVC) storage

   To manually set up a persistent volume and claim for self-hosted Kubernetes, run the following command:

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: timescale-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 10Gi
   EOF
   

3. Deploy TimescaleDB as a StatefulSet

   By default, the TimescaleDB Docker image you are installing on Kubernetes uses the default Postgres database, user and password. To deploy TimescaleDB on Kubernetes, run the following command:

   kubectl apply -f - <<EOF
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: timescaledb
   spec:
     serviceName: timescaledb
     replicas: 1
     selector:
       matchLabels:
         app: timescaledb
     template:
       metadata:
         labels:
           app: timescaledb
       spec:
         containers:
           - name: timescaledb
             image: 'timescale/timescaledb:latest-pg17'
             env:
               - name: POSTGRES_USER
                 value: postgres
               - name: POSTGRES_PASSWORD
                 value: postgres
               - name: POSTGRES_DB
                 value: postgres
               - name: PGDATA
                 value: /var/lib/postgresql/data/pgdata
             ports:
               - containerPort: 5432
             volumeMounts:
               - mountPath: /var/lib/postgresql/data
                 name: timescale-storage
         volumes:
           - name: timescale-storage
             persistentVolumeClaim:
               claimName: timescale-pvc
   EOF
   

4. Allow applications to connect by exposing TimescaleDB within Kubernetes

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: timescaledb
spec:
  selector:
    app: timescaledb
  ports:
    - protocol: TCP
      port: 5432
      targetPort: 5432
  type: ClusterIP
EOF

1. Create a Kubernetes secret to store the database credentials

   kubectl create secret generic timescale-secret \
   --from-literal=PGHOST=timescaledb \
   --from-literal=PGPORT=5432 \
   --from-literal=PGDATABASE=postgres \
   --from-literal=PGUSER=postgres \
   --from-literal=PGPASSWORD=postgres
   

2. Deploy an application that connects to TimescaleDB

   kubectl apply -f - <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: timescale-app
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: timescale-app
     template:
       metadata:
         labels:
           app: timescale-app
       spec:
         containers:
         - name: timescale-container
           image: postgres:latest
           envFrom:
             - secretRef:
                 name: timescale-secret
   EOF
   

3. Test the database connection

   1. Create and run a pod to verify database connectivity using your connection details saved in timescale-secret:

      kubectl run test-pod --image=postgres --restart=Never \
      --env="PGHOST=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGHOST}' | base64 --decode)" \
      --env="PGPORT=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGPORT}' | base64 --decode)" \
      --env="PGDATABASE=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGDATABASE}' | base64 --decode)" \
      --env="PGUSER=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGUSER}' | base64 --decode)" \
      --env="PGPASSWORD=$(kubectl get secret timescale-secret -o=jsonpath='{.data.PGPASSWORD}' | base64 --decode)" \
      -- sleep infinity
      

   2. Launch the Postgres interactive shell within the created test-pod:

      kubectl exec -it test-pod -- bash -c "psql -h \$PGHOST -U \$PGUSER -d \$PGDATABASE"
      

   You see the Postgres interactive terminal.

Install with Postgres Kubernetes operators

You can also use Postgres Kubernetes operators to simplify installation, configuration, and life cycle. The operators which our community members have told us work well are:

• StackGres (includes TimescaleDB images)
• Postgres Operator (Patroni)
• PGO
• CloudNativePG

===== PAGE: https://docs.tigerdata.com/self-hosted/install/installation-source/ =====

Install TimescaleDB from source

TimescaleDB is a Postgres extension for time series and demanding workloads that ingest and query high volumes of data. You can install a TimescaleDB instance on any local system, from source.

This section shows you how to:

• Install and configure TimescaleDB on Postgres - set up a self-hosted Postgres instance to efficiently run TimescaleDB1.
• Add the TimescaleDB extension to your database - enable TimescaleDB features and performance improvements on a database.

The following instructions are for development and testing installations. For a production environment, we strongly recommend that you implement the following, many of which you can achieve using Postgres tooling:

• Incremental backup and database snapshots, with efficient point-in-time recovery.
• High availability replication, ideally with nodes across multiple availability zones.
• Automatic failure detection with fast restarts, for both non-replicated and replicated deployments.
• Asynchronous replicas for scaling reads when needed.
• Connection poolers for scaling client connections.
• Zero-down-time minor version and extension upgrades.
• Forking workflows for major version upgrades and other feature testing.
• Monitoring and observability.

Deploying for production? With a Tiger Cloud service we tune your database for performance and handle scalability, high availability, backups, and management, so you can relax.

Prerequisites

To install TimescaleDB from source, you need the following on your developer environment:

• Postgres:

  Install a supported version of Postgres using the Postgres installation instructions.

  We recommend not using TimescaleDB with Postgres 17.1, 16.5, 15.9, 14.14, 13.17, 12.21. These minor versions introduced a breaking binary interface change that, once identified, was reverted in subsequent minor Postgres versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with Postgres 17.2, 16.6, etc and higher. Users of Tiger Cloud and Platform packages built and distributed by Tiger Data are unaffected.

• Build tools:

  • CMake version 3.11 or later

  • C language compiler for your operating system, such as gcc or clang.

    If you are using a Microsoft Windows system, you can install Visual Studio 2015 or later instead of CMake and a C language compiler. Ensure you install the Visual Studio components for CMake and Git when you run the installer.

Install and configure TimescaleDB on Postgres

This section shows you how to install the latest version of Postgres and TimescaleDB on a supported platform using source supplied by Tiger Data.

1. Install the latest Postgres source

   1. At the command prompt, clone the TimescaleDB GitHub repository:

      git clone https://github.com/timescale/timescaledb
      

   2. Change into the cloned directory:

      cd timescaledb
      

   3. Checkout the latest release. You can find the latest release tag on our Releases page:

      git checkout 2.17.2
      

      This command produces an error that you are now in detached head state. It is expected behavior, and it occurs because you have checked out a tag, and not a branch. Continue with the steps in this procedure as normal.

2. Build the source

   1. Bootstrap the build system:

      ./bootstrap
      

      bootstrap.bat
      

      For installation on Microsoft Windows, you might need to add the pg_config and cmake file locations to your path. In the Windows Search tool, search for system environment variables. The path for pg_config should be C:\Program Files\PostgreSQL\<version>\bin. The path for cmake is within the Visual Studio directory.

   2. Build the extension:

      cd build && make
      

      cmake --build ./build --config Release
      

3. Install TimescaleDB

   make install
   

   cmake --build ./build --config Release --target install
   

4. Configure Postgres

   If you have more than one version of Postgres installed, TimescaleDB can only be associated with one of them. The TimescaleDB build scripts use pg_config to find out where Postgres stores its extension files, so you can use pg_config to find out which Postgres installation TimescaleDB is using.

   1. Locate the postgresql.conf configuration file:

      psql -d postgres -c "SHOW config_file;"
      

   2. Open the postgresql.conf file and update shared_preload_libraries to:

      shared_preload_libraries = 'timescaledb'
      

      If you use other preloaded libraries, make sure they are comma separated.

   3. Tune your Postgres instance for TimescaleDB

      sudo timescaledb-tune
      

      This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

   4. Restart the Postgres instance:

      service postgresql restart
      

      pg_ctl restart
      

5. Set the user password

   1. Log in to Postgres as postgres

      sudo -u postgres psql
      

      You are in the psql shell.

   2. Set the password for postgres

      \password postgres
      

      When you have set the password, type \q to exit psql.

Add the TimescaleDB extension to your database

For improved performance, you enable TimescaleDB on each database on your self-hosted Postgres instance. This section shows you how to enable TimescaleDB for a new database in Postgres using psql from the command line.

1. Connect to a database on your Postgres instance

   In Postgres, the default user and database are both postgres. To use a different database, set <database-name> to the name of that database:

   psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>"
   

2. Add TimescaleDB to the database

   CREATE EXTENSION IF NOT EXISTS timescaledb;
   

3. Check that TimescaleDB is installed

   \dx
   

   You see the list of installed extensions:

   List of installed extensions
   Name     | Version |   Schema   |                                      Description
   -------------+---------+------------+---------------------------------------------------------------------------------------
   plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
   timescaledb | 2.17.2  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
   

   Press q to exit the list of extensions.

And that is it! You have TimescaleDB running on a database on a self-hosted instance of Postgres.

Where to next

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/self-hosted/install/installation-linux/ =====

Install TimescaleDB on Linux

TimescaleDB is a Postgres extension for time series and demanding workloads that ingest and query high volumes of data.

This section shows you how to:

• Install and configure TimescaleDB on Postgres - set up a self-hosted Postgres instance to efficiently run TimescaleDB.
• Add the TimescaleDB extension to your database - enable TimescaleDB features and performance improvements on a database.

The following instructions are for development and testing installations. For a production environment, we strongly recommend that you implement the following, many of which you can achieve using Postgres tooling:

• Incremental backup and database snapshots, with efficient point-in-time recovery.
• High availability replication, ideally with nodes across multiple availability zones.
• Automatic failure detection with fast restarts, for both non-replicated and replicated deployments.
• Asynchronous replicas for scaling reads when needed.
• Connection poolers for scaling client connections.
• Zero-down-time minor version and extension upgrades.
• Forking workflows for major version upgrades and other feature testing.
• Monitoring and observability.

Deploying for production? With a Tiger Cloud service we tune your database for performance and handle scalability, high availability, backups, and management, so you can relax.

Install and configure TimescaleDB on Postgres

This section shows you how to install the latest version of Postgres and TimescaleDB on a supported platform using the packages supplied by Tiger Data.

If you have previously installed Postgres without a package manager, you may encounter errors following these install instructions. Best practice is to fully remove any existing Postgres installations before you begin.

To keep your current Postgres installation, Install from source.

1. Install the latest Postgres packages

   sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget
   

2. Run the Postgres package setup script

   sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
   

3. Add the TimescaleDB package

   echo "deb https://packagecloud.io/timescale/timescaledb/debian/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
   

4. Install the TimescaleDB GPG key

   wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg
   

5. Update your local repository list

   sudo apt update
   

6. Install TimescaleDB

   sudo apt install timescaledb-2-postgresql-17 postgresql-client-17
   

   To install a specific TimescaleDB release, set the version. For example:

   sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'

   Older versions of TimescaleDB may not support all the OS versions listed on this page.

7. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune
   

   By default, this script is included with the timescaledb-tools package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see Configuration. If you have an issue, run sudo apt install timescaledb-tools.

8. Restart Postgres

   sudo systemctl restart postgresql
   

9. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

10. Set the password for postgres

    \password postgres
    

    When you have set the password, type \q to exit psql.

11. Install the latest Postgres packages

    sudo apt install gnupg postgresql-common apt-transport-https lsb-release wget
    

12. Run the Postgres package setup script

    sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
    

    echo "deb https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main" | sudo tee /etc/apt/sources.list.d/timescaledb.list
    

13. Install the TimescaleDB GPG key

    wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/timescaledb.gpg
    

    For Ubuntu 21.10 and earlier use the following command:

    wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add -

14. Update your local repository list

    sudo apt update
    

15. Install TimescaleDB

    sudo apt install timescaledb-2-postgresql-17 postgresql-client-17
    

    To install a specific TimescaleDB release, set the version. For example:

    sudo apt-get install timescaledb-2-postgresql-14='2.6.0*' timescaledb-2-loader-postgresql-14='2.6.0*'

    Older versions of TimescaleDB may not support all the OS versions listed on this page.

16. Tune your Postgres instance for TimescaleDB

    sudo timescaledb-tune
    

    By default, this script is included with the timescaledb-tools package when you install TimescaleDB. Use the prompts to tune your development or production environment. For more information on manual configuration, see Configuration. If you have an issue, run sudo apt install timescaledb-tools.

17. Restart Postgres

    sudo systemctl restart postgresql
    

18. Log in to Postgres as postgres

    sudo -u postgres psql
    

    You are in the psql shell.

19. Set the password for postgres

    \password postgres
    

    When you have set the password, type \q to exit psql.

20. Install the latest Postgres packages

    sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-$(rpm -E %{rhel})-x86_64/pgdg-redhat-repo-latest.noarch.rpm
    

21. Add the TimescaleDB repository

    sudo tee /etc/yum.repos.d/timescale_timescaledb.repo <<EOL
    [timescale_timescaledb]
    name=timescale_timescaledb
    baseurl=https://packagecloud.io/timescale/timescaledb/el/$(rpm -E %{rhel})/\$basearch
    repo_gpgcheck=1
    gpgcheck=0
    enabled=1
    gpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey
    sslverify=1
    sslcacert=/etc/pki/tls/certs/ca-bundle.crt
    metadata_expire=300
    EOL
    

22. Update your local repository list

    sudo yum update
    

23. Install TimescaleDB

    To avoid errors, do not install TimescaleDB Apache 2 Edition and TimescaleDB Community Edition at the same time.

    sudo yum install timescaledb-2-postgresql-17 postgresql17
    

    On Red Hat Enterprise Linux 8 and later, disable the built-in Postgres module:

    sudo dnf -qy module disable postgresql

24. Initialize the Postgres instance

```bash
sudo /usr/pgsql-17/bin/postgresql-17-setup initdb
```

1. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune --pg-config=/usr/pgsql-17/bin/pg_config
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

2. Enable and start Postgres

   sudo systemctl enable postgresql-17
   sudo systemctl start postgresql-17
   

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are now in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

5. Install the latest Postgres packages

   sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/F-$(rpm -E %{fedora})-x86_64/pgdg-fedora-repo-latest.noarch.rpm
   

6. Add the TimescaleDB repository

   sudo tee /etc/yum.repos.d/timescale_timescaledb.repo <<EOL
   [timescale_timescaledb]
   name=timescale_timescaledb
   baseurl=https://packagecloud.io/timescale/timescaledb/el/9/\$basearch
   repo_gpgcheck=1
   gpgcheck=0
   enabled=1
   gpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey
   sslverify=1
   sslcacert=/etc/pki/tls/certs/ca-bundle.crt
   metadata_expire=300
   EOL
   

7. Update your local repository list

   sudo yum update
   

8. Install TimescaleDB

   To avoid errors, do not install TimescaleDB Apache 2 Edition and TimescaleDB Community Edition at the same time.

   sudo yum install timescaledb-2-postgresql-17 postgresql17
   

   On Red Hat Enterprise Linux 8 and later, disable the built-in Postgres module:

   sudo dnf -qy module disable postgresql

9. Initialize the Postgres instance

```bash
sudo /usr/pgsql-17/bin/postgresql-17-setup initdb
```

1. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune --pg-config=/usr/pgsql-17/bin/pg_config
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

2. Enable and start Postgres

   sudo systemctl enable postgresql-17
   sudo systemctl start postgresql-17
   

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are now in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

Tiger Data supports Rocky Linux 8 and 9 on amd64 only.

1. Update your local repository list

   sudo dnf update -y
   sudo dnf install -y epel-release
   

2. Install the latest Postgres packages

   sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
   

3. Add the TimescaleDB repository

   sudo tee /etc/yum.repos.d/timescale_timescaledb.repo <<EOL
   [timescale_timescaledb]
   name=timescale_timescaledb
   baseurl=https://packagecloud.io/timescale/timescaledb/el/9/\$basearch
   repo_gpgcheck=1
   gpgcheck=0
   enabled=1
   gpgkey=https://packagecloud.io/timescale/timescaledb/gpgkey
   sslverify=1
   sslcacert=/etc/pki/tls/certs/ca-bundle.crt
   metadata_expire=300
   EOL
   

4. Disable the built-in PostgreSQL module

   This is for Rocky Linux 9 only.

   sudo dnf module disable postgresql -y
   

5. Install TimescaleDB

   To avoid errors, do not install TimescaleDB Apache 2 Edition and TimescaleDB Community Edition at the same time.

   sudo dnf install -y postgresql16-server postgresql16-contrib timescaledb-2-postgresql-16
   

6. Initialize the Postgres instance

```bash
sudo /usr/pgsql-16/bin/postgresql-16-setup initdb
```

1. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune --pg-config=/usr/pgsql-16/bin/pg_config
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

2. Enable and start Postgres

   sudo systemctl enable postgresql-16
   sudo systemctl start postgresql-16
   

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are now in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

ArchLinux packages are built by the community.

1. Install the latest Postgres and TimescaleDB packages

   sudo pacman -Syu timescaledb timescaledb-tune postgresql-libs
   

2. Initalize your Postgres instance

   sudo -u postgres initdb --locale=en_US.UTF-8 --encoding=UTF8 -D /var/lib/postgres/data --data-checksums
   

3. Tune your Postgres instance for TimescaleDB

   sudo timescaledb-tune
   

   This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

4. Enable and start Postgres

   sudo systemctl enable postgresql.service
   sudo systemctl start postgresql.service
   

5. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

6. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

Job jobbed, you have installed Postgres and TimescaleDB.

Add the TimescaleDB extension to your database

For improved performance, you enable TimescaleDB on each database on your self-hosted Postgres instance. This section shows you how to enable TimescaleDB for a new database in Postgres using psql from the command line.

1. Connect to a database on your Postgres instance

   In Postgres, the default user and database are both postgres. To use a different database, set <database-name> to the name of that database:

   psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>"
   

2. Add TimescaleDB to the database

   CREATE EXTENSION IF NOT EXISTS timescaledb;
   

3. Check that TimescaleDB is installed

   \dx
   

   You see the list of installed extensions:

   List of installed extensions
   Name     | Version |   Schema   |                                      Description
   -------------+---------+------------+---------------------------------------------------------------------------------------
   plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
   timescaledb | 2.17.2  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
   

   Press q to exit the list of extensions.

And that is it! You have TimescaleDB running on a database on a self-hosted instance of Postgres.

Supported platforms

You can deploy TimescaleDB on the following systems:

Operation system	Version
Debian	13 Trixe, 12 Bookworm, 11 Bullseye
Ubuntu	24.04 Noble Numbat, 22.04 LTS Jammy Jellyfish
Red Hat Enterprise	Linux 9, Linux 8
Fedora	Fedora 35, Fedora 34, Fedora 33
Rocky Linux	Rocky Linux 9 (x86_64), Rocky Linux 8
ArchLinux (community-supported)	Check the available packages

Where to next

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/self-hosted/install/self-hosted/ =====

Install self-hosted TimescaleDB

Installation

Refer to the installation documentation for detailed setup instructions.

===== PAGE: https://docs.tigerdata.com/self-hosted/install/installation-docker/ =====

Install TimescaleDB on Docker

TimescaleDB is a Postgres extension for time series and demanding workloads that ingest and query high volumes of data. You can install a TimescaleDB instance on any local system from a pre-built Docker container.

This section shows you how to Install and configure TimescaleDB on Postgres.

The following instructions are for development and testing installations. For a production environment, we strongly recommend that you implement the following, many of which you can achieve using Postgres tooling:

• Incremental backup and database snapshots, with efficient point-in-time recovery.
• High availability replication, ideally with nodes across multiple availability zones.
• Automatic failure detection with fast restarts, for both non-replicated and replicated deployments.
• Asynchronous replicas for scaling reads when needed.
• Connection poolers for scaling client connections.
• Zero-down-time minor version and extension upgrades.
• Forking workflows for major version upgrades and other feature testing.
• Monitoring and observability.

Deploying for production? With a Tiger Cloud service we tune your database for performance and handle scalability, high availability, backups, and management, so you can relax.

Prerequisites

To run, and connect to a Postgres installation on Docker, you need to install:

• Docker
• psql

Install and configure TimescaleDB on Postgres

This section shows you how to install the latest version of Postgres and TimescaleDB on a supported platform using containers supplied by Tiger Data.

1. Run the TimescaleDB Docker image

   The TimescaleDB HA Docker image offers the most complete TimescaleDB experience. It uses Ubuntu, includes TimescaleDB Toolkit, and support for PostGIS and Patroni.

   To install the latest release based on Postgres 17:

   docker pull timescale/timescaledb-ha:pg17
   

   TimescaleDB is pre-created in the default Postgres database and is added by default to any new database you create in this image.

2. Run the container

   Replace </a/local/data/folder> with the path to the folder you want to keep your data in the following command.

   docker run -d --name timescaledb -p 5432:5432  -v </a/local/data/folder>:/pgdata -e PGDATA=/pgdata -e POSTGRES_PASSWORD=password timescale/timescaledb-ha:pg17
   

   If you are running multiple container instances, change the port each Docker instance runs on.

   On UNIX-based systems, Docker modifies Linux IP tables to bind the container. If your system uses Linux Uncomplicated Firewall (UFW), Docker may override your UFW port binding settings. To prevent this, add DOCKER_OPTS="--iptables=false" to /etc/default/docker.

3. Connect to a database on your Postgres instance

   The default user and database are both postgres. You set the password in POSTGRES_PASSWORD in the previous step. The default command to connect to Postgres is:

   psql -d "postgres://postgres:password@localhost/postgres"
   

4. Check that TimescaleDB is installed

   \dx
   

   You see the list of installed extensions:

   Name         | Version |   Schema   |                                      Description
   ---------------------+---------+------------+---------------------------------------------------------------------------------------
   plpgsql             | 1.0     | pg_catalog | PL/pgSQL procedural language
   timescaledb         | 2.20.3  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
   timescaledb_toolkit | 1.21.0  | public     | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities
   (3 rows)
   

   Press q to exit the list of extensions.

More Docker options

If you want to access the container from the host but avoid exposing it to the outside world, you can bind to 127.0.0.1 instead of the public interface, using this command:

docker run -d --name timescaledb -p 127.0.0.1:5432:5432 \
-v </a/local/data/folder>:/pgdata -e PGDATA=/pgdata -e POSTGRES_PASSWORD=password timescale/timescaledb-ha:pg17

If you don't want to install psql and other Postgres client tools locally, or if you are using a Microsoft Windows host system, you can connect using the version of psql that is bundled within the container with this command:

docker exec -it timescaledb psql -U postgres

When you install TimescaleDB using a Docker container, the Postgres settings are inherited from the container. In most cases, you do not need to adjust them. However, if you need to change a setting, you can add -c setting=value to your Docker run command. For more information, see the Docker documentation.

The link provided in these instructions is for the latest version of TimescaleDB on Postgres 17. To find other Docker tags you can use, see the Dockerhub repository.

View logs in Docker

If you have TimescaleDB installed in a Docker container, you can view your logs using Docker, instead of looking in /var/lib/logs or /var/logs. For more information, see the Docker documentation on logs.

1. Run the TimescaleDB Docker image

   The light-weight TimescaleDB Docker image uses Alpine and does not contain TimescaleDB Toolkit or support for PostGIS and Patroni.

   To install the latest release based on Postgres 17:

   docker pull timescale/timescaledb:latest-pg17
   

   TimescaleDB is pre-created in the default Postgres database and added by default to any new database you create in this image.

2. Run the container

   docker run -v </a/local/data/folder>:/pgdata -e PGDATA=/pgdata \
       -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg17
   

   If you are running multiple container instances, change the port each Docker instance runs on.

   On UNIX-based systems, Docker modifies Linux IP tables to bind the container. If your system uses Linux Uncomplicated Firewall (UFW), Docker may override your UFW port binding settings. To prevent this, add DOCKER_OPTS="--iptables=false" to /etc/default/docker.

3. Connect to a database on your Postgres instance

   The default user and database are both postgres. You set the password in POSTGRES_PASSWORD in the previous step. The default command to connect to Postgres in this image is:

   psql -d "postgres://postgres:password@localhost/postgres"
   

4. Check that TimescaleDB is installed

   \dx
   

You see the list of installed extensions:

```sql
Name         | Version |   Schema   |                                      Description
---------------------+---------+------------+---------------------------------------------------------------------------------------
plpgsql             | 1.0     | pg_catalog | PL/pgSQL procedural language
timescaledb         | 2.20.3  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
```

Press `q` to exit the list of extensions.

More Docker options

If you want to access the container from the host but avoid exposing it to the outside world, you can bind to 127.0.0.1 instead of the public interface, using this command:

docker run -v </a/local/data/folder>:/pgdata -e PGDATA=/pgdata \
  -d --name timescaledb -p 127.0.0.1:5432:5432 \
  -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg17

If you don't want to install psql and other Postgres client tools locally, or if you are using a Microsoft Windows host system, you can connect using the version of psql that is bundled within the container with this command:

docker exec -it timescaledb psql -U postgres

Existing containers can be stopped using docker stop and started again with docker start while retaining their volumes and data. When you create a new container using the docker run command, by default you also create a new data volume. When you remove a Docker container with docker rm, the data volume persists on disk until you explicitly delete it. You can use the docker volume ls command to list existing docker volumes. If you want to store the data from your Docker container in a host directory, or you want to run the Docker image on top of an existing data directory, you can specify the directory to mount a data volume using the -v flag:

docker run -d --name timescaledb -p 5432:5432 \
-v </your/data/dir>:/pgdata -e PGDATA=/pgdata \
-e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg17

When you install TimescaleDB using a Docker container, the Postgres settings are inherited from the container. In most cases, you do not need to adjust them. However, if you need to change a setting, you can add -c setting=value to your Docker run command. For more information, see the Docker documentation.

The link provided in these instructions is for the latest version of TimescaleDB on Postgres 16. To find other Docker tags you can use, see the Dockerhub repository.

View logs in Docker

If you have TimescaleDB installed in a Docker container, you can view your logs using Docker, instead of looking in /var/log. For more information, see the Docker documentation on logs.

And that is it! You have TimescaleDB running on a database on a self-hosted instance of Postgres.

Where to next

What next? Try the key features offered by Tiger Data, see the tutorials, interact with the data in your Tiger Cloud service using your favorite programming language, integrate your Tiger Cloud service with a range of third-party tools, plain old Use Tiger Data products, or dive into the API reference.

===== PAGE: https://docs.tigerdata.com/self-hosted/replication-and-ha/configure-replication/ =====

Configure replication

This section outlines how to set up asynchronous streaming replication on one or more database replicas.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

Before you begin, make sure you have at least two separate instances of TimescaleDB running. If you installed TimescaleDB using a Docker container, use a Postgres entry point script to run the configuration. For more advanced examples, see the TimescaleDB Helm Charts repository.

To configure replication on self-hosted TimescaleDB, you need to perform these procedures:

1. Configure the primary database
2. Configure replication parameters
3. Create replication slots
4. Configure host-based authentication parameters
5. Create a base backup on the replica
6. Configure replication and recovery settings
7. Verify that the replica is working

Configure the primary database

To configure the primary database, you need a Postgres user with a role that allows it to initialize streaming replication. This is the user each replica uses to stream from the primary database.

Configuring the primary database

1. On the primary database, as a user with superuser privileges, such as the postgres user, set the password encryption level to scram-sha-256:

   SET password_encryption = 'scram-sha-256';
   

2. Create a new user called repuser:

   CREATE ROLE repuser WITH REPLICATION PASSWORD '<PASSWORD>' LOGIN;
   

The scram-sha-256 encryption level is the most secure password-based authentication available in Postgres. It is only available in Postgres 10 and later.

Configure replication parameters

There are several replication settings that need to be added or edited in the postgresql.conf configuration file.

Configuring replication parameters

1. Set the synchronous_commit parameter to off.
2. Set the max_wal_senders parameter to the total number of concurrent connections from replicas or backup clients. As a minimum, this should equal the number of replicas you intend to have.
3. Set the wal_level parameter to the amount of information written to the Postgres write-ahead log (WAL). For replication to work, there needs to be enough data in the WAL to support archiving and replication. The default value is usually appropriate.
4. Set the max_replication_slots parameter to the total number of replication slots the primary database can support.
5. Set the listen_addresses parameter to the address of the primary database. Do not leave this parameter as the local loopback address, because the remote replicas must be able to connect to the primary to stream the WAL.
6. Restart Postgres to pick up the changes. This must be done before you create replication slots.

The most common streaming replication use case is asynchronous replication with one or more replicas. In this example, the WAL is streamed to the replica, but the primary server does not wait for confirmation that the WAL has been written to disk on either the primary or the replica. This is the most performant replication configuration, but it does carry the risk of a small amount of data loss in the event of a system failure. It also makes no guarantees that the replica is fully up to date with the primary, which could cause inconsistencies between read queries on the primary and the replica. The example configuration for this use case:

listen_addresses = '*'
wal_level = replica
max_wal_senders = 2
max_replication_slots = 2
synchronous_commit = off

If you need stronger consistency on the replicas, or if your query load is heavy enough to cause significant lag between the primary and replica nodes in asynchronous mode, consider a synchronous replication configuration instead. For more information about the different replication modes, see the replication modes section.

Create replication slots

When you have configured postgresql.conf and restarted Postgres, you can create a replication slot for each replica. Replication slots ensure that the primary does not delete segments from the WAL until they have been received by the replicas. This is important in case a replica goes down for an extended time. The primary needs to verify that a WAL segment has been consumed by a replica, so that it can safely delete data. You can use archiving for this purpose, but replication slots provide the strongest protection for streaming replication.

Creating replication slots

1. At the psql slot, create the first replication slot. The name of the slot is arbitrary. In this example, it is called replica_1_slot:

   SELECT * FROM pg_create_physical_replication_slot('replica_1_slot', true);
   

2. Repeat for each required replication slot.

Configure host-based authentication parameters

There are several replication settings that need to be added or edited to the pg_hba.conf configuration file. In this example, the settings restrict replication connections to traffic coming from REPLICATION_HOST_IP as the Postgres user repuser with a valid password. REPLICATION_HOST_IP can initiate streaming replication from that machine without additional credentials. You can change the address and method values to match your security and network settings.

For more information about pg_hba.conf, see the pg_hba documentation.

Configuring host-based authentication parameters

1. Open the pg_hba.conf configuration file and add or edit this line:

   TYPE  DATABASE    USER    ADDRESS METHOD            AUTH_METHOD
   host  replication repuser <REPLICATION_HOST_IP>/32  scram-sha-256
   

2. Restart Postgres to pick up the changes.

Create a base backup on the replica

Replicas work by streaming the primary server's WAL log and replaying its transactions in Postgres recovery mode. To do this, the replica needs to be in a state where it can replay the log. You can do this by restoring the replica from a base backup of the primary instance.

Creating a base backup on the replica

1. Stop Postgres services.

2. If the replica database already contains data, delete it before you run the backup, by removing the Postgres data directory:

   rm -rf <DATA_DIRECTORY>/*
   

   If you don't know the location of the data directory, find it with the show data_directory; command.

3. Restore from the base backup, using the IP address of the primary database and the replication username:

   pg_basebackup -h <PRIMARY_IP> \
   -D <DATA_DIRECTORY> \
   -U repuser -vP -W
   

   The -W flag prompts you for a password. If you are using this command in an automated setup, you might need to use a pgpass file.

4. When the backup is complete, create a standby.signal file in your data directory. When Postgres finds a standby.signal file in its data directory, it starts in recovery mode and streams the WAL through the replication protocol:

   touch <DATA_DIRECTORY>/standby.signal
   

Configure replication and recovery settings

When you have successfully created a base backup and a standby.signal file, you can configure the replication and recovery settings.

Configuring replication and recovery settings

1. In the replica's postgresql.conf file, add details for communicating with the primary server. If you are using streaming replication, the application_name in primary_conninfo should be the same as the name used in the primary's synchronous_standby_names settings:

   primary_conninfo = 'host=<PRIMARY_IP> port=5432 user=repuser
   password=<POSTGRES_USER_PASSWORD> application_name=r1'
   primary_slot_name = 'replica_1_slot'
   

2. Add details to mirror the configuration of the primary database. If you are using asynchronous replication, use these settings:

   hot_standby = on
   wal_level = replica
   max_wal_senders = 2
   max_replication_slots = 2
   synchronous_commit = off
   

   The hot_standby parameter must be set to on to allow read-only queries on the replica. In Postgres 10 and later, this setting is on by default.

3. Restart Postgres to pick up the changes.

Verify that the replica is working

At this point, your replica should be fully synchronized with the primary database and prepared to stream from it. You can verify that it is working properly by checking the logs on the replica, which should look like this:

LOG:  database system was shut down in recovery at 2018-03-09 18:36:23 UTC
LOG:  entering standby mode
LOG:  redo starts at 0/2000028
LOG:  consistent recovery state reached at 0/3000000
LOG:  database system is ready to accept read only connections
LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

Any client can perform reads on the replica. You can verify this by running inserts, updates, or other modifications to your data on the primary database, and then querying the replica to ensure they have been properly copied over.

Replication modes

In most cases, asynchronous streaming replication is sufficient. However, you might require greater consistency between the primary and replicas, especially if you have a heavy workload. Under heavy workloads, replicas can lag far behind the primary, providing stale data to clients reading from the replicas. Additionally, in cases where any data loss is fatal, asynchronous replication might not provide enough of a durability guarantee. The Postgres synchronous_commit feature has several options with varying consistency and performance tradeoffs.

In the postgresql.conf file, set the synchronous_commit parameter to:

• on: This is the default value. The server does not return success until the WAL transaction has been written to disk on the primary and any replicas.
• off: The server returns success when the WAL transaction has been sent to the operating system to write to the WAL on disk on the primary, but does not wait for the operating system to actually write it. This can cause a small amount of data loss if the server crashes when some data has not been written, but it does not result in data corruption. Turning synchronous_commit off is a well-known Postgres optimization for workloads that can withstand some data loss in the event of a system crash.
• local: Enforces on behavior only on the primary server.
• remote_write: The database returns success to a client when the WAL record has been sent to the operating system for writing to the WAL on the replicas, but before confirmation that the record has actually been persisted to disk. This is similar to asynchronous commit, except it waits for the replicas as well as the primary. In practice, the extra wait time incurred waiting for the replicas significantly decreases replication lag.
• remote_apply: Requires confirmation that the WAL records have been written to the WAL and applied to the databases on all replicas. This provides the strongest consistency of any of the synchronous_commit options. In this mode, replicas always reflect the latest state of the primary, and replication lag is nearly non-existent.

If synchronous_standby_names is empty, the settings on, remote_apply, remote_write and local all provide the same synchronization level, and transaction commits wait for the local flush to disk.

This matrix shows the level of consistency provided by each mode:

Mode	WAL Sent to OS (Primary)	WAL Persisted (Primary)	WAL Sent to OS (Primary & Replicas)	WAL Persisted (Primary & Replicas)	Transaction Applied (Primary & Replicas)
Off	✅	❌	❌	❌	❌
Local	✅	✅	❌	❌	❌
Remote Write	✅	✅	✅	❌	❌
On	✅	✅	✅	✅	❌
Remote Apply	✅	✅	✅	✅	✅

The synchronous_standby_names setting is a complementary setting to synchronous_commit. It lists the names of all replicas the primary database supports for synchronous replication, and configures how the primary database waits for them. The synchronous_standby_names setting supports these formats:

• FIRST num_sync (replica_name_1, replica_name_2): This waits for confirmation from the first num_sync replicas before returning success. The list of replica_names determines the relative priority of the replicas. Replica names are determined by the application_name setting on the replicas.
• ANY num_sync (replica_name_1, replica_name_2): This waits for confirmation from num_sync replicas in the provided list, regardless of their priority or position in the list. This is works as a quorum function.

Synchronous replication modes force the primary to wait until all required replicas have written the WAL, or applied the database transaction, depending on the synchronous_commit level. This could cause the primary to hang indefinitely if a required replica crashes. When the replica reconnects, it replays any of the WAL it needs to catch up. Only then is the primary able to resume writes. To mitigate this, provision more than the amount of nodes required under the synchronous_standby_names setting and list them in the FIRST or ANY clauses. This allows the primary to move forward as long as a quorum of replicas have written the most recent WAL transaction. Replicas that were out of service are able to reconnect and replay the missed WAL transactions asynchronously.

Replication diagnostics

The Postgres pg_stat_replication view provides information about each replica. This view is particularly useful for calculating replication lag, which measures how far behind the primary the current state of the replica is. The replay_lag field gives a measure of the seconds between the most recent WAL transaction on the primary, and the last reported database commit on the replica. Coupled with write_lag and flush_lag, this provides insight into how far behind the replica is. The *_lsn fields also provide helpful information. They allow you to compare WAL locations between the primary and the replicas. The state field is useful for determining exactly what each replica is currently doing; the available modes are startup, catchup, streaming, backup, and stopping.

To see the data, on the primary database, run this command:

SELECT * FROM pg_stat_replication;

The output looks like this:

-[ RECORD 1 ]----+------------------------------
pid              | 52343
usesysid         | 16384
usename          | repuser
application_name | r2
client_addr      | 10.0.13.6
client_hostname  |
client_port      | 59610
backend_start    | 2018-02-07 19:07:15.261213+00
backend_xmin     |
state            | streaming
sent_lsn         | 16B/43DB36A8
write_lsn        | 16B/43DB36A8
flush_lsn        | 16B/43DB36A8
replay_lsn       | 16B/43107C28
write_lag        | 00:00:00.009966
flush_lag        | 00:00:00.03208
replay_lag       | 00:00:00.43537
sync_priority    | 2
sync_state       | sync
-[ RECORD 2 ]----+------------------------------
pid              | 54498
usesysid         | 16384
usename          | repuser
application_name | r1
client_addr      | 10.0.13.5
client_hostname  |
client_port      | 43402
backend_start    | 2018-02-07 19:45:41.410929+00
backend_xmin     |
state            | streaming
sent_lsn         | 16B/43DB36A8
write_lsn        | 16B/43DB36A8
flush_lsn        | 16B/43DB36A8
replay_lsn       | 16B/42C3B9C8
write_lag        | 00:00:00.019736
flush_lag        | 00:00:00.044073
replay_lag       | 00:00:00.644004
sync_priority    | 1
sync_state       | sync

Failover

Postgres provides some failover functionality, where the replica is promoted to primary in the event of a failure. This is provided using the pg_ctl command or the trigger_file. However, Postgres does not provide support for automatic failover. For more information, see the Postgres failover documentation. If you require a configurable high availability solution with automatic failover functionality, check out Patroni.

===== PAGE: https://docs.tigerdata.com/self-hosted/replication-and-ha/about-ha/ =====

High availability

High availability (HA) is achieved by increasing redundancy and resilience. To increase redundancy, parts of the system are replicated, so that they are on standby in the event of a failure. To increase resilience, recovery processes switch between these standby resources as quickly as possible.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

Backups

For some systems, recovering from backup alone can be a suitable availability strategy.

For more information about backups in self-hosted TimescaleDB, see the backup and restore section in the TimescaleDB documentation.

Storage redundancy

Storage redundancy refers to having multiple copies of a database's data files. If the storage currently attached to a Postgres instance corrupts or otherwise becomes unavailable, the system can replace its current storage with one of the copies.

Instance redundancy

Instance redundancy refers to having replicas of your database running simultaneously. In the case of a database failure, a replica is an up-to-date, running database that can take over immediately.

Zonal redundancy

While the public cloud is highly reliable, entire portions of the cloud can be unavailable at times. TimescaleDB does not protect against Availability Zone failures unless the user is using HA replicas. We do not currently offer multi-cloud solutions or protection from an AWS Regional failure.

Replication

TimescaleDB supports replication using Postgres's built-in streaming replication. Using logical replication with TimescaleDB is not recommended, as it requires schema synchronization between the primary and replica nodes and replicating partition root tables, which are not currently supported.

Postgres achieves streaming replication by having replicas continuously stream the WAL from the primary database. See the official replication documentation for details. For more information about how Postgres implements Write-Ahead Logging, see their WAL Documentation.

Failover

Postgres offers failover functionality where a replica is promoted to primary in the event of a failure on the primary. This is done using pg_ctl or the trigger_file, but it does not provide out-of-the-box support for automatic failover. Read more in the Postgres failover documentation. Patroni offers a configurable high availability solution with automatic failover functionality.

===== PAGE: https://docs.tigerdata.com/self-hosted/distributed-hypertables/insert/ =====

Insert data

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

You can insert data into a distributed hypertable with an INSERT statement. The syntax looks the same as for a standard hypertable or Postgres table. For example:

INSERT INTO conditions(time, location, temperature, humidity)
  VALUES (NOW(), 'office', 70.0, 50.0);

Optimize data insertion

Distributed hypertables have higher network load than standard hypertables, because they must push inserts from the access node to the data nodes. You can optimize your insertion patterns to reduce load.

Insert data in batches

Reduce load by batching your INSERT statements over many rows of data, instead of performing each insertion as a separate transaction.

The access node first splits the batched data into smaller batches by determining which data node each row should belong to. It then writes each batch to the correct data node.

Optimize insert batch size

When inserting to a distributed hypertable, the access node tries to convert INSERT statements into more efficient COPY operations between the access and data nodes. But this doesn't work if:

• The INSERT statement has a RETURNING clause and
• The hypertable has triggers that could alter the returned data

In this case, the planner uses a multi-row prepared statement to insert into each data node. It splits the original insert statement across these sub-statements. You can view the plan by running an EXPLAIN on your INSERT statement.

In the prepared statement, the access node can buffer a number of rows before flushing them to the data node. By default, the number is 1000. You can optimize this by changing the timescaledb.max_insert_batch_size setting, for example to reduce the number of separate batches that must be sent.

The maximum batch size has a ceiling. This is equal to the maximum number of parameters allowed in a prepared statement, which is currently 32,767 parameters, divided by the number of columns in each row. For example, if you have a distributed hypertable with 10 columns, the highest you can set the batch size is 3276.

For more information on changing timescaledb.max_insert_batch_size, see the section on configuration.

Use a copy statement instead

COPY can perform better than INSERT on a distributed hypertable. But it doesn't support some features, such as conflict handling using the ON CONFLICT clause.

To copy from a file to your hypertable, run:

COPY <HYPERTABLE> FROM '<FILE_PATH>';

When doing a COPY, the access node switches each data node to copy mode. It then streams each row to the correct data node.

===== PAGE: https://docs.tigerdata.com/self-hosted/distributed-hypertables/alter-drop-distributed-hypertables/ =====

Alter and drop distributed hypertables

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

You can alter and drop distributed hypertables in the same way as standard hypertables. To learn more, see:

• Altering hypertables
• Dropping hypertables

When you alter a distributed hypertable, or set privileges on it, the commands are automatically applied across all data nodes. For more information, see the section on multi-node administration.

===== PAGE: https://docs.tigerdata.com/self-hosted/distributed-hypertables/create-distributed-hypertables/ =====

Create distributed hypertables

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

If you have a multi-node environment, you can create a distributed hypertable across your data nodes. First create a standard Postgres table, and then convert it into a distributed hypertable.

You need to set up your multi-node cluster before creating a distributed hypertable. To set up multi-node, see the multi-node section.

Creating a distributed hypertable

1. On the access node of your multi-node cluster, create a standard Postgres table:

   CREATE TABLE conditions (
     time        TIMESTAMPTZ       NOT NULL,
     location    TEXT              NOT NULL,
     temperature DOUBLE PRECISION  NULL,
     humidity    DOUBLE PRECISION  NULL
   );
   

2. Convert the table to a distributed hypertable. Specify the name of the table you want to convert, the column that holds its time values, and a space-partitioning parameter.

   SELECT create_distributed_hypertable('conditions', 'time', 'location');
   

===== PAGE: https://docs.tigerdata.com/self-hosted/distributed-hypertables/foreign-keys/ =====

Create foreign keys in a distributed hypertable

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Tables and values referenced by a distributed hypertable must be present on the access node and all data nodes. To create a foreign key from a distributed hypertable, use distributed_exec to first create the referenced table on all nodes.

Creating foreign keys in a distributed hypertable

1. Create the referenced table on the access node.
2. Use distributed_exec to create the same table on all data nodes and update it with the correct data.
3. Create a foreign key from your distributed hypertable to your referenced table.

===== PAGE: https://docs.tigerdata.com/self-hosted/distributed-hypertables/triggers/ =====

Use triggers on distributed hypertables

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Triggers on distributed hypertables work in much the same way as triggers on standard hypertables, and have the same limitations. But there are some differences due to the data being distributed across multiple nodes:

• Row-level triggers fire on the data node where the row is inserted. The triggers must fire where the data is stored, because BEFORE and AFTER row triggers need access to the stored data. The chunks on the access node do not contain any data, so they have no triggers.
• Statement-level triggers fire once on each affected node, including the access node. For example, if a distributed hypertable includes 3 data nodes, inserting 2 rows of data executes a statement-level trigger on the access node and either 1 or 2 data nodes, depending on whether the rows go to the same or different nodes.
• A replication factor greater than 1 further causes the trigger to fire on multiple nodes. Each replica node fires the trigger.

Create a trigger on a distributed hypertable

Create a trigger on a distributed hypertable by using CREATE TRIGGER as usual. The trigger, and the function it executes, is automatically created on each data node. If the trigger function references any other functions or objects, they need to be present on all nodes before you create the trigger.

Creating a trigger on a distributed hypertable

1. If your trigger needs to reference another function or object, use distributed_exec to create the function or object on all nodes.

2. Create the trigger function on the access node. This example creates a dummy trigger that raises the notice 'trigger fired':

   CREATE OR REPLACE FUNCTION my_trigger_func()
   RETURNS TRIGGER LANGUAGE PLPGSQL AS
   body$
   BEGIN
   RAISE NOTICE 'trigger fired';
   RETURN NEW;
   END
   body$;
   

3. Create the trigger itself on the access node. This example causes the trigger to fire whenever a row is inserted into the hypertable hyper. Note that you don't need to manually create the trigger on the data nodes. This is done automatically for you.

   CREATE TRIGGER my_trigger
   AFTER INSERT ON hyper
   FOR EACH ROW
   EXECUTE FUNCTION my_trigger_func();
   

Avoid processing a trigger multiple times

If you have a statement-level trigger, or a replication factor greater than 1, the trigger fires multiple times. To avoid repetitive firing, you can set the trigger function to check which data node it is executing on.

For example, write a trigger function that raises a different notice on the access node compared to a data node:

CREATE OR REPLACE FUNCTION my_trigger_func()
    RETURNS TRIGGER LANGUAGE PLPGSQL AS
body$
DECLARE
    is_access_node boolean;
BEGIN
    SELECT is_distributed INTO is_access_node
    FROM timescaledb_information.hypertables
    WHERE hypertable_name =
    AND hypertable_schema = ;

    IF is_access_node THEN
       RAISE NOTICE 'trigger fired on the access node';
    ELSE
       RAISE NOTICE 'trigger fired on a data node';
    END IF;

    RETURN NEW;
END
body$;

===== PAGE: https://docs.tigerdata.com/self-hosted/distributed-hypertables/query/ =====

Query data in distributed hypertables

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

You can query a distributed hypertable just as you would query a standard hypertable or Postgres table. For more information, see the section on writing data.

Queries perform best when the access node can push transactions down to the data nodes. To ensure that the access node can push down transactions, check that the enable_partitionwise_aggregate setting is set to on for the access node. By default, it is off.

If you want to use continuous aggregates on your distributed hypertable, see the continuous aggregates section for more information.

===== PAGE: https://docs.tigerdata.com/self-hosted/distributed-hypertables/about-distributed-hypertables/ =====

About distributed hypertables

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Distributed hypertables are hypertables that span multiple nodes. With distributed hypertables, you can scale your data storage across multiple machines. The database can also parallelize some inserts and queries.

A distributed hypertable still acts as if it were a single table. You can work with one in the same way as working with a standard hypertable. To learn more about hypertables, see the hypertables section.

Certain nuances can affect distributed hypertable performance. This section explains how distributed hypertables work, and what you need to consider before adopting one.

Architecture of a distributed hypertable

Distributed hypertables are used with multi-node clusters. Each cluster has an access node and multiple data nodes. You connect to your database using the access node, and the data is stored on the data nodes. For more information about multi-node, see the multi-node section.

You create a distributed hypertable on your access node. Its chunks are stored on the data nodes. When you insert data or run a query, the access node communicates with the relevant data nodes and pushes down any processing if it can.

Space partitioning

Distributed hypertables are always partitioned by time, just like standard hypertables. But unlike standard hypertables, distributed hypertables should also be partitioned by space. This allows you to balance inserts and queries between data nodes, similar to traditional sharding. Without space partitioning, all data in the same time range would write to the same chunk on a single node.

By default, TimescaleDB creates as many space partitions as there are data nodes. You can change this number, but having too many space partitions degrades performance. It increases planning time for some queries, and leads to poorer balancing when mapping items to partitions.

Data is assigned to space partitions by hashing. Each hash bucket in the space dimension corresponds to a data node. One data node may hold many buckets, but each bucket may belong to only one node for each time interval.

When space partitioning is on, 2 dimensions are used to divide data into chunks: the time dimension and the space dimension. You can specify the number of partitions along the space dimension. Data is assigned to a partition by hashing its value on that dimension.

For example, say you use device_id as a space partitioning column. For each row, the value of the device_id column is hashed. Then the row is inserted into the correct partition for that hash value.

Closed and open dimensions for space partitioning

Space partitioning dimensions can be open or closed. A closed dimension has a fixed number of partitions, and usually uses some hashing to match values to partitions. An open dimension does not have a fixed number of partitions, and usually has each chunk cover a certain range. In most cases the time dimension is open and the space dimension is closed.

If you use the create_hypertable command to create your hypertable, then the space dimension is open, and there is no way to adjust this. To create a hypertable with a closed space dimension, create the hypertable with only the time dimension first. Then use the add_dimension command to explicitly add an open device. If you set the range to 1, each device has its own chunks. This can help you work around some limitations of regular space dimensions, and is especially useful if you want to make some chunks readily available for exclusion.

Repartitioning distributed hypertables

You can expand distributed hypertables by adding additional data nodes. If you now have fewer space partitions than data nodes, you need to increase the number of space partitions to make use of your new nodes. The new partitioning configuration only affects new chunks. In this diagram, an extra data node was added during the third time interval. The fourth time interval now includes four chunks, while the previous time intervals still include three:

This can affect queries that span the two different partitioning configurations. For more information, see the section on limitations of query push down.

Replicating distributed hypertables

To replicate distributed hypertables at the chunk level, configure the hypertables to write each chunk to multiple data nodes. This native replication ensures that a distributed hypertable is protected against data node failures and provides an alternative to fully replicating each data node using streaming replication to provide high availability. Only the data nodes are replicated using this method. The access node is not replicated.

For more information about replication and high availability, see the multi-node HA section.

Performance of distributed hypertables

A distributed hypertable horizontally scales your data storage, so you're not limited by the storage of any single machine. It also increases performance for some queries.

Whether, and by how much, your performance increases depends on your query patterns and data partitioning. Performance increases when the access node can push down query processing to data nodes. For example, if you query with a GROUP BY clause, and the data is partitioned by the GROUP BY column, the data nodes can perform the processing and send only the final results to the access node.

If processing can't be done on the data nodes, the access node needs to pull in raw or partially processed data and do the processing locally. For more information, see the limitations of pushing down queries.

Query push down

The access node can use a full or a partial method to push down queries. Computations that can be pushed down include sorts and groupings. Joins on data nodes aren't currently supported.

To see how a query is pushed down to a data node, use EXPLAIN VERBOSE to inspect the query plan and the remote SQL statement sent to each data node.

Full push down

In the full push-down method, the access node offloads all computation to the data nodes. It receives final results from the data nodes and appends them. To fully push down an aggregate query, the GROUP BY clause must include either:

• All the partitioning columns or
• Only the first space-partitioning column

For example, say that you want to calculate the max temperature for each location:

SELECT location, max(temperature)
  FROM conditions
  GROUP BY location;

If location is your only space partition, each data node can compute the maximum on its own subset of the data.

Partial push down

In the partial push-down method, the access node offloads most of the computation to the data nodes. It receives partial results from the data nodes and calculates a final aggregate by combining the partials.

For example, say that you want to calculate the max temperature across all locations. Each data node computes a local maximum, and the access node computes the final result by computing the maximum of all the local maximums:

SELECT max(temperature) FROM conditions;

Limitations of query push down

Distributed hypertables get improved performance when they can push down queries to the data nodes. But the query planner might not be able to push down every query. Or it might only be able to partially push down a query. This can occur for several reasons:

• You changed the partitioning configuration. For example, you added new data nodes and increased the number of space partitions to match. This can cause chunks for the same space value to be stored on different nodes. For instance, say you partition by device_id. You start with 3 partitions, and data for device_B is stored on node 3. You later increase to 4 partitions. New chunks for device_B are now stored on node 4. If you query across the repartitioning boundary, a final aggregate for device_B cannot be calculated on node 3 or node 4 alone. Partially processed data must be sent to the access node for final aggregation. The TimescaleDB query planner dynamically detects such overlapping chunks and reverts to the appropriate partial aggregation plan. This means that you can add data nodes and repartition your data to achieve elasticity without worrying about query results. In some cases, your query could be slightly less performant, but this is rare and the affected chunks usually move quickly out of your retention window.
• The query includes non-immutable functions and expressions. The function cannot be pushed down to the data node, because by definition, it isn't guaranteed to have a consistent result across each node. An example non-immutable function is random(), which depends on the current seed.
• The query includes a job function. The access node assumes the function doesn't exist on the data nodes, and doesn't push it down.

TimescaleDB uses several optimizations to avoid these limitations, and push down as many queries as possible. For example, now() is a non-immutable function. The database converts it to a constant on the access node and pushes down the constant timestamp to the data nodes.

Combine distributed hypertables and standard hypertables

You can use distributed hypertables in the same database as standard hypertables and standard Postgres tables. This mostly works the same way as having multiple standard tables, with a few differences. For example, if you JOIN a standard table and a distributed hypertable, the access node needs to fetch the raw data from the data nodes and perform the JOIN locally.

Limitations

All the limitations of regular hypertables also apply to distributed hypertables. In addition, the following limitations apply specifically to distributed hypertables:

• Distributed scheduling of background jobs is not supported. Background jobs created on an access node are scheduled and executed on this access node without distributing the jobs to data nodes.
• Continuous aggregates can aggregate data distributed across data nodes, but the continuous aggregate itself must live on the access node. This could create a limitation on how far you can scale your installation, but because continuous aggregates are downsamples of the data, this does not usually create a problem.
• Reordering chunks is not supported.
• Tablespaces cannot be attached to a distributed hypertable on the access node. It is still possible to attach tablespaces on data nodes.
• Roles and permissions are assumed to be consistent across the nodes of a distributed database, but consistency is not enforced.
• Joins on data nodes are not supported. Joining a distributed hypertable with another table requires the other table to reside on the access node. This also limits the performance of joins on distributed hypertables.
• Tables referenced by foreign key constraints in a distributed hypertable must be present on the access node and all data nodes. This applies also to referenced values.
• Parallel-aware scans and appends are not supported.
• Distributed hypertables do not natively provide a consistent restore point for backup and restore across nodes. Use the create_distributed_restore_point command, and make sure you take care when you restore individual backups to access and data nodes.
• For native replication limitations, see the native replication section.
• User defined functions have to be manually installed on the data nodes so that the function definition is available on both access and data nodes. This is particularly relevant for functions that are registered with set_integer_now_func.

Note that these limitations concern usage from the access node. Some currently unsupported features might still work on individual data nodes, but such usage is neither tested nor officially supported. Future versions of TimescaleDB might remove some of these limitations.

===== PAGE: https://docs.tigerdata.com/self-hosted/backup-and-restore/logical-backup/ =====

Logical backup with pg_dump and pg_restore

You back up and restore each self-hosted Postgres database with TimescaleDB enabled using the native Postgres pg_dump and pg_restore commands. This also works for compressed hypertables, you don't have to decompress the chunks before you begin.

If you are using pg_dump to backup regularly, make sure you keep track of the versions of Postgres and TimescaleDB you are running. For more information, see Versions are mismatched when dumping and restoring a database.

This page shows you how to:

• Back up and restore an entire database
• Back up and restore individual hypertables

You can also upgrade between different versions of TimescaleDB.

Prerequisites

• A source database to backup from, and a target database to restore to.
• Install the psql and pg_dump Postgres client tools on your migration machine.

Back up and restore an entire database

You backup and restore an entire database using pg_dump and psql.

In terminal:

1. Set your connection strings

   These variables hold the connection information for the source database to backup from and the target database to restore to:

   export SOURCE=postgres://<user>:<password>@<source host>:<source port>/<db_name>
   export TARGET=postgres://<user>:<password>@<source host>:<source port>
   

2. Backup your database

   pg_dump -d "source" \
     -Fc -f <db_name>.bak
   

   You may see some errors while pg_dump is running. See Troubleshooting self-hosted TimescaleDB to check if they can be safely ignored.

3. Restore your database from the backup

   1. Connect to your target database:

      psql -d "target"
      

   2. Create a new database and enable TimescaleDB:

      CREATE DATABASE <restoration database>;
      \c <restoration database>
      CREATE EXTENSION IF NOT EXISTS timescaledb;
      

   3. Put your database in the right state for restoring:

      SELECT timescaledb_pre_restore();
      

   4. Restore the database:

       pg_restore -Fc -d <restoration database> <db_name>.bak
      

   5. Return your database to normal operations:

      SELECT timescaledb_post_restore();
      

      Do not use pg_restore with the -j option. This option does not correctly restore the TimescaleDB catalogs.

Back up and restore individual hypertables

pg_dump provides flags that allow you to specify tables or schemas to back up. However, using these flags means that the dump lacks necessary information that TimescaleDB requires to understand the relationship between them. Even if you explicitly specify both the hypertable and all of its constituent chunks, the dump would still not contain all the information it needs to recreate the hypertable on restore.

To backup individual hypertables, backup the database schema, then backup only the tables you need. You also use this method to backup individual plain tables.

In Terminal:

1. Set your connection strings

   These variables hold the connection information for the source database to backup from and the target database to restore to:

   export SOURCE=postgres://<user>:<password>@<source host>:<source port>/<db_name>
   export TARGET=postgres://<user>:<password>@<source host>:<source port>/<db_name>
   

2. Backup the database schema and individual tables

   1. Back up the hypertable schema:

      pg_dump -s -d source --table   > schema.sql
      

   2. Backup hypertable data to a CSV file:

   For each hypertable to backup:

   psql -d source \
   -c "\COPY (SELECT * FROM ) TO .csv DELIMITER ',' CSV"
   

3. Restore the schema to the target database

   psql -d target < schema.sql
   

4. Restore hypertables from the backup

   For each hypertable to backup:

   1. Recreate the hypertable:

      psql -d target -c "SELECT create_hypertable(, <partition>)"
      

      When you create the new hypertable, you do not need to use the same parameters as existed in the source database. This can provide a good opportunity for you to re-organize your hypertables if you need to. For example, you can change the partitioning key, the number of partitions, or the chunk interval sizes.

   2. Restore the data:

      psql -d target -c "\COPY  FROM .csv CSV"
      

      The standard COPY command in Postgres is single threaded. If you have a lot of data, you can speed up the copy using the timescaledb-parallel-copy.

Best practice is to backup and restore a database at a time. However, if you have superuser access to Postgres instance with TimescaleDB installed, you can use pg_dumpall to back up all Postgres databases in a cluster, including global objects that are common to all databases, namely database roles, tablespaces, and privilege grants. You restore the Postgres instance using psql. For more information, see the Postgres documentation.

===== PAGE: https://docs.tigerdata.com/self-hosted/backup-and-restore/physical/ =====

Physical backups

For full instance physical backups (which are especially useful for starting up new replicas), pg_basebackup works with all TimescaleDB installation types. You can also use any of several external backup and restore managers such as pg_backrest, or barman. For ongoing physical backups, you can use wal-e, although this method is now deprecated. These tools all allow you to take online, physical backups of your entire instance, and many offer incremental backups and other automation options.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

===== PAGE: https://docs.tigerdata.com/self-hosted/backup-and-restore/docker-and-wale/ =====

Ongoing physical backups with Docker & WAL-E

When you run TimescaleDB in a containerized environment, you can use continuous archiving with a WAL-E container. These containers are sometimes referred to as sidecars, because they run alongside the main container. A WAL-E sidecar image works with TimescaleDB as well as regular Postgres. In this section, you can set up archiving to your local filesystem with a main TimescaleDB container called timescaledb, and a WAL-E sidecar called wale. When you are ready to implement this in your production deployment, you can adapt the instructions here to do archiving against cloud providers such as AWS S3, and run it in an orchestration framework such as Kubernetes.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

Run the TimescaleDB container in Docker

To make TimescaleDB use the WAL-E sidecar for archiving, the two containers need to share a network. To do this, you need to create a Docker network and then launch TimescaleDB with archiving turned on, using the newly created network. When you launch TimescaleDB, you need to explicitly set the location of the write-ahead log (POSTGRES_INITDB_WALDIR) and data directory (PGDATA) so that you can share them with the WAL-E sidecar. Both must reside in a Docker volume, by default a volume is created for /var/lib/postgresql/data. When you have started TimescaleDB, you can log in and create tables and data.

This section describes a feature that is deprecated. We strongly recommend that you do not use this feature in a production environment. If you need more information, contact us.

Running the TimescaleDB container in Docker

1. Create the docker container:

   docker network create timescaledb-net
   

2. Launch TimescaleDB, with archiving turned on:

   docker run \
     --name timescaledb \
     --network timescaledb-net \
     -e POSTGRES_PASSWORD=insecure \
     -e POSTGRES_INITDB_WALDIR=/var/lib/postgresql/data/pg_wal \
     -e PGDATA=/var/lib/postgresql/data/pg_data \
     timescale/timescaledb:latest-pg10 postgres \
     -cwal_level=archive \
     -carchive_mode=on \
     -carchive_command="/usr/bin/wget wale/wal-push/%f -O -" \
     -carchive_timeout=600 \
     -ccheckpoint_timeout=700 \
     -cmax_wal_senders=1
   

3. Run TimescaleDB within Docker:

   docker exec -it timescaledb psql -U postgres
   

Perform the backup using the WAL-E sidecar

The WAL-E Docker image runs a web endpoint that accepts WAL-E commands across an HTTP API. This allows Postgres to communicate with the WAL-E sidecar over the internal network to trigger archiving. You can also use the container to invoke WAL-E directly. The Docker image accepts standard WAL-E environment variables to configure the archiving backend, so you can issue commands from services such as AWS S3. For information about configuring, see the official WAL-E documentation.

To enable the WAL-E docker image to perform archiving, it needs to use the same network and data volumes as the TimescaleDB container. It also needs to know the location of the write-ahead log and data directories. You can pass all this information to WAL-E when you start it. In this example, the WAL-E image listens for commands on the timescaledb-net internal network at port 80, and writes backups to ~/backups on the Docker host.

Performing the backup using the WAL-E sidecar

1. Start the WAL-E container with the required information about the container. In this example, the container is called timescaledb-wale:

   docker run \
     --name wale \
     --network timescaledb-net \
     --volumes-from timescaledb \
     -v ~/backups:/backups \
     -e WALE_LOG_DESTINATION=stderr \
     -e PGWAL=/var/lib/postgresql/data/pg_wal \
     -e PGDATA=/var/lib/postgresql/data/pg_data \
     -e PGHOST=timescaledb \
     -e PGPASSWORD=insecure \
     -e PGUSER=postgres \
     -e WALE_FILE_PREFIX=file://localhost/backups \
     timescale/timescaledb-wale:latest
   

2. Start the backup:

   docker exec wale wal-e backup-push /var/lib/postgresql/data/pg_data
   

   Alternatively, you can start the backup using the sidecar's HTTP endpoint. This requires exposing the sidecar's port 80 on the Docker host by mapping it to an open port. In this example, it is mapped to port 8080:

   curl http://localhost:8080/backup-push
   

You should do base backups at regular intervals daily, to minimize the amount of WAL-E replay, and to make recoveries faster. To make new base backups, re-trigger a base backup as shown here, either manually or on a schedule. If you run TimescaleDB on Kubernetes, there is built-in support for scheduling cron jobs that can invoke base backups using the WAL-E container's HTTP API.

Recovery

To recover the database instance from the backup archive, create a new TimescaleDB container, and restore the database and configuration files from the base backup. Then you can relaunch the sidecar and the database.

Restoring database files from backup

1. Create the docker container:

   docker create \
     --name timescaledb-recovered \
     --network timescaledb-net \
     -e POSTGRES_PASSWORD=insecure \
     -e POSTGRES_INITDB_WALDIR=/var/lib/postgresql/data/pg_wal \
     -e PGDATA=/var/lib/postgresql/data/pg_data \
     timescale/timescaledb:latest-pg10 postgres
   

2. Restore the database files from the base backup:

   docker run -it --rm \
     -v ~/backups:/backups \
     --volumes-from timescaledb-recovered \
     -e WALE_LOG_DESTINATION=stderr \
     -e WALE_FILE_PREFIX=file://localhost/backups \
     timescale/timescaledb-wale:latest \wal-e \
     backup-fetch /var/lib/postgresql/data/pg_data LATEST
   

3. Recreate the configuration files. These are backed up from the original database instance:

   docker run -it --rm  \
     --volumes-from timescaledb-recovered \
     timescale/timescaledb:latest-pg10 \
     cp /usr/local/share/postgresql/pg_ident.conf.sample /var/lib/postgresql/data/pg_data/pg_ident.conf
   
   docker run -it --rm  \
     --volumes-from timescaledb-recovered \
     timescale/timescaledb:latest-pg10 \
   
   cp /usr/local/share/postgresql/postgresql.conf.sample /var/lib/postgresql/data/pg_data/postgresql.conf
   
   docker run -it --rm  \
     --volumes-from timescaledb-recovered \
     timescale/timescaledb:latest-pg10 \
   
   sh -c 'echo "local all postgres trust" > /var/lib/postgresql/data/pg_data/pg_hba.conf'
   

4. Create a recovery.conf file that tells Postgres how to recover:

   docker run -it --rm  \
     --volumes-from timescaledb-recovered \
     timescale/timescaledb:latest-pg10 \
   
   sh -c 'echo "restore_command='\''/usr/bin/wget wale/wal-fetch/%f -O -'\''" > /var/lib/postgresql/data/pg_data/recovery.conf'
   

When you have recovered the data and the configuration files, and have created a recovery configuration file, you can relaunch the sidecar. You might need to remove the old one first. When you relaunch the sidecar, it replays the last WAL segments that might be missing from the base backup. The you can relaunch the database, and check that recovery was successful.

Relaunch the recovered database

1. Relaunch the WAL-E sidecar:

   docker run \
     --name wale \
     --network timescaledb-net \
     -v ~/backups:/backups \
     --volumes-from timescaledb-recovered \
     -e WALE_LOG_DESTINATION=stderr \
     -e PGWAL=/var/lib/postgresql/data/pg_wal \
     -e PGDATA=/var/lib/postgresql/data/pg_data \
     -e PGHOST=timescaledb \
     -e PGPASSWORD=insecure \
     -e PGUSER=postgres \
     -e WALE_FILE_PREFIX=file://localhost/backups \
     timescale/timescaledb-wale:latest
   

2. Relaunch the TimescaleDB docker container:

   docker start timescaledb-recovered
   

3. Verify that the database started up and recovered successfully:

   docker logs timescaledb-recovered
   

   Don't worry if you see some archive recovery errors in the log at this stage. This happens because the recovery is not completely finalized until no more files can be found in the archive. See the Postgres documentation on continuous archiving for more information.

===== PAGE: https://docs.tigerdata.com/self-hosted/uninstall/uninstall-timescaledb/ =====

Uninstall TimescaleDB

Postgres is designed to be easily extensible. The extensions loaded into the database can function just like features that are built in. TimescaleDB extends Postgres for time-series data, giving Postgres the high-performance, scalability, and analytical capabilities required by modern data-intensive applications. If you installed TimescaleDB with Homebrew or MacPorts, you can uninstall it without having to uninstall Postgres.

Uninstalling TimescaleDB using Homebrew

1. At the psql prompt, remove the TimescaleDB extension:

   DROP EXTENSION timescaledb;
   

2. At the command prompt, remove timescaledb from shared_preload_libraries in the postgresql.conf configuration file:

   nano /opt/homebrew/var/postgresql@14/postgresql.conf
   shared_preload_libraries = ''
   

3. Save the changes to the postgresql.conf file.

4. Restart Postgres:

   brew services restart postgresql
   

5. Check that the TimescaleDB extension is uninstalled by using the \dx command at the psql prompt. Output is similar to:

   tsdb-# \dx
                                         List of installed extensions
       Name     | Version |   Schema   |                            Description
   -------------+---------+------------+-------------------------------------------------------------------
    plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
   (1 row)
   

6. Uninstall TimescaleDB:

   brew uninstall timescaledb
   

7. Remove all the dependencies and related files:

   brew remove timescaledb
   

Uninstalling TimescaleDB using MacPorts

1. At the psql prompt, remove the TimescaleDB extension:

   DROP EXTENSION timescaledb;
   

2. At the command prompt, remove timescaledb from shared_preload_libraries in the postgresql.conf configuration file:

   nano /opt/homebrew/var/postgresql@14/postgresql.conf
   shared_preload_libraries = ''
   

3. Save the changes to the postgresql.conf file.

4. Restart Postgres:

   port reload postgresql
   

5. Check that the TimescaleDB extension is uninstalled by using the \dx command at the psql prompt. Output is similar to:

   tsdb-# \dx
                                         List of installed extensions
       Name     | Version |   Schema   |                            Description
   -------------+---------+------------+-------------------------------------------------------------------
    plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
   (1 row)
   

6. Uninstall TimescaleDB and the related dependencies:

   port uninstall timescaledb --follow-dependencies
   

===== PAGE: https://docs.tigerdata.com/self-hosted/upgrades/about-upgrades/ =====

About upgrades

A major upgrade is when you upgrade from one major version of TimescaleDB, to the next major version. For example, when you upgrade from TimescaleDB 1 to TimescaleDB 2.

A minor upgrade is when you upgrade within your current major version of TimescaleDB. For example, when you upgrade from TimescaleDB 2.5 to TimescaleDB 2.6.

If you originally installed TimescaleDB using Docker, you can upgrade from within the Docker container. For more information, and instructions, see the Upgrading with Docker section.

When you upgrade the timescaledb extension, the experimental schema is removed by default. To use experimental features after an upgrade, you need to add the experimental schema again.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

Plan your upgrade

• Install the Postgres client tools on your migration machine. This includes psql, and pg_dump.
• Read the release notes for the version of TimescaleDB that you are upgrading to.
• Perform a backup of your database. While TimescaleDB upgrades are performed in-place, upgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster.

If you use the TimescaleDB Toolkit, ensure the timescaledb_toolkit extension is on version 1.6.0, then upgrade the timescaledb extension. If required, you can then later upgrade the timescaledb_toolkit extension to the most recent version.

Check your version

You can check which version of TimescaleDB you are running, at the psql command prompt. Use this to check which version you are running before you begin your upgrade, and again after your upgrade is complete:

\dx timescaledb

    Name     | Version |   Schema   |                             Description
-------------+---------+------------+---------------------------------------------------------------------
 timescaledb | x.y.z   | public     | Enables scalable inserts and complex queries for time-series data
(1 row)

===== PAGE: https://docs.tigerdata.com/self-hosted/upgrades/upgrade-pg/ =====

Upgrade Postgres

TimescaleDB is a Postgres extension. Ensure that you upgrade to compatible versions of TimescaleDB and Postgres.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

Prerequisites

• Install the Postgres client tools on your migration machine. This includes psql, and pg_dump.
• Read the release notes for the version of TimescaleDB that you are upgrading to.
• Perform a backup of your database. While TimescaleDB upgrades are performed in-place, upgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster.

Plan your upgrade path

Best practice is to always use the latest version of TimescaleDB. Subscribe to our releases on GitHub or use Tiger Cloud and always run the latest update without any hassle.

Check the following support matrix against the versions of TimescaleDB and Postgres that you are running currently and the versions you want to update to, then choose your upgrade path.

For example, to upgrade from TimescaleDB 2.13 on Postgres 13 to TimescaleDB 2.18.2 you need to:

1. Upgrade TimescaleDB to 2.15
2. Upgrade Postgres to 14, 15 or 16.
3. Upgrade TimescaleDB to 2.18.2.

You may need to upgrade to the latest Postgres version before you upgrade TimescaleDB. Also, if you use TimescaleDB Toolkit, ensure the timescaledb_toolkit extension is >= v1.6.0 before you upgrade TimescaleDB extension.

TimescaleDB version	Postgres 17	Postgres 16	Postgres 15	Postgres 14	Postgres 13	Postgres 12	Postgres 11	Postgres 10
2.22.x	✅	✅	✅	❌	❌	❌	❌	❌
2.21.x	✅	✅	✅	❌	❌	❌	❌	❌
2.20.x	✅	✅	✅	❌	❌	❌	❌	❌
2.17 - 2.19	✅	✅	✅	✅	❌	❌	❌	❌
2.16.x	❌	✅	✅	✅	❌	❌	❌	❌
2.13 - 2.15	❌	✅	✅	✅	✅	❌	❌	❌
2.12.x	❌	❌	✅	✅	✅	❌	❌	❌
2.10.x	❌	❌	✅	✅	✅	✅	❌	❌
2.5 - 2.9	❌	❌	❌	✅	✅	✅	❌	❌
2.4	❌	❌	❌	❌	✅	✅	❌	❌
2.1 - 2.3	❌	❌	❌	❌	✅	✅	✅	❌
2.0	❌	❌	❌	❌	❌	✅	✅	❌
1.7	❌	❌	❌	❌	❌	✅	✅	✅

We recommend not using TimescaleDB with Postgres 17.1, 16.5, 15.9, 14.14, 13.17, 12.21. These minor versions introduced a breaking binary interface change that, once identified, was reverted in subsequent minor Postgres versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with Postgres 17.2, 16.6, etc and higher. Users of Tiger Cloud and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected.

Upgrade your Postgres instance

You use pg_upgrade to upgrade Postgres in-place. pg_upgrade allows you to retain the data files of your current Postgres installation while binding the new Postgres binary runtime to them.

1. Find the location of the Postgres binary

   Set the OLD_BIN_DIR environment variable to the folder holding the postgres binary. For example, which postgres returns something like /usr/lib/postgresql/16/bin/postgres.

   export OLD_BIN_DIR=/usr/lib/postgresql/16/bin
   

2. Set your connection string

   This variable holds the connection information for the database to upgrade:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   

3. Retrieve the location of the Postgres data folder

   Set the OLD_DATA_DIR environment variable to the value returned by the following:

   psql -d "source" -c "SHOW data_directory ;"
   

   Postgres returns something like:

   ----------------------------
   /home/postgres/pgdata/data
   (1 row)
   

4. Choose the new locations for the Postgres binary and data folders

   For example:

   export NEW_BIN_DIR=/usr/lib/postgresql/17/bin
   export NEW_DATA_DIR=/home/postgres/pgdata/data-17
   

5. Using psql, perform the upgrade:

   pg_upgrade -b $OLD_BIN_DIR -B $NEW_BIN_DIR -d $OLD_DATA_DIR -D $NEW_DATA_DIR
   

If you are moving data to a new physical instance of Postgres, you can use pg_dump and pg_restore to dump your data from the old database, and then restore it into the new, upgraded, database. For more information, see the backup and restore section.

===== PAGE: https://docs.tigerdata.com/self-hosted/upgrades/downgrade/ =====

Downgrade to a previous version of TimescaleDB

If you upgrade to a new TimescaleDB version and encounter problems, you can roll back to a previously installed version. This works in the same way as a minor upgrade.

Downgrading is not supported for all versions. Generally, downgrades between patch versions and between consecutive minor versions are supported. For example, you can downgrade from TimescaleDB 2.5.2 to 2.5.1, or from 2.5.0 to 2.4.2. To check whether you can downgrade from a specific version, see the release notes.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

Plan your downgrade

You can downgrade your on-premise TimescaleDB installation in-place. This means that you do not need to dump and restore your data. However, it is still important that you plan for your downgrade ahead of time.

Before you downgrade:

• Read the release notes for the TimescaleDB version you are downgrading to.
• Check which Postgres version you are currently running. You might need to upgrade to the latest Postgres version before you begin your TimescaleDB downgrade.
• Perform a backup of your database. While TimescaleDB downgrades are performed in-place, downgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster.

Downgrade TimescaleDB to a previous minor version

This downgrade uses the Postgres ALTER EXTENSION function to downgrade to a previous version of the TimescaleDB extension. TimescaleDB supports having different extension versions on different databases within the same Postgres instance. This allows you to upgrade and downgrade extensions independently on different databases. Run the ALTER EXTENSION function on each database to downgrade them individually.

The downgrade script is tested and supported for single-step downgrades. That is, downgrading from the current version, to the previous minor version. Downgrading might not work if you have made changes to your database between upgrading and downgrading.

1. Set your connection string

   This variable holds the connection information for the database to upgrade:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   

2. Connect to your database instance

   psql -X -d source
   

   The -X flag prevents any .psqlrc commands from accidentally triggering the load of a previous TimescaleDB version on session startup.

3. Downgrade the TimescaleDB extension This must be the first command you execute in the current session:

   ALTER EXTENSION timescaledb UPDATE TO '<PREVIOUS_VERSION>';
   

   For example:

   ALTER EXTENSION timescaledb UPDATE TO '2.17.0';
   

4. Check that you have downgraded to the correct version of TimescaleDB

   \dx timescaledb;
   

   Postgres returns something like:

   Name     | Version | Schema |                                      Description
   -------------+---------+--------+---------------------------------------------------------------------------------------
   timescaledb | 2.17.0  | public | Enables scalable inserts and complex queries for time-series data (Community Edition)
   

===== PAGE: https://docs.tigerdata.com/self-hosted/upgrades/minor-upgrade/ =====

Minor TimescaleDB upgrades

A minor upgrade is when you update from TimescaleDB <major version>.x to TimescaleDB <major version>.y. A major upgrade is when you update from TimescaleDB X.<minor version> to Y.<minor version>. You can run different versions of TimescaleDB on different databases within the same Postgres instance. This process uses the Postgres ALTER EXTENSION function to upgrade TimescaleDB independently on different databases.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

This page shows you how to perform a minor upgrade, for major upgrades, see Upgrade TimescaleDB to a major version.

Prerequisites

• Install the Postgres client tools on your migration machine. This includes psql, and pg_dump.
• Read the release notes for the version of TimescaleDB that you are upgrading to.
• Perform a backup of your database. While TimescaleDB upgrades are performed in-place, upgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster.

Check the TimescaleDB and Postgres versions

To see the versions of Postgres and TimescaleDB running in a self-hosted database instance:

1. Set your connection string

   This variable holds the connection information for the database to upgrade:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   

2. Retrieve the version of Postgres that you are running

   psql -X -d source -c "SELECT version();"
   

   Postgres returns something like:

   -----------------------------------------------------------------------------------------------------------------------------------------
   PostgreSQL 17.2 (Ubuntu 17.2-1.pgdg22.04+1) on aarch64-unknown-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
   (1 row)
   

3. Retrieve the version of TimescaleDB that you are running

   psql -X -d source -c "\dx timescaledb;"
   

   Postgres returns something like:

       Name     | Version |   Schema   |                             Description
   -------------+---------+------------+---------------------------------------------------------------------
   timescaledb | 2.17.2   | public     | Enables scalable inserts and complex queries for time-series data
   (1 row)
   

Plan your upgrade path

Best practice is to always use the latest version of TimescaleDB. Subscribe to our releases on GitHub or use Tiger Cloud and always run the latest update without any hassle.

Check the following support matrix against the versions of TimescaleDB and Postgres that you are running currently and the versions you want to update to, then choose your upgrade path.

For example, to upgrade from TimescaleDB 2.13 on Postgres 13 to TimescaleDB 2.18.2 you need to:

1. Upgrade TimescaleDB to 2.15
2. Upgrade Postgres to 14, 15 or 16.
3. Upgrade TimescaleDB to 2.18.2.

You may need to upgrade to the latest Postgres version before you upgrade TimescaleDB. Also, if you use TimescaleDB Toolkit, ensure the timescaledb_toolkit extension is >= v1.6.0 before you upgrade TimescaleDB extension.

TimescaleDB version	Postgres 17	Postgres 16	Postgres 15	Postgres 14	Postgres 13	Postgres 12	Postgres 11	Postgres 10
2.22.x	✅	✅	✅	❌	❌	❌	❌	❌
2.21.x	✅	✅	✅	❌	❌	❌	❌	❌
2.20.x	✅	✅	✅	❌	❌	❌	❌	❌
2.17 - 2.19	✅	✅	✅	✅	❌	❌	❌	❌
2.16.x	❌	✅	✅	✅	❌	❌	❌	❌
2.13 - 2.15	❌	✅	✅	✅	✅	❌	❌	❌
2.12.x	❌	❌	✅	✅	✅	❌	❌	❌
2.10.x	❌	❌	✅	✅	✅	✅	❌	❌
2.5 - 2.9	❌	❌	❌	✅	✅	✅	❌	❌
2.4	❌	❌	❌	❌	✅	✅	❌	❌
2.1 - 2.3	❌	❌	❌	❌	✅	✅	✅	❌
2.0	❌	❌	❌	❌	❌	✅	✅	❌
1.7	❌	❌	❌	❌	❌	✅	✅	✅

We recommend not using TimescaleDB with Postgres 17.1, 16.5, 15.9, 14.14, 13.17, 12.21. These minor versions introduced a breaking binary interface change that, once identified, was reverted in subsequent minor Postgres versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with Postgres 17.2, 16.6, etc and higher. Users of Tiger Cloud and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected.

Implement your upgrade path

You cannot upgrade TimescaleDB and Postgres at the same time. You upgrade each product in the following steps:

1. Upgrade TimescaleDB

   psql -X -d source -c "ALTER EXTENSION timescaledb UPDATE TO '<version number>';"
   

2. If your migration path dictates it, upgrade Postgres

   Follow the procedure in Upgrade Postgres. The version of TimescaleDB installed in your Postgres deployment must be the same before and after the Postgres upgrade.

3. If your migration path dictates it, upgrade TimescaleDB again

   psql -X -d source -c "ALTER EXTENSION timescaledb UPDATE TO '<version number>';"
   

4. Check that you have upgraded to the correct version of TimescaleDB

   psql -X -d source -c "\dx timescaledb;"
   

   Postgres returns something like:

   Name     | Version | Schema |                                      Description
   -------------+---------+--------+---------------------------------------------------------------------------------------
   timescaledb | 2.17.2  | public | Enables scalable inserts and complex queries for time-series data (Community Edition)
   

You are running a shiny new version of TimescaleDB.

===== PAGE: https://docs.tigerdata.com/self-hosted/upgrades/upgrade-docker/ =====

Upgrade TimescaleDB running in Docker

If you originally installed TimescaleDB using Docker, you can upgrade from within the Docker container. This allows you to upgrade to the latest TimescaleDB version while retaining your data.

The timescale/timescaledb-ha* images have the files necessary to run previous versions. Patch releases only contain bugfixes so should always be safe. Non-patch releases may rarely require some extra steps. These steps are mentioned in the release notes for the version of TimescaleDB that you are upgrading to.

After you upgrade the docker image, you run ALTER EXTENSION for all databases using TimescaleDB.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

The examples in this page use a Docker instance called timescaledb. If you have given your Docker instance a different name, replace it when you issue the commands.

Determine the mount point type

When you start your upgraded Docker container, you need to be able to point the new Docker image to the location that contains the data from your previous version. To do this, you need to work out where the current mount point is. The current mount point varies depending on whether your container is using volume mounts, or bind mounts.

1. Find the mount type used by your Docker container:

   docker inspect timescaledb --format='{{range .Mounts }}{{.Type}}{{end}}'
   

   This returns either volume or bind.

2. Note the volume or bind used by your container:

   docker inspect timescaledb --format='{{range .Mounts }}{{.Name}}{{end}}'
   

   Docker returns the <volume ID>. You see something like this:

   069ba64815f0c26783b81a5f0ca813227fde8491f429cf77ed9a5ae3536c0b2c
   

   docker inspect timescaledb --format='{{range .Mounts }}{{.Source}}{{end}}'
   

   Docker returns the <bind path>. You see something like this:

   /path/to/data
   

   You use this value when you perform the upgrade.

Upgrade TimescaleDB within Docker

To upgrade TimescaleDB within Docker, you need to download the upgraded image, stop the old container, and launch the new container pointing to your existing data.

1. Pull the latest TimescaleDB image

   This command pulls the latest version of TimescaleDB running on Postgres 17:

   docker pull timescale/timescaledb-ha:pg17
   

   If you're using another version of Postgres, look for the relevant tag in the TimescaleDB HA repository on Docker Hub.

2. Stop the old container, and remove it

   docker stop timescaledb
   docker rm timescaledb
   

3. Launch a new container with the upgraded Docker image

   Launch based on your mount point type:

   docker run -v <volume ID>:/pgdata -e PGDATA=/pgdata
     -d --name timescaledb -p 5432:5432 timescale/timescaledb-ha:pg17
   

   docker run -v <bind path>:/pgdata -e PGDATA=/pgdata -d --name timescaledb \
     -p 5432:5432 timescale/timescaledb-ha:pg17
   

4. Connect to the upgraded instance using psql with the -X flag

   docker exec -it timescaledb psql -U postgres -X
   

5. At the psql prompt, use the ALTER command to upgrade the extension

   ALTER EXTENSION timescaledb UPDATE;
   CREATE EXTENSION IF NOT EXISTS timescaledb_toolkit;
   ALTER EXTENSION timescaledb_toolkit UPDATE;
   

The TimescaleDB Toolkit extension is packaged with TimescaleDB HA, it includes additional hyperfunctions to help you with queries and data analysis.

If you have multiple databases, update each database separately.

1. Pull the latest TimescaleDB image

   This command pulls the latest version of TimescaleDB running on Postgres 17.

   docker pull timescale/timescaledb:latest-pg17
   

   If you're using another version of Postgres, look for the relevant tag in the TimescaleDB light repository on Docker Hub.

2. Stop the old container, and remove it

   docker stop timescaledb
   docker rm timescaledb
   

3. Launch a new container with the upgraded Docker image

   Launch based on your mount point type:

   docker run -v  <volume ID>:/pgdata -e PGDATA=/pgdata \
     -d --name timescaledb -p 5432:5432 timescale/timescaledb:latest-pg17
   

   docker run -v <bind path>:/pgdata -e PGDATA=/pgdata -d --name timescaledb \
     -p 5432:5432 timescale/timescaledb:latest-pg17
   

4. Connect to the upgraded instance using psql with the -X flag

   docker exec -it timescaledb psql -U postgres -X
   

5. At the psql prompt, use the ALTER command to upgrade the extension

   ALTER EXTENSION timescaledb UPDATE;
   

If you have multiple databases, you need to update each database separately.

===== PAGE: https://docs.tigerdata.com/self-hosted/upgrades/major-upgrade/ =====

Major TimescaleDB upgrades

A major upgrade is when you update from TimescaleDB X.<minor version> to Y.<minor version>. A minor upgrade is when you update from TimescaleDB <major version>.x, to TimescaleDB <major version>.y. You can run different versions of TimescaleDB on different databases within the same Postgres instance. This process uses the Postgres ALTER EXTENSION function to upgrade TimescaleDB independently on different databases.

When you perform a major upgrade, new policies are automatically configured based on your current configuration. In order to verify your policies post upgrade, in this upgrade process you export your policy settings before upgrading.

Tiger Cloud is a fully managed service with automatic backup and restore, high availability with replication, seamless scaling and resizing, and much more. You can try Tiger Cloud free for thirty days.

This page shows you how to perform a major upgrade. For minor upgrades, see Upgrade TimescaleDB to a minor version.

Prerequisites

• Install the Postgres client tools on your migration machine. This includes psql, and pg_dump.
• Read the release notes for the version of TimescaleDB that you are upgrading to.
• Perform a backup of your database. While TimescaleDB upgrades are performed in-place, upgrading is an intrusive operation. Always make sure you have a backup on hand, and that the backup is readable in the case of disaster.

Check the TimescaleDB and Postgres versions

To see the versions of Postgres and TimescaleDB running in a self-hosted database instance:

1. Set your connection string

   This variable holds the connection information for the database to upgrade:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   

2. Retrieve the version of Postgres that you are running

   psql -X -d source -c "SELECT version();"
   

   Postgres returns something like:

   -----------------------------------------------------------------------------------------------------------------------------------------
   PostgreSQL 17.2 (Ubuntu 17.2-1.pgdg22.04+1) on aarch64-unknown-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
   (1 row)
   

3. Retrieve the version of TimescaleDB that you are running

   psql -X -d source -c "\dx timescaledb;"
   

   Postgres returns something like:

       Name     | Version |   Schema   |                             Description
   -------------+---------+------------+---------------------------------------------------------------------
   timescaledb | 2.17.2   | public     | Enables scalable inserts and complex queries for time-series data
   (1 row)
   

Plan your upgrade path

Best practice is to always use the latest version of TimescaleDB. Subscribe to our releases on GitHub or use Tiger Cloud and always get latest update without any hassle.

Check the following support matrix against the versions of TimescaleDB and Postgres that you are running currently and the versions you want to update to, then choose your upgrade path.

For example, to upgrade from TimescaleDB 1.7 on Postgres 12 to TimescaleDB 2.17.2 on Postgres 15 you need to:

1. Upgrade TimescaleDB to 2.10
2. Upgrade Postgres to 15
3. Upgrade TimescaleDB to 2.17.2.

You may need to upgrade to the latest Postgres version before you upgrade TimescaleDB.

TimescaleDB version	Postgres 17	Postgres 16	Postgres 15	Postgres 14	Postgres 13	Postgres 12	Postgres 11	Postgres 10
2.22.x	✅	✅	✅	❌	❌	❌	❌	❌
2.21.x	✅	✅	✅	❌	❌	❌	❌	❌
2.20.x	✅	✅	✅	❌	❌	❌	❌	❌
2.17 - 2.19	✅	✅	✅	✅	❌	❌	❌	❌
2.16.x	❌	✅	✅	✅	❌	❌	❌	❌
2.13 - 2.15	❌	✅	✅	✅	✅	❌	❌	❌
2.12.x	❌	❌	✅	✅	✅	❌	❌	❌
2.10.x	❌	❌	✅	✅	✅	✅	❌	❌
2.5 - 2.9	❌	❌	❌	✅	✅	✅	❌	❌
2.4	❌	❌	❌	❌	✅	✅	❌	❌
2.1 - 2.3	❌	❌	❌	❌	✅	✅	✅	❌
2.0	❌	❌	❌	❌	❌	✅	✅	❌
1.7	❌	❌	❌	❌	❌	✅	✅	✅

We recommend not using TimescaleDB with Postgres 17.1, 16.5, 15.9, 14.14, 13.17, 12.21. These minor versions introduced a breaking binary interface change that, once identified, was reverted in subsequent minor Postgres versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. When you build from source, best practice is to build with Postgres 17.2, 16.6, etc and higher. Users of Tiger Cloud and platform packages for Linux, Windows, MacOS, Docker, and Kubernetes are unaffected.

Check for failed retention policies

When you upgrade from TimescaleDB 1 to TimescaleDB 2, scripts automatically configure updated features to work as expected with the new version. However, not everything works in exactly the same way as previously.

Before you begin this major upgrade, check the database log for errors related to failed retention policies that could have occurred in TimescaleDB 1. You can either remove the failing policies entirely, or update them to be compatible with your existing continuous aggregates.

If incompatible retention policies are present when you perform the upgrade, the ignore_invalidation_older_than setting is automatically turned off, and a notice is shown.

Export your policy settings

1. Set your connection string

   This variable holds the connection information for the database to upgrade:

   export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
   

2. Connect to your Postgres deployment

   psql -d source
   

3. Save your policy statistics settings to a .csv file

   COPY (SELECT * FROM timescaledb_information.policy_stats)
   TO policy_stats.csv csv header
   

4. Save your continuous aggregates settings to a .csv file

   COPY (SELECT * FROM timescaledb_information.continuous_aggregate_stats)
   TO continuous_aggregate_stats.csv csv header
   

5. Save your drop chunk policies to a .csv file

   COPY (SELECT * FROM timescaledb_information.drop_chunks_policies)
   TO drop_chunk_policies.csv csv header
   

6. Save your reorder policies to a .csv file

   COPY (SELECT * FROM timescaledb_information.reorder_policies)
   TO reorder_policies.csv csv header
   

7. Exit your psql session

   \q;
   

Implement your upgrade path

You cannot upgrade TimescaleDB and Postgres at the same time. You upgrade each product in the following steps:

1. Upgrade TimescaleDB

   psql -X -d source -c "ALTER EXTENSION timescaledb UPDATE TO '<version number>';"
   

2. If your migration path dictates it, upgrade Postgres

   Follow the procedure in Upgrade Postgres. The version of TimescaleDB installed in your Postgres deployment must be the same before and after the Postgres upgrade.

3. If your migration path dictates it, upgrade TimescaleDB again

   psql -X -d source -c "ALTER EXTENSION timescaledb UPDATE TO '<version number>';"
   

4. Check that you have upgraded to the correct version of TimescaleDB

   psql -X -d source -c "\dx timescaledb;"
   

   Postgres returns something like:

   Name     | Version | Schema |                                      Description
   -------------+---------+--------+---------------------------------------------------------------------------------------
   timescaledb | 2.17.2  | public | Enables scalable inserts and complex queries for time-series data (Community Edition)
   

To upgrade TimescaleDB in a Docker container, see the Docker container upgrades section.

Verify the updated policy settings and jobs

1. Verify the continuous aggregate policy jobs

   SELECT * FROM timescaledb_information.jobs
     WHERE application_name LIKE 'Refresh Continuous%';
   

   Postgres returns something like:

   -[ RECORD 1 ]-----+--------------------------------------------------
   job_id            | 1001
   application_name  | Refresh Continuous Aggregate Policy [1001]
   schedule_interval | 01:00:00
   max_runtime       | 00:00:00
   max_retries       | -1
   retry_period      | 01:00:00
   proc_schema       | _timescaledb_internal
   proc_name         | policy_refresh_continuous_aggregate
   owner             | postgres
   scheduled         | t
   config            | {"start_offset": "20 days", "end_offset": "10
   days", "mat_hypertable_id": 2}
   next_start        | 2020-10-02 12:38:07.014042-04
   hypertable_schema | _timescaledb_internal
   hypertable_name   | _materialized_hypertable_2
   

2. Verify the information for each policy type that you exported before you upgraded.

   For continuous aggregates, take note of the config information to verify that all settings were converted correctly.

3. Verify that all jobs are scheduled and running as expected

   SELECT * FROM timescaledb_information.job_stats
     WHERE job_id = 1001;
   

   Postgres returns something like:

   -[ RECORD 1 ]----------+------------------------------
   hypertable_schema      | _timescaledb_internal
   hypertable_name        | _materialized_hypertable_2
   job_id                 | 1001
   last_run_started_at    | 2020-10-02 09:38:06.871953-04
   last_successful_finish | 2020-10-02 09:38:06.932675-04
   last_run_status        | Success
   job_status             | Scheduled
   last_run_duration      | 00:00:00.060722
   next_scheduled_run     | 2020-10-02 10:38:06.932675-04
   total_runs             | 1
   total_successes        | 1
   total_failures         | 0
   

You are running a shiny new version of TimescaleDB.

===== PAGE: https://docs.tigerdata.com/self-hosted/multinode-timescaledb/multinode-ha/ =====

High availability with multi-node

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

A multi-node installation of TimescaleDB can be made highly available by setting up one or more standbys for each node in the cluster, or by natively replicating data at the chunk level.

Using standby nodes relies on streaming replication and you set it up in a similar way to configuring single-node HA, although the configuration needs to be applied to each node independently.

To replicate data at the chunk level, you can use the built-in capabilities of multi-node TimescaleDB to avoid having to replicate entire data nodes. The access node still relies on a streaming replication standby, but the data nodes need no additional configuration. Instead, the existing pool of data nodes share responsibility to host chunk replicas and handle node failures.

There are advantages and disadvantages to each approach. Setting up standbys for each node in the cluster ensures that standbys are identical at the instance level, and this is a tried and tested method to provide high availability. However, it also requires more setting up and maintenance for the mirror cluster.

Native replication typically requires less resources, nodes, and configuration, and takes advantage of built-in capabilities, such as adding and removing data nodes, and different replication factors on each distributed hypertable. However, only chunks are replicated on the data nodes.

The rest of this section discusses native replication. To set up standbys for each node, follow the instructions for single node HA.

Native replication

Native replication is a set of capabilities and APIs that allow you to build a highly available multi-node TimescaleDB installation. At the core of native replication is the ability to write copies of a chunk to multiple data nodes in order to have alternative chunk replicas in case of a data node failure. If one data node fails, its chunks should be available on at least one other data node. If a data node is permanently lost, a new data node can be added to the cluster, and lost chunk replicas can be re-replicated from other data nodes to reach the number of desired chunk replicas.

Native replication in TimescaleDB is under development and currently lacks functionality for a complete high-availability solution. Some functionality described in this section is still experimental. For production environments, we recommend setting up standbys for each node in a multi-node cluster.

Automation

Similar to how high-availability configurations for single-node Postgres uses a system like Patroni for automatically handling fail-over, native replication requires an external entity to orchestrate fail-over, chunk re-replication, and data node management. This orchestration is not provided by default in TimescaleDB and therefore needs to be implemented separately. The sections below describe how to enable native replication and the steps involved to implement high availability in case of node failures.

Configuring native replication

The first step to enable native replication is to configure a standby for the access node. This process is identical to setting up a single node standby.

The next step is to enable native replication on a distributed hypertable. Native replication is governed by the replication_factor, which determines how many data nodes a chunk is replicated to. This setting is configured separately for each hypertable, which means the same database can have some distributed hypertables that are replicated and others that are not.

By default, the replication factor is set to 1, so there is no native replication. You can increase this number when you create the hypertable. For example, to replicate the data across a total of three data nodes:

SELECT create_distributed_hypertable('conditions', 'time', 'location',
 replication_factor => 3);

Alternatively, you can use the set_replication_factor call to change the replication factor on an existing distributed hypertable. Note, however, that only new chunks are replicated according to the updated replication factor. Existing chunks need to be re-replicated by copying those chunks to new data nodes (see the node failures section below).

When native replication is enabled, the replication happens whenever you write data to the table. On every INSERT and COPY call, each row of the data is written to multiple data nodes. This means that you don't need to do any extra steps to have newly ingested data replicated. When you query replicated data, the query planner only includes one replica of each chunk in the query plan.

Node failures

When a data node fails, inserts that attempt to write to the failed node result in an error. This is to preserve data consistency in case the data node becomes available again. You can use the alter_data_node call to mark a failed data node as unavailable by running this query:

SELECT alter_data_node('data_node_2', available => false);

Setting available => false means that the data node is no longer used for reads and writes queries.

To fail over reads, the alter_data_node call finds all the chunks for which the unavailable data node is the primary query target and fails over to a chunk replica on another data node. However, if some chunks do not have a replica to fail over to, a warning is raised. Reads continue to fail for chunks that do not have a chunk replica on any other data nodes.

To fail over writes, any activity that intends to write to the failed node marks the involved chunk as stale for the specific failed node by changing the metadata on the access node. This is only done for natively replicated chunks. This allows you to continue to write to other chunk replicas on other data nodes while the failed node has been marked as unavailable. Writes continue to fail for chunks that do not have a chunk replica on any other data nodes. Also note that chunks on the failed node which do not get written into are not affected.

When you mark a chunk as stale, the chunk becomes under-replicated. When the failed data node becomes available then such chunks can be re-balanced using the copy_chunk API.

If waiting for the data node to come back is not an option, either because it takes too long or the node is permanently failed, one can delete it instead. To be able to delete a data node, all of its chunks must have at least one replica on other data nodes. For example:

SELECT delete_data_node('data_node_2', force => true);
WARNING:  distributed hypertable "conditions" is under-replicated

Use the force option when you delete the data node if the deletion means that the cluster no longer achieves the desired replication factor. This would be the normal case unless the data node has no chunks or the distributed hypertable has more chunk replicas than the configured replication factor.

You cannot force the deletion of a data node if it would mean that a multi-node cluster permanently loses data.

When you have successfully removed a failed data node, or marked a failed data node unavailable, some data chunks might lack replicas but queries and inserts work as normal again. However, the cluster stays in a vulnerable state until all chunks are fully replicated.

When you have restored a failed data node or marked it available again, you can see the chunks that need to be replicated with this query:

SELECT chunk_schema, chunk_name, replica_nodes, non_replica_nodes
FROM timescaledb_experimental.chunk_replication_status
WHERE hypertable_name = 'conditions' AND num_replicas < desired_num_replicas;

The output from this query looks like this:

     chunk_schema      |      chunk_name       | replica_nodes |     non_replica_nodes
-----------------------+-----------------------+---------------+---------------------------
 _timescaledb_internal | _dist_hyper_1_1_chunk | {data_node_3} | {data_node_1,data_node_2}
 _timescaledb_internal | _dist_hyper_1_3_chunk | {data_node_1} | {data_node_2,data_node_3}
 _timescaledb_internal | _dist_hyper_1_4_chunk | {data_node_3} | {data_node_1,data_node_2}
(3 rows)

With the information from the chunk replication status view, an under-replicated chunk can be copied to a new node to ensure the chunk has the sufficient number of replicas. For example:

CALL timescaledb_experimental.copy_chunk('_timescaledb_internal._dist_hyper_1_1_chunk', 'data_node_3', 'data_node_2');

When you restore chunk replication, the operation uses more than one transaction. This means that it cannot be automatically rolled back. If you cancel the operation before it is completed, an operation ID for the copy is logged. You can use this operation ID to clean up any state left by the cancelled operation. For example:

CALL timescaledb_experimental.cleanup_copy_chunk_operation('ts_copy_1_31');

===== PAGE: https://docs.tigerdata.com/self-hosted/multinode-timescaledb/multinode-setup/ =====

Set up multi-node on self-hosted TimescaleDB

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

To set up multi-node on a self-hosted TimescaleDB instance, you need:

• A Postgres instance to act as an access node (AN)
• One or more Postgres instances to act as data nodes (DN)
• TimescaleDB installed and set up on all nodes
• Access to a superuser role, such as postgres, on all nodes

The access and data nodes must begin as individual TimescaleDB instances. They should be hosts with a running Postgres server and a loaded TimescaleDB extension. For more information about installing self-hosted TimescaleDB instances, see the installation instructions. Additionally, you can configure high availability with multi-node to increase redundancy and resilience.

The multi-node TimescaleDB architecture consists of an access node (AN) which stores metadata for the distributed hypertable and performs query planning across the cluster, and a set of data nodes (DNs) which store subsets of the distributed hypertable dataset and execute queries locally. For more information about the multi-node architecture, see about multi-node.

If you intend to use continuous aggregates in your multi-node environment, check the additional considerations in the continuous aggregates section.

Set up multi-node on self-hosted TimescaleDB

When you have installed TimescaleDB on the access node and as many data nodes as you require, you can set up multi-node and create a distributed hypertable.

Before you begin, make sure you have considered what partitioning method you want to use for your multi-node cluster. For more information about multi-node and architecture, see the About multi-node section.

Setting up multi-node on self-hosted TimescaleDB

1. On the access node (AN), run this command and provide the hostname of the first data node (DN1) you want to add:

   SELECT add_data_node('dn1', 'dn1.example.com')
   

2. Repeat for all other data nodes:

   SELECT add_data_node('dn2', 'dn2.example.com')
   SELECT add_data_node('dn3', 'dn3.example.com')
   

3. On the access node, create the distributed hypertable with your chosen partitioning. In this example, the distributed hypertable is called example, and it is partitioned on time and location:

   SELECT create_distributed_hypertable('example', 'time', 'location');
   

4. Insert some data into the hypertable. For example:

   INSERT INTO example VALUES ('2020-12-14 13:45', 1, '1.2.3.4');
   

When you have set up your multi-node installation, you can configure your cluster. For more information, see the configuration section.

===== PAGE: https://docs.tigerdata.com/self-hosted/multinode-timescaledb/multinode-auth/ =====

Multi-node authentication

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

When you have your instances set up, you need to configure them to accept connections from the access node to the data nodes. The authentication mechanism you choose for this can be different than the one used by external clients to connect to the access node.

How you set up your multi-node cluster depends on which authentication mechanism you choose. The options are:

• Trust authentication. This is the simplest approach, but also the least secure. This is a good way to start if you are trying out multi-node, but is not recommended for production clusters.
• Pasword authentication. Every user role requires an internal password for establishing connections between the access node and the data nodes. This method is easier to set up than certificate authentication, but provides only a basic level of protection.
• Certificate authentication. Every user role requires a certificate from a certificate authority to establish connections between the access node and the data nodes. This method is more complex to set up than password authentication, but more secure and easier to automate.

Going beyond the simple trust approach to create a secure system can be complex, but it is important to secure your database appropriately for your environment. We do not recommend any one security model, but encourage you to perform a risk assessment and implement the security model that best suits your environment.

Trust authentication

Trusting all incoming connections is the quickest way to get your multi-node environment up and running, but it is not a secure method of operation. Use this only for developing a proof of concept, do not use this method for production installations.

The trust authentication method allows insecure access to all nodes. Do not use this method in production. It is not a secure method of operation.

Setting up trust authentication

1. Connect to the access node with psql, and locate the pg_hba.conf file:

   SHOW hba_file;
   

2. Open the pg_hba.conf file in your preferred text editor, and add this line. In this example, the access node is located at IP 192.0.2.20 with a mask length of 32. You can add one of these two lines:

   
   
   host    all             all             192.0.2.20/32            trust
   
   
   host    all             all             192.0.2.20      255.255.255.255    trust
   
   

3. At the command prompt, reload the server configuration:

   pg_ctl reload
   

   On some operating systems, you might need to use the pg_ctlcluster command instead.

4. If you have not already done so, add the data nodes to the access node. For instructions, see the multi-node setup section.

5. On the access node, create the trust role. In this example, we call the role testrole:

   CREATE ROLE testrole;
   

   OPTIONAL: If external clients need to connect to the access node as testrole, add the LOGIN option when you create the role. You can also add the PASSWORD option if you want to require external clients to enter a password.

6. Allow the trust role to access the foreign server objects for the data nodes. Make sure you include all the data node names:

   GRANT USAGE ON FOREIGN SERVER <data node name>, <data node name>, ... TO testrole;
   

7. On the access node, use the distributed_exec command to add the role to all the data nodes:

   CALL distributed_exec($$ CREATE ROLE testrole LOGIN $$);
   

Make sure you create the role with the LOGIN privilege on the data nodes, even if you don't use this privilege on the access node. For all other privileges, ensure they are same on the access node and the data nodes.

Password authentication

Password authentication requires every user role to know a password before it can establish a connection between the access node and the data nodes. This internal password is only used by the access node and it does not need to be the same password as the client uses to connect to the access node. External users do not need to share the internal password at all, it can be set up and administered by the database administrator.

The access node stores the internal password so that it can verify the correct password has been provided by a data node. We recommend that you store the password on the access node in a local password file, and this section shows you how to set this up. However, if it works better in your environment, you can use user mappings to store your passwords instead. This is slightly less secure than a local pasword file, because it requires one mapping for each data node in your cluster.

This section sets up your password authentication using SCRAM SHA-256 password authentication. For other password authentication methods, see the Postgres authentication documentation.

Before you start, check that you can use the postgres username to log in to your access node.

Setting up password authentication

1. On the access node, open the postgresql.conf configuration file, and add or edit this line:

   password_encryption = 'scram-sha-256'  # md5 or scram-sha-256
   

2. Repeat for each of the data nodes.

3. On each of the data nodes, at the psql prompt, locate the pg_hba.conf configuration file:

   SHOW hba_file
   

4. On each of the data nodes, open the pg_hba.conf configuration file, and add or edit this line to enable encrypted authentication to the access node:

   host    all       all   192.0.2.20   scram-sha-256 #where '192.0.2.20' is the access node IP
   

5. On the access node, open or create the password file at data/passfile. This file stores the passwords for each role that the access node connects to on the data nodes. If you need to change the location of the password file, adjust the timescaledb.passfile setting in the postgresql.conf configuration file.

6. On the access node, open the passfile file, and add a line like this for each user, starting with the postgres user:

   *:*:*:postgres:xyzzy #assuming 'xyzzy' is the password for the 'postgres' user
   

7. On the access node, at the command prompt, change the permissions of the passfile file:

   chmod 0600 passfile
   

8. On the access node, and on each of the data nodes, reload the server configuration to pick up the changes:

   pg_ctl reload
   

9. If you have not already done so, add the data nodes to the access node. For instructions, see the multi-node setup section.

10. On the access node, at the psql prompt, create additional roles, and grant them access to foreign server objects for the data nodes:

    CREATE ROLE testrole PASSWORD 'clientpass' LOGIN;
    GRANT USAGE ON FOREIGN SERVER <data node name>, <data node name>, ... TO testrole;
    

    The clientpass password is used by external clients to connect to the access node as user testrole. If the access node is configured to accept other authentication methods, or the role is not a login role, then you might not need to do this step.

11. On the access node, add the new role to each of the data nodes with distributed_exec. Make sure you add the PASSWORD parameter to specify a different password to use when connecting to the data nodes with role testrole:

    CALL distributed_exec($$ CREATE ROLE testrole PASSWORD 'internalpass' LOGIN $$);
    

12. On the access node, add the new role to the passfile you created earlier, by adding this line:

    *:*:*:testrole:internalpass #assuming 'internalpass' is the password used to connect to data nodes
    

Any user passwords that you created before you set up password authentication need to be re-created so that they use the new encryption method.

Certificate authentication

This method is a bit more complex to set up than password authentication, but it is more secure, easier to automate, and can be customized to your security environment.

To use certificates, the access node and each data node need three files:

• The root CA certificate, called root.crt. This certificate serves as the root of trust in the system. It is used to verify the other certificates.
• A node certificate, called server.crt. This certificate provides the node with a trusted identity in the system.
• A node certificate key, called server.key. This provides proof of ownership of the node certificate. Make sure you keep this file private on the node where it is generated.

You can purchase certificates from a commercial certificate authority (CA), or generate your own self-signed CA. This section shows you how to use your access node certificate to create and sign new user certificates for the data nodes.

Keys and certificates serve different purposes on the data nodes and access node. For the access node, a signed certificate is used to verify user certificates for access. For the data nodes, a signed certificate authenticates the node to the access node.

Generating a self-signed root certificate for the access node

1. On the access node, at the command prompt, generate a private key called auth.key:

   openssl genpkey -algorithm rsa -out auth.key
   

2. Generate a self-signed root certificate for the certificate authority (CA), called root.cert:

   openssl req -new -key auth.key -days 3650 -out root.crt -x509
   

3. Complete the questions asked by the script to create your root certificate. Type your responses in, press enter to accept the default value shown in brackets, or type . to leave the field blank. For example:

   Country Name (2 letter code) [AU]:US
   State or Province Name (full name) [Some-State]:New York
   Locality Name (eg, city) []:New York
   Organization Name (eg, company) [Internet Widgets Pty Ltd]:Example Company Pty Ltd
   Organizational Unit Name (eg, section) []:
   Common Name (e.g. server FQDN or YOUR name) []:http://cert.example.com/
   Email Address []:
   

When you have created the root certificate on the access node, you can generate certificates and keys for each of the data nodes. To do this, you need to create a certificate signing request (CSR) for each data node.

The default names for the key is server.key, and for the certificate is server.crt. They are stored in together, in the data directory on the data node instance.

The default name for the CSR is server.csr and you need to sign it using the root certificate you created on the access node.

Generating keys and certificates for data nodes

1. On the access node, generate a certificate signing request (CSR) called server.csr, and create a new key called server.key:

   openssl req -out server.csr -new -newkey rsa:2048 -nodes \
   -keyout server.key
   

2. Sign the CSR using the root certificate CA you created earlier, called auth.key:

   openssl ca -extensions v3_intermediate_ca -days 3650 -notext \
   -md sha256 -in server.csr -out server.crt
   

3. Move the server.crt and server.key files from the access node, on to each data node, in the data directory. Depending on your network setup, you might need to use portable media.

4. Copy the root certificate file root.crt from the access node, on to each data node, in the data directory. Depending on your network setup, you might need to use portable media.

When you have created the certificates and keys, and moved all the files into the right places on the data nodes, you can configure the data nodes to use SSL authentication.

Configuring data nodes to use SSL authentication

1. On each data node, open the postgresql.conf configuration file and add or edit the SSL settings to enable certificate authentication:

   ssl = on
   ssl_ca_file = 'root.crt'
   ssl_cert_file = 'server.crt'
   ssl_key_file = 'server.key'
   

2. If you want the access node to use certificate authentication for login, make these changes on the access node as well.

3. On each data node, open the pg_hba.conf configuration file, and add or edit this line to allow any SSL user log in with client certificate authentication:

   hostssl   all       all         all       cert    clientcert=1
   

If you are using the default names for your certificate and key, you do not need to explicitly set them. The configuration looks for server.crt and server.key by default. If you use different names for your certificate and key, make sure you specify the correct names in the postgresql.conf configuration file.

When your data nodes are configured to use SSL certificate authentication, you need to create a signed certificate and key for your access node. This allows the access node to log in to the data nodes.

Creating certificates and keys for the access node

1. On the access node, as the postgres user, compute a base name for the certificate files using md5sum, generate a subject identifier, and create names for the key and certificate files:

   pguser=postgres
   base=`echo -n $pguser | md5sum | cut -c1-32`
   subj="/C=US/ST=New York/L=New York/O=Timescale/OU=Engineering/CN=$pguser"
   key_file="timescaledb/certs/$base.key"
   crt_file="timescaledb/certs/$base.crt"
   

2. Generate a new random user key:

   openssl genpkey -algorithm RSA -out "$key_file"
   

3. Generate a certificate signing request (CSR). This file is temporary, stored in the data directory, and is deleted later on:

   openssl req -new -sha256 -key $key_file -out "$base.csr" -subj "$subj"
   

4. Sign the CSR with the access node key:

   openssl ca -batch -keyfile server.key -extensions v3_intermediate_ca \
     -days 3650 -notext -md sha256 -in "$base.csr" -out "$crt_file"
   rm $base.csr
   

5. Append the node certificate to the user certificate. This completes the certificate verification chain and makes sure that all certificates are available on the data node, up to the trusted certificate stored in root.crt:

   cat >>$crt_file <server.crt
   

By default, the user key files and certificates are stored on the access node in the data directory, under timescaledb/certs. You can change this location using the timescaledb.ssl_dir configuration variable.

Your data nodes are now set up to accept certificate authentication, the data and access nodes have keys, and the postgres user has a certificate. If you have not already done so, add the data nodes to the access node. For instructions, see the multi-node setup section. The final step is add additional user roles.

Setting up additional user roles

1. On the access node, at the psql prompt, create the new user and grant permissions:

   CREATE ROLE testrole;
   GRANT USAGE ON FOREIGN SERVER <data node name>, <data node name>, ... TO testrole;
   

   If you need external clients to connect to the access node as testrole, make sure you also add the LOGIN option. You can also enable password authentication by adding the PASSWORD option.

2. On the access node, use the distributed_exec command to add the role to all the data nodes:

   CALL distributed_exec($$ CREATE ROLE testrole LOGIN $$);
   

===== PAGE: https://docs.tigerdata.com/self-hosted/multinode-timescaledb/multinode-grow-shrink/ =====

Grow and shrink multi-node

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

When you are working within a multi-node environment, you might discover that you need more or fewer data nodes in your cluster over time. You can choose how many of the available nodes to use when creating a distributed hypertable. You can also add and remove data nodes from your cluster, and move data between chunks on data nodes as required to free up storage.

See which data nodes are in use

You can check which data nodes are in use by a distributed hypertable, using this query. In this example, our distributed hypertable is called conditions:

SELECT hypertable_name, data_nodes
FROM timescaledb_information.hypertables
WHERE hypertable_name = 'conditions';

The result of this query looks like this:

hypertable_name |              data_nodes
-----------------+---------------------------------------
conditions      | {data_node_1,data_node_2,data_node_3}

Choose how many nodes to use for a distributed hypertable

By default, when you create a distributed hypertable, it uses all available data nodes. To restrict it to specific nodes, pass the data_nodes argument to create_distributed_hypertable.

Attach a new data node

When you add additional data nodes to a database, you need to add them to the distributed hypertable so that your database can use them.

Attaching a new data node to a distributed hypertable

1. On the access node, at the psql prompt, add the data node:

   SELECT add_data_node('node3', host => 'dn3.example.com');
   

2. Attach the new data node to the distributed hypertable:

   SELECT attach_data_node('node3', hypertable => 'hypertable_name');
   

When you attach a new data node, the partitioning configuration of the distributed hypertable is updated to account for the additional data node, and the number of hash partitions are automatically increased to match. You can prevent this happening by setting the function parameter repartition to FALSE.

Move data between chunks Experimental

When you attach a new data node to a distributed hypertable, you can move existing data in your hypertable to the new node to free up storage on the existing nodes and make better use of the added capacity.

The ability to move chunks between data nodes is an experimental feature that is under active development. We recommend that you do not use this feature in a production environment.

Move data using this query:

CALL timescaledb_experimental.move_chunk('_timescaledb_internal._dist_hyper_1_1_chunk', 'data_node_3', 'data_node_2');

The move operation uses a number of transactions, which means that you cannot roll the transaction back automatically if something goes wrong. If a move operation fails, the failure is logged with an operation ID that you can use to clean up any state left on the involved nodes.

Clean up after a failed move using this query. In this example, the operation ID of the failed move is ts_copy_1_31:

CALL timescaledb_experimental.cleanup_copy_chunk_operation('ts_copy_1_31');

Remove a data node

You can also remove data nodes from an existing distributed hypertable.

You cannot remove a data node that still contains data for the distributed hypertable. Before you remove the data node, check that is has had all of its data deleted or moved, or that you have replicated the data on to other data nodes.

Remove a data node using this query. In this example, our distributed hypertable is called conditions:

SELECT detach_data_node('node1', hypertable => 'conditions');

===== PAGE: https://docs.tigerdata.com/self-hosted/multinode-timescaledb/multinode-administration/ =====

Multi-node administration

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Multi-node TimescaleDB allows you to administer your cluster directly from the access node. When your environment is set up, you do not need to log directly into the data nodes to administer your database.

When you perform an administrative task, such as adding a new column, changing privileges, or adding an index on a distributed hypertable, you can perform the task from the access node and it is applied to all the data nodes. If a command is executed on a regular table, however, the effects of that command are only applied locally on the access node. Similarly, if a command is executed directly on a data node, the result is only visible on that data node.

Commands that create or modify schemas, roles, tablespaces, and settings in a distributed database are not automatically distributed either. That is because these objects and settings sometimes need to be different on the access node compared to the data nodes, or even vary among data nodes. For example, the data nodes could have unique CPU, memory, and disk configurations. The node differences make it impossible to assume that a single configuration works for all nodes. Further, some settings need to be different on the publicly accessible access node compared to data nodes, such as having different connection limits. A role might not have the LOGIN privilege on the access node, but it needs this privilege on data nodes so that the access node can connect.

Roles and tablespaces are also shared across multiple databases on the same instance. Some of these databases might be distributed and some might not be, or be configured with a different set of data nodes. Therefore, it is not possible to know for sure when a role or tablespace should be distributed to a data node given that these commands can be executed from within different databases, that need not be distributed.

To administer a multi-node cluster from the access node, you can use the distributed_exec function. This function allows full control over creating and configuring, database settings, schemas, roles, and tablespaces across all data nodes.

The rest of this section describes in more detail how specific administrative tasks are handled in a multi-node environment.

Distributed role management

In a multi-node environment, you need to manage roles on each Postgres instance independently, because roles are instance-level objects that are shared across both distributed and non-distributed databases that each can be configured with a different set of data nodes or none at all. Therefore, an access node does not automatically distribute roles or role management commands across its data nodes. When a data node is added to a cluster, it is assumed that it already has the proper roles necessary to be consistent with the rest of the nodes. If this is not the case, you might encounter unexpected errors when you try to create or alter objects that depend on a role that is missing or set incorrectly.

To help manage roles from the access node, you can use the distributed_exec function. This is useful for creating and configuring roles across all data nodes in the current database.

Creating a distributed role

When you create a distributed role, it is important to consider that the same role might require different configuration on the access node compared to the data nodes. For example, a user might require a password to connect to the access node, while certificate authentication is used between nodes within the cluster. You might also want a connection limit for external connections, but allow unlimited internal connections to data nodes. For example, the following user can use a password to make 10 connections to the access node but has no limits connecting to the data nodes:

CREATE ROLE alice WITH LOGIN PASSWORD 'mypassword' CONNECTION LIMIT 10;
CALL distributed_exec($$ CREATE ROLE alice WITH LOGIN CONNECTION LIMIT -1; $$);

For more information about setting up authentication, see the multi-node authentication section.

Some roles can also be configured without the LOGIN attribute on the access node. This allows you to switch to the role locally, but not connect with the user from a remote location. However, to be able to connect from the access node to a data node as that user, the data nodes need to have the role configured with the LOGIN attribute enabled. To create a non-login role for a multi-node setup, use these commands:

CREATE ROLE alice WITHOUT LOGIN;
CALL distributed_exec($$ CREATE ROLE alice WITH LOGIN; $$);

To allow a new role to create distributed hypertables it also needs to be granted usage on data nodes, for example:

GRANT USAGE ON FOREIGN SERVER dn1,dn2,dn3 TO alice;

By granting usage on some data nodes, but not others, you can restrict usage to a subset of data nodes based on the role.

Alter a distributed role

When you alter a distributed role, use the same process as creating roles. The role needs to be altered on the access node and on the data nodes in two separate steps. For example, add the CREATEROLE attribute to a role as follows:

ALTER ROLE alice CREATEROLE;
CALL distributed_exec($$ ALTER ROLE alice CREATEROLE; $$);

Manage distributed databases

A distributed database can contain both distributed and non-distributed objects. In general, when a command is issued to alter a distributed object, it applies to all nodes that have that object (or a part of it).

However, in some cases settings should be different depending on node, because nodes might be provisioned differently (having, for example, varying levels of CPU, memory, and disk capabilities) and the role of the access node is different from a data node's.

This section describes how and when commands on distributed objects are applied across all data nodes when executed from within a distributed database.

Alter a distributed database

The ALTER DATABASE command is only applied locally on the access node. This is because database-level configuration often needs to be different across nodes. For example, this is a setting that might differ depending on the CPU capabilities of the node:

ALTER DATABASE mydatabase SET max_parallel_workers TO 12;

The database names can also differ between nodes, even if the databases are part of the same distributed database. When you rename a data node's database, also make sure to update the configuration of the data node on the access node so that it references the new database name.

Drop a distributed database

When you drop a distributed database on the access node, it does not automatically drop the corresponding databases on the data nodes. In this case, you need to connect directly to each data node and drop the databases locally.

A distributed database is not automatically dropped across all nodes, because the information about data nodes lives within the distributed database on the access node, but it is not possible to read it when executing the drop command since it cannot be issued when connected to the database.

Additionally, if a data node has permanently failed, you need to be able to drop a database even if one or more data nodes are not responding.

It is also good practice to leave the data intact on a data node if possible. For example, you might want to back up a data node even after a database was dropped on the access node.

Alternatively, you can delete the data nodes with the drop_database option prior to dropping the database on the access node:

SELECT * FROM delete_data_node('dn1', drop_database => true);

Create, alter, and drop schemas

When you create, alter, or drop schemas, the commands are not automatically applied across all data nodes. A missing schema is, however, created when a distributed hypertable is created, and the schema it belongs to does not exist on a data node.

To manually create a schema across all data nodes, use this command:

CREATE SCHEMA newschema;
CALL distributed_exec($$ CREATE SCHEMA newschema $$);

If a schema is created with a particular authorization, then the authorized role must also exist on the data nodes prior to issuing the command. The same things applies to altering the owner of an existing schema.

Prepare for role removal with DROP OWNED

The DROP OWNED command is used to drop all objects owned by a role and prepare the role for removal. Execute the following commands to prepare a role for removal across all data nodes in a distributed database:

DROP OWNED BY alice CASCADE;
CALL distributed_exec($$ DROP OWNED BY alice CASCADE $$);

Note, however, that the role might still own objects in other databases after these commands have been executed.

Manage privileges

Privileges configured using GRANT or REVOKE statements are applied to all data nodes when they are run on a distributed hypertable. When granting privileges on other objects, the command needs to be manually distributed with distributed_exec.

Set default privileges

Default privileges need to be manually modified using distributed_exec, if they are to apply across all data nodes. The roles and schemas that the default privileges reference need to exist on the data nodes prior to executing the command.

New data nodes are assumed to already have any altered default privileges. The default privileges are not automatically applied retrospectively to new data nodes.

Manage tablespaces

Nodes might be configured with different disks, and therefore tablespaces need to be configured manually on each node. In particular, an access node might not have the same storage configuration as data nodes, since it typically does not store a lot of data. Therefore, it is not possible to assume that the same tablespace configuration exists across all nodes in a multi-node cluster.

===== PAGE: https://docs.tigerdata.com/self-hosted/multinode-timescaledb/about-multinode/ =====

About multi-node

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

If you have a larger petabyte-scale workload, you might need more than one TimescaleDB instance. TimescaleDB multi-node allows you to run and manage a cluster of databases, which can give you faster data ingest, and more responsive and efficient queries for large workloads.

In some cases, your queries could be slower in a multi-node cluster due to the extra network communication between the various nodes. Queries perform the best when the query processing is distributed among the nodes and the result set is small relative to the queried dataset. It is important that you understand multi-node architecture before you begin, and plan your database according to your specific requirements.

Multi-node architecture

Multi-node TimescaleDB allows you to tie several databases together into a logical distributed database to combine the processing power of many physical Postgres instances.

One of the databases exists on an access node and stores metadata about the other databases. The other databases are located on data nodes and hold the actual data. In theory, a Postgres instance can serve as both an access node and a data node at the same time in different databases. However, it is recommended not to have mixed setups, because it can be complicated, and server instances are often provisioned differently depending on the role they serve.

For self-hosted installations, create a server that can act as an access node, then use that access node to create data nodes on other servers.

When you have configured multi-node TimescaleDB, the access node coordinates the placement and access of data chunks on the data nodes. In most cases, it is recommend that you use multidimensional partitioning to distribute data across chunks in both time and space dimensions. The figure in this section shows how an access node (AN) partitions data in the same time interval across multiple data nodes (DN1, DN2, and DN3).

A database user connects to the access node to issue commands and execute queries, similar to how one connects to a regular single node TimescaleDB instance. In most cases, connecting directly to the data nodes is not necessary.

Because TimescaleDB exists as an extension within a specific database, it is possible to have both distributed and non-distributed databases on the same access node. It is also possible to have several distributed databases that use different sets of physical instances as data nodes. In this section, however, it is assumed that you have a single distributed database with a consistent set of data nodes.

Distributed hypertables

If you use a regular table or hypertable on a distributed database, they are not automatically distributed. Regular tables and hypertables continue to work as usual, even when the underlying database is distributed. To enable multi-node capabilities, you need to explicitly create a distributed hypertable on the access node to make use of the data nodes. A distributed hypertable is similar to a regular hypertable, but with the difference that chunks are distributed across data nodes instead of on local storage. By distributing the chunks, the processing power of the data nodes is combined to achieve higher ingest throughput and faster queries. However, the ability to achieve good performance is highly dependent on how the data is partitioned across the data nodes.

To achieve good ingest performance, write the data in batches, with each batch containing data that can be distributed across many data nodes. To achieve good query performance, spread the query across many nodes and have a result set that is small relative to the amount of processed data. To achieve this, it is important to consider an appropriate partitioning method.

Partitioning methods

Data that is ingested into a distributed hypertable is spread across the data nodes according to the partitioning method you have chosen. Queries that can be sent from the access node to multiple data nodes and processed simultaneously generally run faster than queries that run on a single data node, so it is important to think about what kind of data you have, and the type of queries you want to run.

TimescaleDB multi-node currently supports capabilities that make it best suited for large-volume time-series workloads that are partitioned on time, and a space dimension such as location. If you usually run wide queries that aggregate data across many locations and devices, choose this partitioning method. For example, a query like this is faster on a database partitioned on time,location, because it spreads the work across all the data nodes in parallel:

SELECT time_bucket('1 hour', time) AS hour, location, avg(temperature)
FROM conditions
GROUP BY hour, location
ORDER BY hour, location
LIMIT 100;

Partitioning on time and a space dimension such as location, is also best if you need faster insert performance. If you partition only on time, and your inserts are generally occuring in time order, then you are always writing to one data node at a time. Partitioning on time and location means your time-ordered inserts are spread across multiple data nodes, which can lead to better performance.

If you mostly run deep time queries on a single location, you might see better performance by partitioning solely on the time dimension, or on a space dimension other than location. For example, a query like this is faster on a database partitioned on time only, because the data for a single location is spread across all the data nodes, rather than being on a single one:

SELECT time_bucket('1 hour', time) AS hour, avg(temperature)
FROM conditions
WHERE location = 'office_1'
GROUP BY hour
ORDER BY hour
LIMIT 100;

Transactions and consistency model

Transactions that occur on distributed hypertables are atomic, just like those on regular hypertables. This means that a distributed transaction that involves multiple data nodes is guaranteed to either succeed on all nodes or on none of them. This guarantee is provided by the two-phase commit protocol, which is used to implement distributed transactions in TimescaleDB.

However, the read consistency of a distributed hypertable is different to a regular hypertable. Because a distributed transaction is a set of individual transactions across multiple nodes, each node can commit its local transaction at a slightly different time due to network transmission delays or other small fluctuations. As a consequence, the access node cannot guarantee a fully consistent snapshot of the data across all data nodes. For example, a distributed read transaction might start when another concurrent write transaction is in its commit phase and has committed on some data nodes but not others. The read transaction can therefore use a snapshot on one node that includes the other transaction's modifications, while the snapshot on another data node might not include them.

If you need stronger read consistency in a distributed transaction, then you can use consistent snapshots across all data nodes. However, this requires a lot of coordination and management, which can negatively effect performance, and it is therefore not implemented by default for distributed hypertables.

Using continuous aggregates in a multi-node environment

If you are using self-hosted TimescaleDB in a multi-node environment, there are some additional considerations for continuous aggregates.

When you create a continuous aggregate within a multi-node environment, the continuous aggregate should be created on the access node. While it is possible to create a continuous aggregate on data nodes, it interferes with the continuous aggregates on the access node and can cause problems.

When you refresh a continuous aggregate on an access node, it computes a single window to update the time buckets. This could slow down your query if the actual number of rows that were updated is small, but widely spread apart. This is aggravated if the network latency is high if, for example, you have remote data nodes.

Invalidation logs are on kept on the data nodes, which is designed to limit the amount of data that needs to be transferred. However, some statements send invalidations directly to the log, for example, when dropping a chunk or truncate a hypertable. This action could slow down performance, in comparison to a local update. Additionally, if you have infrequent refreshes but a lot of changes to the hypertable, the invalidation logs could get very large, which could cause performance issues. Make sure you are maintaining your invalidation log size to avoid this, for example, by refreshing the continuous aggregate frequently.

For more information about setting up multi-node, see the multi-node section

===== PAGE: https://docs.tigerdata.com/self-hosted/multinode-timescaledb/multinode-config/ =====

Multi-node configuration

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

In addition to the regular TimescaleDB configuration, it is recommended that you also configure additional settings specific to multi-node operation.

Update settings

Each of these settings can be configured in the postgresql.conf file on the individual node. The postgresql.conf file is usually in the data directory, but you can locate the correct path by connecting to the node with psql and giving this command:

SHOW config_file;

After you have modified the postgresql.conf file, reload the configuration to see your changes:

pg_ctl reload

max_prepared_transactions

If not already set, ensure that max_prepared_transactions is a non-zero value on all data nodes is set to 150 as a starting point.

enable_partitionwise_aggregate

On the access node, set the enable_partitionwise_aggregate parameter to on. This ensures that queries are pushed down to the data nodes, and improves query performance.

jit

On the access node, set jit to off. Currently, JIT does not work well with distributed queries. However, you can enable JIT on the data nodes successfully.

statement_timeout

On the data nodes, disable statement_timeout. If you need to enable this, enable and configure it on the access node only. This setting is disabled by default in Postgres, but can be useful if your specific environment is suited.

wal_level

On the data nodes, set the wal_level to logical or higher to move or copy chunks between data nodes. If you are moving many chunks in parallel, consider increasing max_wal_senders and max_replication_slots as well.

Transaction isolation level

For consistency, if the transaction isolation level is set to READ COMMITTED it is automatically upgraded to REPEATABLE READ whenever a distributed operation occurs. If the isolation level is SERIALIZABLE, it is not changed.

===== PAGE: https://docs.tigerdata.com/self-hosted/multinode-timescaledb/multinode-maintenance/ =====

Multi-node maintenance tasks

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Various maintenance activities need to be carried out for effective upkeep of the distributed multi-node setup. You can use cron or another scheduling system outside the database to run these below maintenance jobs on a regular schedule if you prefer. Also make sure that the jobs are scheduled separately for each database that contains distributed hypertables.

Maintaining distributed transactions

A distributed transaction runs across multiple data nodes, and can remain in a non-completed state if a data node reboots or experiences temporary issues. The access node keeps a log of distributed transactions so that nodes that haven't completed their part of the distributed transaction can complete it later when they become available. This transaction log requires regular cleanup to remove transactions that have completed, and complete those that haven't. We highly recommended that you configure the access node to run a maintenance job that regularly cleans up any unfinished distributed transactions. For example:

= 2.12">

CREATE OR REPLACE PROCEDURE data_node_maintenance(job_id int, config jsonb)
LANGUAGE SQL AS
$$
    SELECT _timescaledb_functions.remote_txn_heal_data_node(fs.oid)
    FROM pg_foreign_server fs, pg_foreign_data_wrapper fdw
    WHERE fs.srvfdw = fdw.oid
    AND fdw.fdwname = 'timescaledb_fdw';
$$;

SELECT add_job('data_node_maintenance', '5m');

CREATE OR REPLACE PROCEDURE data_node_maintenance(job_id int, config jsonb)
LANGUAGE SQL AS
$$
    SELECT _timescaledb_internal.remote_txn_heal_data_node(fs.oid)
    FROM pg_foreign_server fs, pg_foreign_data_wrapper fdw
    WHERE fs.srvfdw = fdw.oid
    AND fdw.fdwname = 'timescaledb_fdw';
$$;

SELECT add_job('data_node_maintenance', '5m');

Statistics for distributed hypertables

On distributed hypertables, the table statistics need to be kept updated. This allows you to efficiently plan your queries. Because of the nature of distributed hypertables, you can't use the auto-vacuum tool to gather statistics. Instead, you can explicitly ANALYZE the distributed hypertable periodically using a maintenance job, like this:

CREATE OR REPLACE PROCEDURE distributed_hypertables_analyze(job_id int, config jsonb)
LANGUAGE plpgsql AS
$$
DECLARE r record;
BEGIN
FOR r IN SELECT hypertable_schema, hypertable_name
              FROM timescaledb_information.hypertables
              WHERE is_distributed ORDER BY 1, 2
LOOP
EXECUTE format('ANALYZE %I.%I', r.hypertable_schema, r.hypertable_name);
END LOOP;
END
$$;

SELECT add_job('distributed_hypertables_analyze', '12h');

You can merge the jobs in this example into a single maintenance job if you prefer. However, analyzing distributed hypertables should be done less frequently than remote transaction healing activity. This is because the former could analyze a large number of remote chunks everytime and can be expensive if called too frequently.

===== PAGE: https://docs.tigerdata.com/self-hosted/migration/migrate-influxdb/ =====

Migrate data to TimescaleDB from InfluxDB

You can migrate data to TimescaleDB from InfluxDB using the Outflux tool. Outflux is an open source tool built by Tiger Data for fast, seamless migrations. It pipes exported data directly to self-hosted TimescaleDB, and manages schema discovery, validation, and creation.

Outflux works with earlier versions of InfluxDB. It does not work with InfluxDB version 2 and later.

Prerequisites

Before you start, make sure you have:

• A running instance of InfluxDB and a means to connect to it.
• An self-hosted TimescaleDB instance and a means to connect to it.
• Data in your InfluxDB instance.

Procedures

To import data from Outflux, follow these procedures:

1. Install Outflux
2. Discover, validate, and transfer schema to self-hosted TimescaleDB (optional)
3. Migrate data to Timescale

Install Outflux

Install Outflux from the GitHub repository. There are builds for Linux, Windows, and MacOS.

1. Go to the releases section of the Outflux repository.
2. Download the latest compressed tarball for your platform.
3. Extract it to a preferred location.

If you prefer to build Outflux from source, see the Outflux README for instructions.

To get help with Outflux, run ./outflux --help from the directory where you installed it.

Discover, validate, and transfer schema

Outflux can:

• Discover the schema of an InfluxDB measurement
• Validate whether a table exists that can hold the transferred data
• Create a new table to satisfy the schema requirements if no valid table exists

Outflux's migrate command does schema transfer and data migration in one step. For more information, see the migrate section. Use this section if you want to validate and transfer your schema independently of data migration.

To transfer your schema from InfluxDB to Timescale, run outflux schema-transfer:

outflux schema-transfer <DATABASE_NAME> <INFLUX_MEASUREMENT_NAME> \
--input-server=http://localhost:8086 \
--output-conn="dbname=tsdb user=tsdbadmin"

To transfer all measurements from the database, leave out the measurement name argument.

This example uses the postgres user and database to connect to the self-hosted TimescaleDB instance. For other connection options and configuration, see the Outflux Github repo.

Schema transfer options

Outflux's schema-transfer can use 1 of 4 schema strategies:

• ValidateOnly: checks that self-hosted TimescaleDB is installed and that the specified database has a properly partitioned hypertable with the correct columns, but doesn't perform modifications
• CreateIfMissing: runs the same checks as ValidateOnly, and creates and properly partitions any missing hypertables
• DropAndCreate: drops any existing table with the same name as the measurement, and creates a new hypertable and partitions it properly
• DropCascadeAndCreate: performs the same action as DropAndCreate, and also executes a cascade table drop if there is an existing table with the same name as the measurement

You can specify your schema strategy by passing a value to the --schema-strategy option in the schema-transfer command. The default strategy is CreateIfMissing.

By default, each tag and field in InfluxDB is treated as a separate column in your TimescaleDB tables. To transfer tags and fields as a single JSONB column, use the flag --tags-as-json.

Migrate data to TimescaleDB

Transfer your schema and migrate your data all at once with the migrate command.

For example, run:

outflux migrate <DATABASE_NAME> <INFLUX_MEASUREMENT_NAME> \
--input-server=http://localhost:8086 \
--output-conn="dbname=tsdb user=tsdbadmin"

The schema strategy and connection options are the same as for schema-transfer. For more information, see Discover, validate, and transfer schema.

In addition, outflux migrate also takes the following flags:

• --limit: Pass a number, N, to --limit to export only the first N rows, ordered by time.
• --from and to: Pass a timestamp to --from or --to to specify a time window of data to migrate.
• chunk-size: Changes the size of data chunks transferred. Data is pulled from the InfluxDB server in chunks of default size 15 000.
• batch-size: Changes the number of rows in an insertion batch. Data is inserted into a self-hosted TimescaleDB database in batches that are 8000 rows by default.

For more flags, see the Github documentation for outflux migrate. Alternatively, see the command line help:

outflux migrate --help

===== PAGE: https://docs.tigerdata.com/self-hosted/migration/entire-database/ =====

Migrate the entire database at once

Migrate smaller databases by dumping and restoring the entire database at once. This method works best on databases smaller than 100 GB. For larger databases, consider migrating your schema and data separately.

Depending on your database size and network speed, migration can take a very long time. You can continue reading from your source database during this time, though performance could be slower. To avoid this problem, fork your database and migrate your data from the fork. If you write to tables in your source database during the migration, the new writes might not be transferred to Timescale. To avoid this problem, see Live migration.

Prerequisites

Before you begin, check that you have:

• Installed the Postgres pg_dump and pg_restore utilities.
• Installed a client for connecting to Postgres. These instructions use psql, but any client works.
• Created a new empty database in your self-hosted TimescaleDB instance. For more information, see Install TimescaleDB. Provision your database with enough space for all your data.
• Checked that any other Postgres extensions you use are compatible with Timescale. For more information, see the list of compatible extensions. Install your other Postgres extensions.
• Checked that you're running the same major version of Postgres on both your target and source databases. For information about upgrading Postgres on your source database, see the upgrade instructions for self-hosted TimescaleDB.
• Checked that you're running the same major version of TimescaleDB on both your target and source databases. For more information, see upgrade self-hosted TimescaleDB.

To speed up migration, compress your data into the columnstore. You can compress any chunks where data is not currently inserted, updated, or deleted. When you finish the migration, you can decompress chunks back to the rowstore as needed for normal operation. For more information about the rowstore and columnstore compression, see hypercore.

Migrating the entire database at once

1. Dump all the data from your source database into a dump.bak file, using your source database connection details. If you are prompted for a password, use your source database credentials:

   pg_dump -U <SOURCE_DB_USERNAME> -W \
   -h <SOURCE_DB_HOST> -p <SOURCE_DB_PORT> -Fc -v \
   -f dump.bak <SOURCE_DB_NAME>
   

2. Connect to your self-hosted TimescaleDB instance using your connection details:

   psql “postgres://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DATABASE>?sslmode=require”
   

3. Prepare your self-hosted TimescaleDB instance for data restoration by using timescaledb_pre_restore to stop background workers:

   SELECT timescaledb_pre_restore();
   

4. At the command prompt, restore the dumped data from the dump.bak file into your self-hosted TimescaleDB instance, using your connection details. To avoid permissions errors, include the --no-owner flag:

   pg_restore -U tsdbadmin -W \
   -h <CLOUD_HOST> -p <CLOUD_PORT> --no-owner \
   -Fc -v -d tsdb dump.bak
   

5. At the psql prompt, return your self-hosted TimescaleDB instance to normal operations by using the timescaledb_post_restore command:

   SELECT timescaledb_post_restore();
   

6. Update your table statistics by running ANALYZE on your entire dataset:

   ANALYZE;
   

===== PAGE: https://docs.tigerdata.com/self-hosted/migration/schema-then-data/ =====

Migrate schema and data separately

Migrate larger databases by migrating your schema first, then migrating the data. This method copies each table or chunk separately, which allows you to restart midway if one copy operation fails.

For smaller databases, it may be more convenient to migrate your entire database at once. For more information, see the section on choosing a migration method.

This method does not retain continuous aggregates calculated using already-deleted data. For example, if you delete raw data after a month but retain downsampled data in a continuous aggregate for a year, the continuous aggregate loses any data older than a month upon migration. If you must keep continuous aggregates calculated using deleted data, migrate your entire database at once. For more information, see the section on choosing a migration method.

The procedure to migrate your database requires these steps:

• Migrate schema pre-data
• Restore hypertables in Timescale
• Copy data from the source database
• Restore data into Timescale
• Migrate schema post-data
• Recreate continuous aggregates (optional)
• Recreate policies (optional)
• Update table statistics

Depending on your database size and network speed, steps that involve copying data can take a very long time. You can continue reading from your source database during this time, though performance could be slower. To avoid this problem, fork your database and migrate your data from the fork. If you write to the tables in your source database during the migration, the new writes might not be transferred to Timescale. To avoid this problem, see the section on migrating an active database.

Prerequisites

Before you begin, check that you have:

• Installed the Postgres pg_dump and pg_restore utilities.
• Installed a client for connecting to Postgres. These instructions use psql, but any client works.
• Created a new empty database in a self-hosted TimescaleDB instance. For more information, see the Install TimescaleDB. Provision your database with enough space for all your data.
• Checked that any other Postgres extensions you use are compatible with TimescaleDB. For more information, see the list of compatible extensions. Install your other Postgres extensions.
• Checked that you're running the same major version of Postgres on both your self-hosted TimescaleDB instance and your source database. For information about upgrading Postgres on your source database, see the upgrade instructions for self-hosted TimescaleDB and Managed Service for TimescaleDB.
• Checked that you're running the same major version of TimescaleDB on both your target and source database. For more information, see upgrading TimescaleDB.

Migrate schema pre-data

Migrate your pre-data from your source database to self-hosted TimescaleDB. This includes table and schema definitions, as well as information on sequences, owners, and settings. This doesn't include Timescale-specific schemas.

Migrating schema pre-data

1. Dump the schema pre-data from your source database into a dump_pre_data.bak file, using your source database connection details. Exclude Timescale-specific schemas. If you are prompted for a password, use your source database credentials:

   pg_dump -U <SOURCE_DB_USERNAME> -W \
   -h <SOURCE_DB_HOST> -p <SOURCE_DB_PORT> -Fc -v \
   --section=pre-data --exclude-schema="_timescaledb*" \
   -f dump_pre_data.bak <DATABASE_NAME>
   

2. Restore the dumped data from the dump_pre_data.bak file into your self-hosted TimescaleDB instance, using your self-hosted TimescaleDB connection details. To avoid permissions errors, include the --no-owner flag:

   pg_restore -U tsdbadmin -W \
   -h <HOST> -p <PORT> --no-owner -Fc \
   -v -d tsdb dump_pre_data.bak
   

Restore hypertables in your self-hosted TimescaleDB instance

After pre-data migration, your hypertables from your source database become regular Postgres tables in Timescale. Recreate your hypertables in your self-hosted TimescaleDB instance to restore them.

Restoring hypertables in your self-hosted TimescaleDB instance

1. Connect to your self-hosted TimescaleDB instance:

   psql "postgres://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DATABSE>?sslmode=require"
   

2. Restore the hypertable:

   SELECT create_hypertable(
      '',
      by_range('<COLUMN_NAME>', INTERVAL '<CHUNK_INTERVAL>')
   );
   

The by_range dimension builder is an addition to TimescaleDB 2.13.

Copy data from the source database

After restoring your hypertables, return to your source database to copy your data, table by table.

Copying data from your source database

1. Connect to your source database:

   psql "postgres://<SOURCE_DB_USERNAME>:<SOURCE_DB_PASSWORD>@<SOURCE_DB_HOST>:<SOURCE_DB_PORT>/<SOURCE_DB_NAME>?sslmode=require"
   

2. Dump the data from the first table into a .csv file:

   \COPY (SELECT * FROM ) TO .csv CSV
   

   Repeat for each table and hypertable you want to migrate.

If your tables are very large, you can migrate each table in multiple pieces. Split each table by time range, and copy each range individually. For example:

\COPY (SELECT * FROM  WHERE time > '2021-11-01' AND time < '2011-11-02') TO .csv CSV

Restore data into Timescale

When you have copied your data into .csv files, you can restore it to self-hosted TimescaleDB by copying from the .csv files. There are two methods: using regular Postgres COPY, or using the TimescaleDB timescaledb-parallel-copy function. In tests, timescaledb-parallel-copy is 16% faster. The timescaledb-parallel-copy tool is not included by default. You must install the function.

Because COPY decompresses data, any compressed data in your source database is now stored uncompressed in your .csv files. If you provisioned your self-hosted TimescaleDB storage for your compressed data, the uncompressed data may take too much storage. To avoid this problem, periodically recompress your data as you copy it in. For more information on compression, see the compression section.

Restoring data into a Tiger Cloud service with timescaledb-parallel-copy

1. At the command prompt, install timescaledb-parallel-copy:

   go get github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy
   

2. Use timescaledb-parallel-copy to import data into your Tiger Cloud service. Set <NUM_WORKERS> to twice the number of CPUs in your database. For example, if you have 4 CPUs, <NUM_WORKERS> should be 8.

   timescaledb-parallel-copy \
   --connection "host=<HOST> \
   user=tsdbadmin password=<PASSWORD> \
   port=<PORT> \
   dbname=tsdb \
   sslmode=require
   " \
   --table  \
   --file <FILE_NAME>.csv \
   --workers <NUM_WORKERS> \
   --reporting-period 30s
   

   Repeat for each table and hypertable you want to migrate.

Restoring data into a Tiger Cloud service with COPY

1. Connect to your Tiger Cloud service:

   psql "postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

2. Restore the data to your Tiger Cloud service:

   \copy  FROM '.csv' WITH (FORMAT CSV);
   

   Repeat for each table and hypertable you want to migrate.

Migrate schema post-data

When you have migrated your table and hypertable data, migrate your Postgres schema post-data. This includes information about constraints.

Migrating schema post-data

1. At the command prompt, dump the schema post-data from your source database into a dump_post_data.dump file, using your source database connection details. Exclude Timescale-specific schemas. If you are prompted for a password, use your source database credentials:

   pg_dump -U <SOURCE_DB_USERNAME> -W \
   -h <SOURCE_DB_HOST> -p <SOURCE_DB_PORT> -Fc -v \
   --section=post-data --exclude-schema="_timescaledb*" \
   -f dump_post_data.dump <DATABASE_NAME>
   

2. Restore the dumped schema post-data from the dump_post_data.dump file into your Tiger Cloud service, using your connection details. To avoid permissions errors, include the --no-owner flag:

   pg_restore -U tsdbadmin -W \
   -h <HOST> -p <PORT> --no-owner -Fc \
   -v -d tsdb dump_post_data.dump
   

Troubleshooting

If you see these errors during the migration process, you can safely ignore them. The migration still occurs successfully.

pg_restore: error: could not execute query: ERROR:  relation "<relation_name>" already exists

pg_restore: error: could not execute query: ERROR:  trigger "ts_insert_blocker" for relation "<relation_name>" already exists

Recreate continuous aggregates

Continuous aggregates aren't migrated by default when you transfer your schema and data separately. You can restore them by recreating the continuous aggregate definitions and recomputing the results on your Tiger Cloud service. The recomputed continuous aggregates only aggregate existing data in your Tiger Cloud service. They don't include deleted raw data.

Recreating continuous aggregates

1. Connect to your source database:

   psql "postgres://<SOURCE_DB_USERNAME>:<SOURCE_DB_PASSWORD>@<SOURCE_DB_HOST>:<SOURCE_DB_PORT>/<SOURCE_DB_NAME>?sslmode=require"
   

2. Get a list of your existing continuous aggregate definitions:

   SELECT view_name, view_definition FROM timescaledb_information.continuous_aggregates;
   

   This query returns the names and definitions for all your continuous aggregates. For example:

   view_name       |                                            view_definition
   ----------------+--------------------------------------------------------------------------------------------------------
   avg_fill_levels |  SELECT round(avg(fill_measurements.fill_level), 2) AS avg_fill_level,                                +
                   |     time_bucket('01:00:00'::interval, fill_measurements."time") AS bucket,                            +
                   |     fill_measurements.sensor_id                                                                       +
                   |     FROM fill_measurements                                                                            +
                   |     GROUP BY (time_bucket('01:00:00'::interval, fill_measurements."time")), fill_measurements.sensor_id;
   (1 row)
   

3. Connect to your Tiger Cloud service:

   psql "postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

4. Recreate each continuous aggregate definition:

   CREATE MATERIALIZED VIEW <VIEW_NAME>
   WITH (timescaledb.continuous) AS
   <VIEW_DEFINITION>
   

Recreate policies

By default, policies aren't migrated when you transfer your schema and data separately. Recreate them on your Tiger Cloud service.

Recreating policies

1. Connect to your source database:

   psql "postgres://<SOURCE_DB_USERNAME>:<SOURCE_DB_PASSWORD>@<SOURCE_DB_HOST>:<SOURCE_DB_PORT>/<SOURCE_DB_NAME>?sslmode=require"
   

2. Get a list of your existing policies. This query returns a list of all your policies, including continuous aggregate refresh policies, retention policies, compression policies, and reorder policies:

   SELECT application_name, schedule_interval, retry_period,
       config, hypertable_name
       FROM timescaledb_information.jobs WHERE owner = '<SOURCE_DB_USERNAME>';
   

3. Connect to your Tiger Cloud service:

   psql "postgres://tsdbadmin:<PASSWORD>@<HOST>:<PORT>/tsdb?sslmode=require"
   

4. Recreate each policy. For more information about recreating policies, see the sections on continuous-aggregate refresh policies, retention policies, Hypercore policies, and reorder policies.

Update table statistics

Update your table statistics by running ANALYZE on your entire dataset. Note that this might take some time depending on the size of your database:

ANALYZE;

Troubleshooting

If you see errors of the following form when you run ANALYZE, you can safely ignore them:

WARNING:  skipping "" --- only superuser can analyze it

The skipped tables and indexes correspond to system catalogs that can't be accessed. Skipping them does not affect statistics on your data.

===== PAGE: https://docs.tigerdata.com/self-hosted/migration/same-db/ =====

Migrate data to self-hosted TimescaleDB from the same Postgres instance

You can migrate data into a TimescaleDB hypertable from a regular Postgres table. This method assumes that you have TimescaleDB set up in the same database instance as your existing table.

Prerequisites

Before beginning, make sure you have installed and set up TimescaleDB.

You also need a table with existing data. In this example, the source table is named old_table. Replace the table name with your actual table name. The example also names the destination table new_table, but you might want to use a more descriptive name.

Migrate data

Migrate your data into TimescaleDB from within the same database.

Migrating data

1. Call CREATE TABLE to make a new table based on your existing table.

   You can create your indexes at the same time, so you don't have to recreate them manually. Or you can create the table without indexes, which makes data migration faster.

   CREATE TABLE new_table (
       LIKE old_table INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES
   ) WITH (
       tsdb.hypertable,
       tsdb.partition_column='<the name of the time column>'
   );
   

   CREATE TABLE new_table (
       LIKE old_table INCLUDING DEFAULTS INCLUDING CONSTRAINTS EXCLUDING INDEXES
   ) WITH (
       tsdb.hypertable,
       tsdb.partition_column='<the name of the time column>'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

2. Insert data from the old table to the new table.

   INSERT INTO new_table
     SELECT * FROM old_table;
   

3. If you created your new table without indexes, recreate your indexes now.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/mst/corrupt-index-duplicate/ =====

Corrupted unique index has duplicated rows

When you try to rebuild index with REINDEX it fails because of conflicting duplicated rows.

To identify conflicting duplicate rows, you need to run a query that counts the number of rows for each combination of columns included in the index definition.

For example, this route table has a unique_route_index index defining unique rows based on the combination of the source and destination columns:

CREATE TABLE route(
    source TEXT,
    destination TEXT,
    description TEXT
    );

CREATE UNIQUE INDEX unique_route_index
    ON route (source, destination);

If the unique_route_index is corrupt, you can find duplicated rows in the route table using this query:

SELECT
    source,
    destination,
    count
FROM
    (SELECT
        source,
        destination,
        COUNT(*) AS count
    FROM route
    GROUP BY
        source,
        destination) AS foo
WHERE count > 1;

The query groups the data by the same source and destination fields defined in the index, and filters any entries with more than one occurrence.

Resolve the problematic entries in the rows by manually deleting or merging the entries until no duplicates exist. After all duplicate entries are removed, you can use the REINDEX command to rebuild the index.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/mst/changing-owner-permission-denied/ =====

Permission denied when changing ownership of tables and hypertables

You might see this error when using the ALTER TABLE command to change the ownership of tables or hypertables.

This use of ALTER TABLE is blocked because the tsdbadmin user is not a superuser.

To change table ownership, use the REASSIGN command instead:

REASSIGN OWNED BY <current_role> TO <desired_role>

===== PAGE: https://docs.tigerdata.com/_troubleshooting/mst/transaction-wraparound/ =====

Postgres transaction ID wraparound

The transaction control mechanism in Postgres assigns a transaction ID to every row that is modified in the database; these IDs control the visibility of that row to other concurrent transactions. The transaction ID is a 32-bit number where two billion IDs are always in the visible past and the remaining IDs are reserved for future transactions and are not visible to the running transaction. To avoid a transaction wraparound of old rows, Postgres requires occasional cleanup and freezing of old rows. This ensures that existing rows are visible when more transactions are created. You can manually freeze the old rows by executing VACUUM FREEZE. It can also be done automatically using the autovacuum daemon when a configured number of transactions has been created since the last freeze point.

In Managed Service for TimescaleDB, the transaction limit is set according to the size of the database, up to 1.5 billion transactions. This ensures 500 million transaction IDs are available before a forced freeze and avoids churning stable data in existing tables. To check your transaction freeze limits, you can execute show autovacuum_freeze_max_age in your Postgres instance. When the limit is reached, autovacuum starts freezing the old rows. Some applications do not automatically adjust the configuration when the Postgres settings change, which can result in unnecessary warnings. For example, PGHero's default settings alert when 500 million transactions have been created instead of alerting after 1.5 billion transactions. To avoid this, change the value of the transaction_id_danger setting from 1,500,000,000 to 500,000,000, to receive warnings when the transaction limit reaches 1.5 billion.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/mst/low-disk-memory-cpu/ =====

Service is running low on disk, memory, or CPU

When your database reaches 90% of your allocated disk, memory, or CPU resources, an automated message with the text above is sent to your email address.

You can resolve this by logging in to your Managed Service for TimescaleDB account and increasing your available resources. From the Managed Service for TimescaleDB Dashboard, select the service that you want to increase resources for. In the Overview tab, locate the Service Plan section, and click Upgrade Plan. Select the plan that suits your requirements, and click Upgrade to enable the additional resources.

If you run out of resources regularly, you might need to consider using your resources more efficiently. Consider enabling Hypercore, using continuous aggregates, or configuring data retention to reduce the amount of resources your database uses.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/mst/forgotten-password/ =====

Reset password

It happens to us all, you want to login to MST Console, and the password is somewhere next to your keys, wherever they are.

To reset your password:

1. Open MST Portal.
2. Click Forgot password.
3. Enter your email address, then click Reset password.

A secure reset password link is sent to the email associated with this account. Click the link and update your password.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/mst/resolving-dns/ =====

Problem resolving DNS

services require a DNS record. When you launch a new service the DNS record is created, and it can take some time for the new name to propagate to DNS servers around the world.

If you move an existing service to a new Cloud provider or region, the service is rebuilt in the new region in the background. When the service has been rebuilt in the new region, the DNS records are updated. This could cause a short interruption to your service while the DNS changes are propagated.

If you are unable to resolve DNS, wait a few minutes and try again.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/upgrade-no-update-path/ =====

TimescaleDB upgrade fails with no update path

In some cases, when you use the ALTER EXTENSION timescaledb UPDATE command to upgrade, it might fail with the above error.

This occurs if the list of available extensions does not include the version you are trying to upgrade to, and it can occur if the package was not installed correctly in the first place. To correct the problem, install the upgrade package, restart Postgres, verify the version, and then attempt the upgrade again.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/pg_dump-version-mismatch/ =====

Versions are mismatched when dumping and restoring a database

The Postgres pg_dump command does not allow you to specify which version of the extension to use when backing up. This can create problems if you have a more recent version installed. For example, if you create the backup using an older version of TimescaleDB, and when you restore it uses the current version, without giving you an opportunity to upgrade first.

You can work around this problem when you are restoring from backup by making sure the new Postgres instance has the same extension version as the original database before you perform the restore. After the data is restored, you can upgrade the version of TimescaleDB.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/upgrade-fails-already-loaded/ =====

Upgrading fails with an error saying "old version has already been loaded"

When you use the ALTER EXTENSION timescaledb UPDATE command to upgrade, this error might appear.

This occurs if you don't run ALTER EXTENSION timescaledb UPDATE command as the first command after starting a new session using psql or if you use tab completion when running the command. Tab completion triggers metadata queries in the background which prevents the alter extension from being the first command.

To correct the problem, execute the ALTER EXTENSION command like this:

psql -X -c 'ALTER EXTENSION timescaledb UPDATE;'

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/migration-errors-perms/ =====

Errors encountered during a pg_dump migration

The pg_restore function tries to apply the TimescaleDB extension when it copies your schema. This can cause a permissions error. If you already have the TimescaleDB extension installed, you can safely ignore this.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/pg_restore-errors/ =====

Errors occur after restoring from file dump

You might see the errors above when running pg_restore. When loading from a logical dump make sure that you set timescaledb.restoring to true before loading the dump.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/install-timescaledb-could-not-access-file/ =====

Can't access file "timescaledb" after installation

If your Postgres logs have this error preventing it from starting up, you should double check that the TimescaleDB files have been installed to the correct location. Our installation methods use pg_config to get Postgres's location. However if you have multiple versions of Postgres installed on the same machine, the location pg_config points to may not be for the version you expect. To check which version TimescaleDB used:

$ pg_config --version
PostgreSQL 12.3

If that is the correct version, double check that the installation path is the one you'd expect. For example, for Postgres 11.0 installed via Homebrew on macOS it should be /usr/local/Cellar/postgresql/11.0/bin:

$ pg_config --bindir
/usr/local/Cellar/postgresql/11.0/bin

If either of those steps is not the version you are expecting, you need to either (a) uninstall the incorrect version of Postgres if you can or (b) update your PATH environmental variable to have the correct path of pg_config listed first, that is, by prepending the full path:

export PATH = /usr/local/Cellar/postgresql/11.0/bin:$PATH

Then, reinstall TimescaleDB and it should find the correct installation path.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/update-error-third-party-tool/ =====

Error updating TimescaleDB when using a third-party Postgres admin tool

The update command ALTER EXTENSION timescaledb UPDATE must be the first command executed upon connection to a database. Some admin tools execute commands before this, which can disrupt the process. Try manually updating the database with psql. For instructions, see the updating guide.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/windows-install-library-not-loaded/ =====

Error loading the timescaledb extension

If you see a message saying that Postgres cannot load the TimescaleDB library timescaledb-<version>.dll, start a new psql session to your self-hosted instance and create the timescaledb extension as the first command:

psql -X -d "postgres://<user>:<password>@<source_host>:<source_port>/<db_name>" -c "CREATE EXTENSION IF NOT EXISTS timescaledb;"

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/pg_dump-errors/ =====

Errors occur when running pg_dump

You might see the errors above when running pg_dump. You can safely ignore these. Your hypertable data is still accurately copied.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/background-worker-failed-start/ =====

Failed to start a background worker

You might see this error message in the logs if background workers aren't properly configured.

To fix this error, make sure that max_worker_processes, max_parallel_workers, and timescaledb.max_background_workers are properly set. timescaledb.max_background_workers should equal the number of databases plus the number of concurrent background workers. max_worker_processes should equal the sum of timescaledb.max_background_workers and max_parallel_workers.

For more information, see the worker configuration docs.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/toolkit-cannot-create-upgrade-extension/ =====

Install or upgrade of TimescaleDB Toolkit fails

In some cases, when you create the TimescaleDB Toolkit extension, or upgrade it with the ALTER EXTENSION timescaledb_toolkit UPDATE command, it might fail with the above error.

This occurs if the list of available extensions does not include the version you are trying to upgrade to, and it can occur if the package was not installed correctly in the first place. To correct the problem, install the upgrade package, restart Postgres, verify the version, and then attempt the update again.

Troubleshooting TimescaleDB Toolkit setup

1. If you're installing Toolkit from a package, check your package manager's local repository list. Make sure the TimescaleDB repository is available and contains Toolkit. For instructions on adding the TimescaleDB repository, see the installation guides:

   • Linux installation guide

2. Update your local repository list with apt update or yum update.

3. Restart your Postgres service.

4. Check that the right version of Toolkit is among your available extensions:

   SELECT * FROM pg_available_extensions
     WHERE name = 'timescaledb_toolkit';
   

   The result should look like this:

   -[ RECORD 1 ]-----+--------------------------------------------------------------------------------------
   name              | timescaledb_toolkit
   default_version   | 1.6.0
   installed_version | 1.6.0
   comment           | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities
   

5. Retry CREATE EXTENSION or ALTER EXTENSION.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/pg_dump-permission-denied/ =====

Permission denied for table job_errors when running pg_dump

When the pg_dump tool tries to acquire a lock on the job_errors table, if the user doesn't have the required SELECT permission, it results in this error.

To resolve this issue, use a superuser account to grant the necessary permissions to the user requiring the pg_dump tool. Use this command to grant permissions to <TEST_USER>:

GRANT SELECT ON TABLE _timescaledb_internal.job_errors TO <TEST_USER>;

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/update-timescaledb-could-not-access-file/ =====

Can't access file "timescaledb-VERSION" after update

If the error occurs immediately after updating your version of TimescaleDB and the file mentioned is from the previous version, it is probably due to an incomplete update process. Within the greater Postgres server instance, each database that has TimescaleDB installed needs to be updated with the SQL command ALTER EXTENSION timescaledb UPDATE; while connected to that database. Otherwise, the database looks for the previous version of the TimescaleDB files.

See our update docs for more info.

===== PAGE: https://docs.tigerdata.com/_troubleshooting/self-hosted/migration-errors/ =====

Errors encountered during a pg_dump migration

If you see these errors during the migration process, you can safely ignore them. The migration still occurs successfully.

===== PAGE: https://docs.tigerdata.com/tutorials/financial-tick-data/financial-tick-dataset/ =====

Analyze financial tick data - Set up the dataset

This tutorial uses a dataset that contains second-by-second trade data for the most-traded crypto-assets. You optimize this time-series data in a a hypertable called assets_real_time. You also create a separate table of asset symbols in a regular Postgres table named assets.

The dataset is updated on a nightly basis and contains data from the last four weeks, typically around 8 million rows of data. Trades are recorded in real-time from 180+ cryptocurrency exchanges.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Optimize time-series data in a hypertable

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Create a hypertable to store the real-time cryptocurrency data

   Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data:

   CREATE TABLE crypto_ticks (
       "time" TIMESTAMPTZ,
       symbol TEXT,
       price DOUBLE PRECISION,
       day_volume NUMERIC
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time',
      tsdb.segmentby='symbol',
      tsdb.orderby='time DESC'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

Create a standard Postgres table for relational data

When you have relational data that enhances your time-series data, store that data in standard Postgres relational tables.

1. Add a table to store the asset symbol and name in a relational table

   CREATE TABLE crypto_assets (
       symbol TEXT UNIQUE,
       "name" TEXT
   );
   

You now have two tables within your Tiger Cloud service. A hypertable named crypto_ticks, and a normal Postgres table named crypto_assets.

Load financial data

This tutorial uses real-time cryptocurrency data, also known as tick data, from Twelve Data. To ingest data into the tables that you created, you need to download the dataset, then upload the data to your Tiger Cloud service.

1. Unzip crypto_sample.zip to a <local folder>.

   This test dataset contains second-by-second trade data for the most-traded crypto-assets and a regular table of asset symbols and company names.

   To import up to 100GB of data directly from your current Postgres-based database, migrate with downtime using native Postgres tooling. To seamlessly import 100GB-10TB+ of data, use the live migration tooling supplied by Tiger Data. To add data from non-Postgres data sources, see Import and ingest data.

2. In Terminal, navigate to <local folder> and connect to your service.

   psql -d "postgres://<username>:<password>@<host>:<port>/<database-name>"
   

   The connection information for a service is available in the file you downloaded when you created it.

3. At the psql prompt, use the COPY command to transfer data into your Tiger Cloud service. If the .csv files aren't in your current directory, specify the file paths in these commands:

   \COPY crypto_ticks FROM 'tutorial_sample_tick.csv' CSV HEADER;
   

   \COPY crypto_assets FROM 'tutorial_sample_assets.csv' CSV HEADER;
   

   Because there are millions of rows of data, the COPY process could take a few minutes depending on your internet connection and local client resources.

Connect Grafana to Tiger Cloud

To visualize the results of your queries, enable Grafana to read the data in your service:

1. Log in to Grafana

   In your browser, log in to either:

   • Self-hosted Grafana: at http://localhost:3000/. The default credentials are admin, admin.
   • Grafana Cloud: use the URL and credentials you set when you created your account.

2. Add your service as a data source

   1. Open Connections > Data sources, then click Add new data source.

   2. Select PostgreSQL from the list.

   3. Configure the connection:

      • Host URL, Database name, Username, and Password

        Configure using your connection details. Host URL is in the format <host>:<port>.

      • TLS/SSL Mode: select require.

      • PostgreSQL options: enable TimescaleDB.

      • Leave the default setting for all other fields.

   4. Click Save & test.

   Grafana checks that your details are set correctly.

===== PAGE: https://docs.tigerdata.com/tutorials/financial-tick-data/financial-tick-compress/ =====

Compress your data using hypercore

Over time you end up with a lot of data. Since this data is mostly immutable, you can compress it to save space and avoid incurring additional cost.

TimescaleDB is built for handling event-oriented data such as time-series and fast analytical queries, it comes with support of hypercore featuring the columnstore.

Hypercore enables you to store the data in a vastly more efficient format allowing up to 90x compression ratio compared to a normal Postgres table. However, this is highly dependent on the data and configuration.

Hypercore is implemented natively in Postgres and does not require special storage formats. When you convert your data from the rowstore to the columnstore, TimescaleDB uses Postgres features to transform the data into columnar format. The use of a columnar format allows a better compression ratio since similar data is stored adjacently. For more details on the columnar format, see hypercore.

A beneficial side effect of compressing data is that certain queries are significantly faster, since less data has to be read into memory.

Optimize your data in the columnstore

To compress the data in the crypto_ticks table, do the following:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed. You can also connect to your service using psql.

2. Convert data to the columnstore:

   You can do this either automatically or manually:

   • Automatically convert chunks in the hypertable to the columnstore at a specific time interval:

     CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '1d');
     

   • Manually convert all chunks in the hypertable to the columnstore:

     CALL convert_to_columnstore(c) from show_chunks('crypto_ticks') c;
     

3. Now that you have converted the chunks in your hypertable to the columnstore, compare the size of the dataset before and after compression:

   SELECT
       pg_size_pretty(before_compression_total_bytes) as before,
       pg_size_pretty(after_compression_total_bytes) as after
    FROM hypertable_columnstore_stats('crypto_ticks');
   

   This shows a significant improvement in data usage:

   before | after
   --------+-------
   694 MB | 75 MB
   (1 row)
   

Take advantage of query speedups

Previously, data in the columnstore was segmented by the block_id column value. This means fetching data by filtering or grouping on that column is more efficient. Ordering is set to time descending. This means that when you run queries which try to order data in the same way, you see performance benefits.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed.

2. Run the following query:

   SELECT
       time_bucket('1 day', time) AS bucket,
       symbol,
       FIRST(price, time) AS "open",
       MAX(price) AS high,
       MIN(price) AS low,
       LAST(price, time) AS "close",
       LAST(day_volume, time) AS day_volume
   FROM crypto_ticks
   GROUP BY bucket, symbol;
   

   Performance speedup is of two orders of magnitude, around 15 ms when compressed in the columnstore and 1 second when decompressed in the rowstore.

===== PAGE: https://docs.tigerdata.com/tutorials/financial-tick-data/financial-tick-query/ =====

Analyze financial tick data - Query the data

Turning raw, real-time tick data into aggregated candlestick views is a common task for users who work with financial data. TimescaleDB includes hyperfunctions that you can use to store and query your financial data more easily. Hyperfunctions are SQL functions within TimescaleDB that make it easier to manipulate and analyze time-series data in Postgres with fewer lines of code.

There are three hyperfunctions that are essential for calculating candlestick values: time_bucket(), FIRST(), and LAST(). The time_bucket() hyperfunction helps you aggregate records into buckets of arbitrary time intervals based on the timestamp value. FIRST() and LAST() help you calculate the opening and closing prices. To calculate highest and lowest prices, you can use the standard Postgres aggregate functions MIN and MAX.

In TimescaleDB, the most efficient way to create candlestick views is to use continuous aggregates. In this tutorial, you create a continuous aggregate for a candlestick time bucket, and then query the aggregate with different refresh policies. Finally, you can use Grafana to visualize your data as a candlestick chart.

Create a continuous aggregate

To look at OHLCV values, the most effective way is to create a continuous aggregate. In this tutorial, you create a continuous aggregate to aggregate data for each day. You then set the aggregate to refresh every day, and to aggregate the last two days' worth of data.

Creating a continuous aggregate

1. Connect to the Tiger Cloud service that contains the Twelve Data cryptocurrency dataset.

2. At the psql prompt, create the continuous aggregate to aggregate data every minute:

   CREATE MATERIALIZED VIEW one_day_candle
   WITH (timescaledb.continuous) AS
       SELECT
           time_bucket('1 day', time) AS bucket,
           symbol,
           FIRST(price, time) AS "open",
           MAX(price) AS high,
           MIN(price) AS low,
           LAST(price, time) AS "close",
           LAST(day_volume, time) AS day_volume
       FROM crypto_ticks
       GROUP BY bucket, symbol;
   

   When you create the continuous aggregate, it refreshes by default.

3. Set a refresh policy to update the continuous aggregate every day, if there is new data available in the hypertable for the last two days:

   SELECT add_continuous_aggregate_policy('one_day_candle',
       start_offset => INTERVAL '3 days',
       end_offset => INTERVAL '1 day',
       schedule_interval => INTERVAL '1 day');
   

Query the continuous aggregate

When you have your continuous aggregate set up, you can query it to get the OHLCV values.

Querying the continuous aggregate

1. Connect to the Tiger Cloud service that contains the Twelve Data cryptocurrency dataset.

2. At the psql prompt, use this query to select all Bitcoin OHLCV data for the past 14 days, by time bucket:

   SELECT * FROM one_day_candle
   WHERE symbol = 'BTC/USD' AND bucket >= NOW() - INTERVAL '14 days'
   ORDER BY bucket;
   

   The result of the query looks like this:

            bucket         | symbol  |  open   |  high   |   low   |  close  | day_volume
   ------------------------+---------+---------+---------+---------+---------+------------
    2022-11-24 00:00:00+00 | BTC/USD |   16587 | 16781.2 | 16463.4 | 16597.4 |      21803
    2022-11-25 00:00:00+00 | BTC/USD | 16597.4 | 16610.1 | 16344.4 | 16503.1 |      20788
    2022-11-26 00:00:00+00 | BTC/USD | 16507.9 | 16685.5 | 16384.5 | 16450.6 |      12300
   

Graph OHLCV data

When you have extracted the raw OHLCV data, you can use it to graph the result in a candlestick chart, using Grafana. To do this, you need to have Grafana set up to connect to your self-hosted TimescaleDB instance.

Graphing OHLCV data

1. Ensure you have Grafana installed, and you are using the TimescaleDB database that contains the Twelve Data dataset set up as a data source.

2. In Grafana, from the Dashboards menu, click New Dashboard. In the New Dashboard page, click Add a new panel.

3. In the Visualizations menu in the top right corner, select Candlestick from the list. Ensure you have set the Twelve Data dataset as your data source.

4. Click Edit SQL and paste in the query you used to get the OHLCV values.

5. In the Format as section, select Table.

6. Adjust elements of the table as required, and click Apply to save your graph to the dashboard.

   

===== PAGE: https://docs.tigerdata.com/tutorials/blockchain-analyze/blockchain-dataset/ =====

Analyze the Bitcoin blockchain - set up dataset

Ingest data into a Tiger Cloud service

This tutorial uses a dataset that contains Bitcoin blockchain data for the past five days, in a hypertable named transactions.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Optimize time-series data using hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed. You can also connect to your service using psql.

2. Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data:

   CREATE TABLE transactions (
      time TIMESTAMPTZ NOT NULL,
      block_id INT,
      hash TEXT,
      size INT,
      weight INT,
      is_coinbase BOOLEAN,
      output_total BIGINT,
      output_total_usd DOUBLE PRECISION,
      fee BIGINT,
      fee_usd DOUBLE PRECISION,
      details JSONB
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time',
      tsdb.segmentby='block_id',
      tsdb.orderby='time DESC'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

3. Create an index on the hash column to make queries for individual transactions faster:

   CREATE INDEX hash_idx ON public.transactions USING HASH (hash);
   

4. Create an index on the block_id column to make block-level queries faster:

When you create a hypertable, it is partitioned on the time column. TimescaleDB automatically creates an index on the time column. However, you'll often filter your time-series data on other columns as well. You use indexes to improve query performance.

```sql
CREATE INDEX block_idx ON public.transactions (block_id);
```

1. Create a unique index on the time and hash columns to make sure you don't accidentally insert duplicate records:

   CREATE UNIQUE INDEX time_hash_idx ON public.transactions (time, hash);
   

Load financial data

The dataset contains around 1.5 million Bitcoin transactions, the trades for five days. It includes information about each transaction, along with the value in satoshi. It also states if a trade is a coinbase transaction, and the reward a coin miner receives for mining the coin.

To ingest data into the tables that you created, you need to download the dataset and copy the data to your database.

1. Download the bitcoin_sample.zip file. The file contains a .csv file that contains Bitcoin transactions for the past five days. Download:

   bitcoin_sample.zip

2. In a new terminal window, run this command to unzip the .csv files:

   unzip bitcoin_sample.zip
   

3. In Terminal, navigate to the folder where you unzipped the Bitcoin transactions, then connect to your service using psql.

4. At the psql prompt, use the COPY command to transfer data into your Tiger Cloud service. If the .csv files aren't in your current directory, specify the file paths in these commands:

   \COPY transactions FROM 'tutorial_bitcoin_sample.csv' CSV HEADER;
   

   Because there is over a million rows of data, the COPY process could take a few minutes depending on your internet connection and local client resources.

Connect Grafana to Tiger Cloud

To visualize the results of your queries, enable Grafana to read the data in your service:

1. Log in to Grafana

   In your browser, log in to either:

   • Self-hosted Grafana: at http://localhost:3000/. The default credentials are admin, admin.
   • Grafana Cloud: use the URL and credentials you set when you created your account.

2. Add your service as a data source

   1. Open Connections > Data sources, then click Add new data source.

   2. Select PostgreSQL from the list.

   3. Configure the connection:

      • Host URL, Database name, Username, and Password

        Configure using your connection details. Host URL is in the format <host>:<port>.

      • TLS/SSL Mode: select require.

      • PostgreSQL options: enable TimescaleDB.

      • Leave the default setting for all other fields.

   4. Click Save & test.

   Grafana checks that your details are set correctly.

===== PAGE: https://docs.tigerdata.com/tutorials/blockchain-analyze/analyze-blockchain-query/ =====

Analyze the Bitcoin blockchain - query the data

When you have your dataset loaded, you can create some continuous aggregates, and start constructing queries to discover what your data tells you. This tutorial uses TimescaleDB hyperfunctions to construct queries that are not possible in standard Postgres.

In this section, you learn how to write queries that answer these questions:

• Is there any connection between the number of transactions and the transaction fees?
• Does the transaction volume affect the BTC-USD rate?
• Do more transactions in a block mean the block is more expensive to mine?
• What percentage of the average miner's revenue comes from fees compared to block rewards?
• How does block weight affect miner fees?
• What's the average miner revenue per block?

Create continuous aggregates

You can use continuous aggregates to simplify and speed up your queries. For this tutorial, you need three continuous aggregates, focusing on three aspects of the dataset: Bitcoin transactions, blocks, and coinbase transactions. In each continuous aggregate definition, the time_bucket() function controls how large the time buckets are. The examples all use 1-hour time buckets.

Continuous aggregate: transactions

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, create a continuous aggregate called one_hour_transactions. This view holds aggregated data about each hour of transactions:

   CREATE MATERIALIZED VIEW one_hour_transactions
   WITH (timescaledb.continuous) AS
   SELECT time_bucket('1 hour', time) AS bucket,
      count(*) AS tx_count,
      sum(fee) AS total_fee_sat,
      sum(fee_usd) AS total_fee_usd,
      stats_agg(fee) AS stats_fee_sat,
      avg(size) AS avg_tx_size,
      avg(weight) AS avg_tx_weight,
      count(
            CASE
               WHEN (fee > output_total) THEN hash
               ELSE NULL
            END) AS high_fee_count
     FROM transactions
     WHERE (is_coinbase IS NOT TRUE)
   GROUP BY bucket;
   

3. Add a refresh policy to keep the continuous aggregate up-to-date:

   SELECT add_continuous_aggregate_policy('one_hour_transactions',
      start_offset => INTERVAL '3 hours',
      end_offset => INTERVAL '1 hour',
      schedule_interval => INTERVAL '1 hour');
   

4. Create a continuous aggregate called one_hour_blocks. This view holds aggregated data about all the blocks that were mined each hour:

   CREATE MATERIALIZED VIEW one_hour_blocks
   WITH (timescaledb.continuous) AS
   SELECT time_bucket('1 hour', time) AS bucket,
      block_id,
      count(*) AS tx_count,
      sum(fee) AS block_fee_sat,
      sum(fee_usd) AS block_fee_usd,
      stats_agg(fee) AS stats_tx_fee_sat,
      avg(size) AS avg_tx_size,
      avg(weight) AS avg_tx_weight,
      sum(size) AS block_size,
      sum(weight) AS block_weight,
      max(size) AS max_tx_size,
      max(weight) AS max_tx_weight,
      min(size) AS min_tx_size,
      min(weight) AS min_tx_weight
   FROM transactions
   WHERE is_coinbase IS NOT TRUE
   GROUP BY bucket, block_id;
   

5. Add a refresh policy to keep the continuous aggregate up-to-date:

   SELECT add_continuous_aggregate_policy('one_hour_blocks',
      start_offset => INTERVAL '3 hours',
      end_offset => INTERVAL '1 hour',
      schedule_interval => INTERVAL '1 hour');
   

6. Create a continuous aggregate called one_hour_coinbase. This view holds aggregated data about all the transactions that miners received as rewards each hour:

   CREATE MATERIALIZED VIEW one_hour_coinbase
   WITH (timescaledb.continuous) AS
   SELECT time_bucket('1 hour', time) AS bucket,
      count(*) AS tx_count,
      stats_agg(output_total, output_total_usd) AS stats_miner_revenue,
      min(output_total) AS min_miner_revenue,
      max(output_total) AS max_miner_revenue
   FROM transactions
   WHERE is_coinbase IS TRUE
   GROUP BY bucket;
   

7. Add a refresh policy to keep the continuous aggregate up-to-date:

   SELECT add_continuous_aggregate_policy('one_hour_coinbase',
      start_offset => INTERVAL '3 hours',
      end_offset => INTERVAL '1 hour',
      schedule_interval => INTERVAL '1 hour');
   

Is there any connection between the number of transactions and the transaction fees?

Transaction fees are a major concern for blockchain users. If a blockchain is too expensive, you might not want to use it. This query shows you whether there's any correlation between the number of Bitcoin transactions and the fees. The time range for this analysis is the last 2 days.

If you choose to visualize the query in Grafana, you can see the average transaction volume and the average fee per transaction, over time. These trends might help you decide whether to submit a transaction now or wait a few days for fees to decrease.

Finding a connection between the number of transactions and the transaction fees

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to average transaction volume and the fees from the one_hour_transactions continuous aggregate:

   SELECT
    bucket AS "time",
    tx_count as "tx volume",
    average(stats_fee_sat) as fees
   FROM one_hour_transactions
   WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-2 days')
   ORDER BY 1;
   

3. The data you get back looks a bit like this:

           time          | tx volume |        fees
   ------------------------+-----------+--------------------
   2023-11-20 01:00:00+00 |      2602 | 105963.45810914681
   2023-11-20 02:00:00+00 |     33037 | 26686.814117504615
   2023-11-20 03:00:00+00 |     42077 | 22875.286546094067
   2023-11-20 04:00:00+00 |     46021 | 20280.843180287262
   2023-11-20 05:00:00+00 |     20828 | 24694.472969080085
   ...
   

4. To visualize this in Grafana, create a new panel, select the Bitcoin dataset as your data source, and type the query from the previous step. In the Format as section, select Time series.

   

Does the transaction volume affect the BTC-USD rate?

In cryptocurrency trading, there's a lot of speculation. You can adopt a data-based trading strategy by looking at correlations between blockchain metrics, such as transaction volume and the current exchange rate between Bitcoin and US Dollars.

If you choose to visualize the query in Grafana, you can see the average transaction volume, along with the BTC to US Dollar conversion rate.

Finding the transaction volume and the BTC-USD rate

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to return the trading volume and the BTC to US Dollar exchange rate:

   SELECT
    bucket AS "time",
    tx_count as "tx volume",
    total_fee_usd / (total_fee_sat*0.00000001) AS "btc-usd rate"
   FROM one_hour_transactions
   WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-2 days')
   ORDER BY 1;
   

3. The data you get back looks a bit like this:

             time          | tx volume |    btc-usd rate
   ------------------------+-----------+--------------------
    2023-06-13 08:00:00+00 |     20063 | 25975.888587931426
    2023-06-13 09:00:00+00 |     16984 |  25976.00446352126
    2023-06-13 10:00:00+00 |     15856 | 25975.988587014584
    2023-06-13 11:00:00+00 |     24967 |  25975.89166787936
    2023-06-13 12:00:00+00 |      8575 | 25976.004209699528
    ...
   

4. To visualize this in Grafana, create a new panel, select the Bitcoin dataset as your data source, and type the query from the previous step. In the Format as section, select Time series.

5. To make this visualization more useful, add an override to put the fees on a different Y-axis. In the options panel, add an override for the btc-usd rate field for Axis > Placement and choose Right.

   

Do more transactions in a block mean the block is more expensive to mine?

The number of transactions in a block can influence the overall block mining fee. For this analysis, a larger time frame is required, so increase the analyzed time range to 5 days.

If you choose to visualize the query in Grafana, you can see that the more transactions in a block, the higher the mining fee becomes.

Finding if more transactions in a block mean the block is more expensive to mine

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to return the number of transactions in a block, compared to the mining fee:

   SELECT
    bucket as "time",
    avg(tx_count) AS transactions,
    avg(block_fee_sat)*0.00000001 AS "mining fee"
   FROM one_hour_blocks
   WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days')
   GROUP BY bucket
   ORDER BY 1;
   

3. The data you get back looks a bit like this:

             time          |     transactions      |       mining fee
   ------------------------+-----------------------+------------------------
    2023-06-10 08:00:00+00 | 2322.2500000000000000 | 0.29221418750000000000
    2023-06-10 09:00:00+00 | 3305.0000000000000000 | 0.50512649666666666667
    2023-06-10 10:00:00+00 | 3011.7500000000000000 | 0.44783255750000000000
    2023-06-10 11:00:00+00 | 2874.7500000000000000 | 0.39303009500000000000
    2023-06-10 12:00:00+00 | 2339.5714285714285714 | 0.25590717142857142857
   ...
   

4. To visualize this in Grafana, create a new panel, select the Bitcoin dataset as your data source, and type the query from the previous step. In the Format as section, select Time series.

5. To make this visualization more useful, add an override to put the fees on a different Y-axis. In the options panel, add an override for the mining fee field for Axis > Placement and choose Right.

   

You can extend this analysis to find if there is the same correlation between block weight and mining fee. More transactions should increase the block weight, and boost the miner fee as well.

If you choose to visualize the query in Grafana, you can see the same kind of high correlation between block weight and mining fee. The relationship weakens when the block weight gets close to its maximum value, which is 4 million weight units, in which case it's impossible for a block to include more transactions.

Finding if higher block weight means the block is more expensive to mine

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to return the block weight, compared to the mining fee:

   SELECT
    bucket as "time",
    avg(block_weight) as "block weight",
    avg(block_fee_sat*0.00000001) as "mining fee"
   FROM one_hour_blocks
   WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days')
   group by bucket
   ORDER BY 1;
   

3. The data you get back looks a bit like this:

             time          |     block weight     |       mining fee
   ------------------------+----------------------+------------------------
    2023-06-10 08:00:00+00 | 3992809.250000000000 | 0.29221418750000000000
    2023-06-10 09:00:00+00 | 3991766.333333333333 | 0.50512649666666666667
    2023-06-10 10:00:00+00 | 3992918.250000000000 | 0.44783255750000000000
    2023-06-10 11:00:00+00 | 3991873.000000000000 | 0.39303009500000000000
    2023-06-10 12:00:00+00 | 3992934.000000000000 | 0.25590717142857142857
   ...
   

4. To visualize this in Grafana, create a new panel, select the Bitcoin dataset as your data source, and type the query from the previous step. In the Format as section, select Time series.

5. To make this visualization more useful, add an override to put the fees on a different Y-axis. In the options panel, add an override for the mining fee field for Axis > Placement and choose Right.

   

What percentage of the average miner's revenue comes from fees compared to block rewards?

In the previous queries, you saw that mining fees are higher when block weights and transaction volumes are higher. This query analyzes the data from a different perspective. Miner revenue is not only made up of miner fees, it also includes block rewards for mining a new block. This reward is currently 6.25 BTC, and it gets halved every four years. This query looks at how much of a miner's revenue comes from fees, compares to block rewards.

If you choose to visualize the query in Grafana, you can see that most miner revenue actually comes from block rewards. Fees never account for more than a few percentage points of overall revenue.

Finding what percentage of the average miner's revenue comes from fees compared to block rewards

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to return coinbase transactions, along with the block fees and rewards:

   WITH coinbase AS (
      SELECT block_id, output_total AS coinbase_tx FROM transactions
      WHERE is_coinbase IS TRUE and time > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days')
   )
   SELECT
      bucket as "time",
      avg(block_fee_sat)*0.00000001 AS "fees",
      FIRST((c.coinbase_tx - block_fee_sat), bucket)*0.00000001 AS "reward"
   FROM one_hour_blocks b
   INNER JOIN coinbase c ON c.block_id = b.block_id
   GROUP BY bucket
   ORDER BY 1;
   

3. The data you get back looks a bit like this:

             time          |          fees          |   reward
   ------------------------+------------------------+------------
    2023-06-10 08:00:00+00 | 0.28247062857142857143 | 6.25000000
    2023-06-10 09:00:00+00 | 0.50512649666666666667 | 6.25000000
    2023-06-10 10:00:00+00 | 0.44783255750000000000 | 6.25000000
    2023-06-10 11:00:00+00 | 0.39303009500000000000 | 6.25000000
    2023-06-10 12:00:00+00 | 0.25590717142857142857 | 6.25000000
   ...
   

4. To visualize this in Grafana, create a new panel, select the Bitcoin dataset as your data source, and type the query from the previous step. In the Format as section, select Time series.

5. To make this visualization more useful, stack the series to 100%. In the options panel, in the Graph styles section, for Stack series select 100%.

   

How does block weight affect miner fees?

You've already found that more transactions in a block mean it's more expensive to mine. In this query, you ask if the same is true for block weights? The more transactions a block has, the larger its weight, so the block weight and mining fee should be tightly correlated. This query uses a 12-hour moving average to calculate the block weight and block mining fee over time.

If you choose to visualize the query in Grafana, you can see that the block weight and block mining fee are tightly connected. In practice, you can also see the four million weight units size limit. This means that there's still room to grow for individual blocks, and they could include even more transactions.

Finding how block weight affects miner fees

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to return block weight, along with the block fees and rewards:

   WITH stats AS (
      SELECT
          bucket,
          stats_agg(block_weight, block_fee_sat) AS block_stats
      FROM one_hour_blocks
      WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days')
      GROUP BY bucket
   )
   SELECT
      bucket as "time",
      average_y(rolling(block_stats) OVER (ORDER BY bucket RANGE '12 hours' PRECEDING)) AS "block weight",
      average_x(rolling(block_stats) OVER (ORDER BY bucket RANGE '12 hours' PRECEDING))*0.00000001 AS "mining fee"
   FROM stats
   ORDER BY 1;
   

3. The data you get back looks a bit like this:

             time          |    block weight    |     mining fee
   ------------------------+--------------------+---------------------
    2023-06-10 09:00:00+00 | 3991766.3333333335 |  0.5051264966666666
    2023-06-10 10:00:00+00 | 3992424.5714285714 | 0.47238710285714286
    2023-06-10 11:00:00+00 |            3992224 | 0.44353000909090906
    2023-06-10 12:00:00+00 |  3992500.111111111 | 0.37056557222222225
    2023-06-10 13:00:00+00 |         3992446.65 | 0.39728022799999996
   ...
   

4. To visualize this in Grafana, create a new panel, select the Bitcoin dataset as your data source, and type the query from the previous step. In the Format as section, select Time series.

5. To make this visualization more useful, add an override to put the fees on a different Y-axis. In the options panel, add an override for the mining fee field for Axis > Placement and choose Right.

   

What's the average miner revenue per block?

In this final query, you analyze how much revenue miners actually generate by mining a new block on the blockchain, including fees and block rewards. To make the analysis more interesting, add the Bitcoin to US Dollar exchange rate, and increase the time range.

Finding the average miner revenue per block

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to return the average miner revenue per block, with a 12-hour moving average:

   SELECT
      bucket as "time",
      average_y(rolling(stats_miner_revenue) OVER (ORDER BY bucket RANGE '12 hours' PRECEDING))*0.00000001 AS "revenue in BTC",
       average_x(rolling(stats_miner_revenue) OVER (ORDER BY bucket RANGE '12 hours' PRECEDING)) AS "revenue in USD"
   FROM one_hour_coinbase
   WHERE bucket > date_add('2023-11-22 00:00:00+00', INTERVAL '-5 days')
   ORDER BY 1;
   

3. The data you get back looks a bit like this:

             time          |   revenue in BTC   |   revenue in USD
   ------------------------+--------------------+--------------------
    2023-06-09 14:00:00+00 |       6.6732841925 |        176922.1133
    2023-06-09 15:00:00+00 |  6.785046736363636 |  179885.1576818182
    2023-06-09 16:00:00+00 |       6.7252952905 | 178301.02735000002
    2023-06-09 17:00:00+00 |  6.716377454814815 |  178064.5978074074
    2023-06-09 18:00:00+00 |    6.7784206471875 |   179709.487309375
   ...
   

4. To visualize this in Grafana, create a new panel, select the Bitcoin dataset as your data source, and type the query from the previous step. In the Format as section, select Time series.

5. To make this visualization more useful, add an override to put the US Dollars on a different Y-axis. In the options panel, add an override for the mining fee field for Axis > Placement and choose Right.

   

===== PAGE: https://docs.tigerdata.com/tutorials/nyc-taxi-cab/dataset-nyc/ =====

Query time-series data tutorial - set up dataset

This tutorial uses a dataset that contains historical data from the New York City Taxi and Limousine Commission NYC TLC, in a hypertable named rides. It also includes a separate tables of payment types and rates, in a regular Postgres table named payment_types, and rates.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Optimize time-series data in hypertables

Time-series data represents how a system, process, or behavior changes over time. Hypertables are Postgres tables that help you improve insert and query performance by automatically partitioning your data by time. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range.

Hypertables exist alongside regular Postgres tables. You interact with hypertables and regular Postgres tables in the same way. You use regular Postgres tables for relational data.

1. Create a hypertable to store the taxi trip data

   CREATE TABLE "rides"(
       vendor_id TEXT,
       pickup_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
       dropoff_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
       passenger_count NUMERIC,
       trip_distance NUMERIC,
       pickup_longitude  NUMERIC,
       pickup_latitude   NUMERIC,
       rate_code         INTEGER,
       dropoff_longitude NUMERIC,
       dropoff_latitude  NUMERIC,
       payment_type INTEGER,
       fare_amount NUMERIC,
       extra NUMERIC,
       mta_tax NUMERIC,
       tip_amount NUMERIC,
       tolls_amount NUMERIC,
       improvement_surcharge NUMERIC,
       total_amount NUMERIC
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='pickup_datetime',
      tsdb.create_default_indexes=false
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

2. Add another dimension to partition your hypertable more efficiently

   SELECT add_dimension('rides', by_hash('payment_type', 2));
   

3. Create an index to support efficient queries

   Index by vendor, rate code, and passenger count:

   CREATE INDEX ON rides (vendor_id, pickup_datetime DESC);
   CREATE INDEX ON rides (rate_code, pickup_datetime DESC);
   CREATE INDEX ON rides (passenger_count, pickup_datetime DESC);
   

Create standard Postgres tables for relational data

When you have other relational data that enhances your time-series data, you can create standard Postgres tables just as you would normally. For this dataset, there are two other tables of data, called payment_types and rates.

1. Add a relational table to store the payment types data

   CREATE TABLE IF NOT EXISTS "payment_types"(
       payment_type INTEGER,
       description TEXT
   );
   INSERT INTO payment_types(payment_type, description) VALUES
   (1, 'credit card'),
   (2, 'cash'),
   (3, 'no charge'),
   (4, 'dispute'),
   (5, 'unknown'),
   (6, 'voided trip');
   

2. Add a relational table to store the rates data

   CREATE TABLE IF NOT EXISTS "rates"(
       rate_code   INTEGER,
       description TEXT
   );
   INSERT INTO rates(rate_code, description) VALUES
   (1, 'standard rate'),
   (2, 'JFK'),
   (3, 'Newark'),
   (4, 'Nassau or Westchester'),
   (5, 'negotiated fare'),
   (6, 'group ride');
   

You can confirm that the scripts were successful by running the \dt command in the psql command line. You should see this:

           List of relations
 Schema |     Name      | Type  |  Owner
--------+---------------+-------+----------
 public | payment_types | table | tsdbadmin
 public | rates         | table | tsdbadmin
 public | rides         | table | tsdbadmin
(3 rows)

Load trip data

When you have your database set up, you can load the taxi trip data into the rides hypertable.

This is a large dataset, so it might take a long time, depending on your network connection.

1. Download the dataset:

nyc_data.tar.gz

1. Use your file manager to decompress the downloaded dataset, and take a note of the path to the nyc_data_rides.csv file.

2. At the psql prompt, copy the data from the nyc_data_rides.csv file into your hypertable. Make sure you point to the correct path, if it is not in your current working directory:

   \COPY rides FROM nyc_data_rides.csv CSV;
   

You can check that the data has been copied successfully with this command:

SELECT * FROM rides LIMIT 5;

You should get five records that look like this:

-[ RECORD 1 ]---------+--------------------
vendor_id             | 1
pickup_datetime       | 2016-01-01 00:00:01
dropoff_datetime      | 2016-01-01 00:11:55
passenger_count       | 1
trip_distance         | 1.20
pickup_longitude      | -73.979423522949219
pickup_latitude       | 40.744613647460938
rate_code             | 1
dropoff_longitude     | -73.992034912109375
dropoff_latitude      | 40.753944396972656
payment_type          | 2
fare_amount           | 9
extra                 | 0.5
mta_tax               | 0.5
tip_amount            | 0
tolls_amount          | 0
improvement_surcharge | 0.3
total_amount          | 10.3

===== PAGE: https://docs.tigerdata.com/tutorials/nyc-taxi-cab/index/ =====

Query time-series data tutorial

New York City is home to about 9 million people. This tutorial uses historical data from New York's yellow taxi network, provided by the New York City Taxi and Limousine Commission NYC TLC. The NYC TLC tracks over 200,000 vehicles making about 1 million trips each day. Because nearly all of this data is time-series data, proper analysis requires a purpose-built time-series database, like Timescale.

Prerequisites

Before you begin, make sure you have:

• Signed up for a free Tiger Data account.

Steps in this tutorial

This tutorial covers:

1. Setting up your dataset: Set up and connect to a Timescale service, and load data into your database using psql.
2. Querying your dataset: Analyze a dataset containing NYC taxi trip data using Tiger Cloud and Postgres.
3. Bonus: Store data efficiently: Learn how to store and query your NYC taxi trip data more efficiently using compression feature of Timescale.

About querying data with Timescale

This tutorial uses the NYC taxi data to show you how to construct queries for time-series data. The analysis you do in this tutorial is similar to the kind of analysis data science organizations use to do things like plan upgrades, set budgets, and allocate resources.

It starts by teaching you how to set up and connect to a Tiger Cloud service, create tables, and load data into the tables using psql.

You then learn how to conduct analysis and monitoring on your dataset. It walks you through using Postgres queries to obtain information, including how to use JOINs to combine your time-series data with relational or business data.

If you have been provided with a pre-loaded dataset on your Tiger Cloud service, go directly to the queries section.

===== PAGE: https://docs.tigerdata.com/tutorials/nyc-taxi-cab/query-nyc/ =====

Query time-series data tutorial - query the data

When you have your dataset loaded, you can start constructing some queries to discover what your data tells you. In this section, you learn how to write queries that answer these questions:

• How many rides take place each day?
• What is the average fare amount?
• How many rides of each rate type were taken?
• What kind of trips are going to and from airports?
• How many rides took place on New Year's Day 2016?

How many rides take place every day?

This dataset contains ride data for January 2016. To find out how many rides took place each day, you can use a SELECT statement. In this case, you want to count the total number of rides each day, and show them in a list by date.

Finding how many rides take place every day

1. Connect to the Tiger Cloud service that contains the NYC taxi dataset.

2. At the psql prompt, use this query to select all rides taken in the first week of January 2016, and return a count of rides for each day:

   SELECT date_trunc('day', pickup_datetime) as day,
   COUNT(*) FROM rides
   WHERE pickup_datetime < '2016-01-08'
   GROUP BY day
   ORDER BY day;
   

   The result of the query looks like this:

            day         | count
   ---------------------+--------
    2016-01-01 00:00:00 | 345037
    2016-01-02 00:00:00 | 312831
    2016-01-03 00:00:00 | 302878
    2016-01-04 00:00:00 | 316171
    2016-01-05 00:00:00 | 343251
    2016-01-06 00:00:00 | 348516
    2016-01-07 00:00:00 | 364894
   

What is the average fare amount?

You can include a function in your SELECT query to determine the average fare paid by each passenger.

Finding the average fare amount

1. Connect to the Tiger Cloud service that contains the NYC taxi dataset.

2. At the psql prompt, use this query to select all rides taken in the first week of January 2016, and return the average fare paid on each day:

   SELECT date_trunc('day', pickup_datetime)
   AS day, avg(fare_amount)
   FROM rides
   WHERE pickup_datetime < '2016-01-08'
   GROUP BY day
   ORDER BY day;
   

   The result of the query looks like this:

            day         |         avg
   ---------------------+---------------------
    2016-01-01 00:00:00 | 12.8569325028909943
    2016-01-02 00:00:00 | 12.4344713599355563
    2016-01-03 00:00:00 | 13.0615900461571986
    2016-01-04 00:00:00 | 12.2072927308323660
    2016-01-05 00:00:00 | 12.0018670885154013
    2016-01-06 00:00:00 | 12.0002329017893009
    2016-01-07 00:00:00 | 12.1234180337303436
   

How many rides of each rate type were taken?

Taxis in New York City use a range of different rate types for different kinds of trips. For example, trips to the airport are charged at a flat rate from any location within the city. This section shows you how to construct a query that shows you the nuber of trips taken for each different fare type. It also uses a JOIN statement to present the data in a more informative way.

Finding the number of rides for each fare type

1. Connect to the Tiger Cloud service that contains the NYC taxi dataset.

2. At the psql prompt, use this query to select all rides taken in the first week of January 2016, and return the total number of trips taken for each rate code:

   SELECT rate_code, COUNT(vendor_id) AS num_trips
   FROM rides
   WHERE pickup_datetime < '2016-01-08'
   GROUP BY rate_code
   ORDER BY rate_code;
   

   The result of the query looks like this:

    rate_code | num_trips
   -----------+-----------
            1 |   2266401
            2 |     54832
            3 |      4126
            4 |       967
            5 |      7193
            6 |        17
           99 |        42
   

This output is correct, but it's not very easy to read, because you probably don't know what the different rate codes mean. However, the rates table in the dataset contains a human-readable description of each code. You can use a JOIN statement in your query to connect the rides and rates tables, and present information from both in your results.

Displaying the number of rides for each fare type

1. Connect to the Tiger Cloud service that contains the NYC taxi dataset.

2. At the psql prompt, copy this query to select all rides taken in the first week of January 2016, join the rides and rates tables, and return the total number of trips taken for each rate code, with a description of the rate code:

   SELECT rates.description, COUNT(vendor_id) AS num_trips
   FROM rides
   JOIN rates ON rides.rate_code = rates.rate_code
   WHERE pickup_datetime < '2016-01-08'
   GROUP BY rates.description
   ORDER BY LOWER(rates.description);
   

   The result of the query looks like this:

         description      | num_trips
   -----------------------+-----------
    group ride            |        17
    JFK                   |     54832
    Nassau or Westchester |       967
    negotiated fare       |      7193
    Newark                |      4126
    standard rate         |   2266401
   

What kind of trips are going to and from airports

There are two primary airports in the dataset: John F. Kennedy airport, or JFK, is represented by rate code 2; Newark airport, or EWR, is represented by rate code 3.

Information about the trips that are going to and from the two airports is useful for city planning, as well as for organizations like the NYC Tourism Bureau.

This section shows you how to construct a query that returns trip information for trips going only to the new main airports.

Finding what kind of trips are going to and from airports

1. Connect to the Tiger Cloud service that contains the NYC taxi dataset.

2. At the psql prompt, use this query to select all rides taken to and from JFK and Newark airports, in the first week of January 2016, and return the number of trips to that airport, the average trip duration, average trip cost, and average number of passengers:

   SELECT rates.description,
       COUNT(vendor_id) AS num_trips,
       AVG(dropoff_datetime - pickup_datetime) AS avg_trip_duration,
       AVG(total_amount) AS avg_total,
       AVG(passenger_count) AS avg_passengers
   FROM rides
   JOIN rates ON rides.rate_code = rates.rate_code
   WHERE rides.rate_code IN (2,3) AND pickup_datetime < '2016-01-08'
   GROUP BY rates.description
   ORDER BY rates.description;
   

   The result of the query looks like this:

    description | num_trips | avg_trip_duration |      avg_total      |   avg_passengers
   -------------+-----------+-------------------+---------------------+--------------------
    JFK         |     54832 | 00:46:44.614222   | 63.7791311642836300 | 1.8062080536912752
    Newark      |      4126 | 00:34:45.575618   | 84.3841783809985458 | 1.8979641299079011
   

How many rides took place on New Year's Day 2016?

New York City is famous for the Ball Drop New Year's Eve celebration in Times Square. Thousands of people gather to bring in the New Year and then head out into the city: to their favorite bar, to gather with friends for a meal, or back home. This section shows you how to construct a query that returns the number of taxi trips taken on 1 January, 2016, in 30 minute intervals.

In Postgres, it's not particularly easy to segment the data by 30 minute time intervals. To do this, you would need to use a TRUNC function to calculate the quotient of the minute that a ride began in divided by 30, then truncate the result to take the floor of that quotient. When you had that result, you could multiply the truncated quotient by 30.

In your Tiger Cloud service, you can use the time_bucket function to segment the data into time intervals instead.

Finding how many rides took place on New Year's Day 2016

1. Connect to the Tiger Cloud service that contains the NYC taxi dataset.

2. At the psql prompt, use this query to select all rides taken on the first day of January 2016, and return a count of rides for each 30 minute interval:

   SELECT time_bucket('30 minute', pickup_datetime) AS thirty_min, count(*)
   FROM rides
   WHERE pickup_datetime < '2016-01-02 00:00'
   GROUP BY thirty_min
   ORDER BY thirty_min;
   

   The result of the query starts like this:

        thirty_min      | count
   ---------------------+-------
    2016-01-01 00:00:00 | 10920
    2016-01-01 00:30:00 | 14350
    2016-01-01 01:00:00 | 14660
    2016-01-01 01:30:00 | 13851
    2016-01-01 02:00:00 | 13260
    2016-01-01 02:30:00 | 12230
    2016-01-01 03:00:00 | 11362
   

===== PAGE: https://docs.tigerdata.com/tutorials/nyc-taxi-cab/compress-nyc/ =====

Query time-series data tutorial - set up compression

You have now seen how to create a hypertable for your NYC taxi trip data and query it. When ingesting a dataset like this is seldom necessary to update old data and over time the amount of data in the tables grows. Over time you end up with a lot of data and since this is mostly immutable you can compress it to save space and avoid incurring additional cost.

It is possible to use disk-oriented compression like the support offered by ZFS and Btrfs but since TimescaleDB is build for handling event-oriented data (such as time-series) it comes with support for compressing data in hypertables.

TimescaleDB compression allows you to store the data in a vastly more efficient format allowing up to 20x compression ratio compared to a normal Postgres table, but this is of course highly dependent on the data and configuration.

TimescaleDB compression is implemented natively in Postgres and does not require special storage formats. Instead it relies on features of Postgres to transform the data into columnar format before compression. The use of a columnar format allows better compression ratio since similar data is stored adjacently. For more details on how the compression format looks, you can look at the compression design section.

A beneficial side-effect of compressing data is that certain queries are significantly faster since less data has to be read into memory.

Compression setup

1. Connect to the Tiger Cloud service that contains the dataset using, for example psql.

2. Enable compression on the table and pick suitable segment-by and order-by column using the ALTER TABLE command:

   ALTER TABLE rides
   SET (
       timescaledb.compress,
       timescaledb.compress_segmentby='vendor_id',
       timescaledb.compress_orderby='pickup_datetime DESC'
   );
   

   Depending on the choice if segment-by and order-by column you can get very different performance and compression ratio. To learn more about how to pick the correct columns, see here.

3. You can manually compress all the chunks of the hypertable using compress_chunk in this manner:

   SELECT compress_chunk(c) from show_chunks('rides') c;
   

   You can also automate compression by adding a compression policy which will be covered below.

4. Now that you have compressed the table you can compare the size of the dataset before and after compression:

   SELECT
       pg_size_pretty(before_compression_total_bytes) as before,
       pg_size_pretty(after_compression_total_bytes) as after
    FROM hypertable_compression_stats('rides');
   

   This shows a significant improvement in data usage:

   before  | after
   ---------+--------
   1741 MB | 603 MB
   

Add a compression policy

To avoid running the compression step each time you have some data to compress you can set up a compression policy. The compression policy allows you to compress data that is older than a particular age, for example, to compress all chunks that are older than 8 days:

SELECT add_compression_policy('rides', INTERVAL '8 days');

Compression policies run on a regular schedule, by default once every day, which means that you might have up to 9 days of uncompressed data with the setting above.

You can find more information on compression policies in the add_compression_policy section.

Taking advantage of query speedups

Previously, compression was set up to be segmented by vendor_id column value. This means fetching data by filtering or grouping on that column will be more efficient. Ordering is also set to time descending so if you run queries which try to order data with that ordering, you should see performance benefits.

For instance, if you run the query example from previous section:

SELECT rate_code, COUNT(vendor_id) AS num_trips
FROM rides
WHERE pickup_datetime < '2016-01-08'
GROUP BY rate_code
ORDER BY rate_code;

You should see a decent performance difference when the dataset is compressed and when is decompressed. Try it yourself by running the previous query, decompressing the dataset and running it again while timing the execution time. You can enable timing query times in psql by running:

    \timing

To decompress the whole dataset, run:

    SELECT decompress_chunk(c) from show_chunks('rides') c;

On an example setup, speedup performance observed was pretty significant, 700 ms when compressed vs 1,2 sec when decompressed.

Try it yourself and see what you get!

===== PAGE: https://docs.tigerdata.com/tutorials/blockchain-query/blockchain-compress/ =====

Compress your data using hypercore

Over time you end up with a lot of data. Since this data is mostly immutable, you can compress it to save space and avoid incurring additional cost.

TimescaleDB is built for handling event-oriented data such as time-series and fast analytical queries, it comes with support of hypercore featuring the columnstore.

Hypercore enables you to store the data in a vastly more efficient format allowing up to 90x compression ratio compared to a normal Postgres table. However, this is highly dependent on the data and configuration.

Hypercore is implemented natively in Postgres and does not require special storage formats. When you convert your data from the rowstore to the columnstore, TimescaleDB uses Postgres features to transform the data into columnar format. The use of a columnar format allows a better compression ratio since similar data is stored adjacently. For more details on the columnar format, see hypercore.

A beneficial side effect of compressing data is that certain queries are significantly faster, since less data has to be read into memory.

Optimize your data in the columnstore

To compress the data in the transactions table, do the following:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed. You can also connect to your service using psql.

2. Convert data to the columnstore:

   You can do this either automatically or manually:

   • Automatically convert chunks in the hypertable to the columnstore at a specific time interval:

     CALL add_columnstore_policy('transactions', after => INTERVAL '1d');
     

   • Manually convert all chunks in the hypertable to the columnstore:

     DO $$
     DECLARE
        chunk_name TEXT;
     BEGIN
        FOR chunk_name IN (SELECT c FROM show_chunks('transactions') c)
        LOOP
           RAISE NOTICE 'Converting chunk: %', chunk_name; -- Optional: To see progress
           CALL convert_to_columnstore(chunk_name);
        END LOOP;
        RAISE NOTICE 'Conversion to columnar storage complete for all chunks.'; -- Optional: Completion message
     END$$;
     

Take advantage of query speedups

Previously, data in the columnstore was segmented by the block_id column value. This means fetching data by filtering or grouping on that column is more efficient. Ordering is set to time descending. This means that when you run queries which try to order data in the same way, you see performance benefits.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed.

2. Run the following query:

   WITH recent_blocks AS (
    SELECT block_id FROM transactions
    WHERE is_coinbase IS TRUE
    ORDER BY time DESC
    LIMIT 5
   )
   SELECT
    t.block_id, count(*) AS transaction_count,
    SUM(weight) AS block_weight,
    SUM(output_total_usd) AS block_value_usd
   FROM transactions t
   INNER JOIN recent_blocks b ON b.block_id = t.block_id
   WHERE is_coinbase IS NOT TRUE
   GROUP BY t.block_id;
   

   Performance speedup is of two orders of magnitude, around 15 ms when compressed in the columnstore and 1 second when decompressed in the rowstore.

===== PAGE: https://docs.tigerdata.com/tutorials/blockchain-query/blockchain-dataset/ =====

Query the Bitcoin blockchain - set up dataset

Ingest data into a Tiger Cloud service

This tutorial uses a dataset that contains Bitcoin blockchain data for the past five days, in a hypertable named transactions.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Optimize time-series data using hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. The in-Console editors display the query speed. You can also connect to your service using psql.

2. Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data:

   CREATE TABLE transactions (
      time TIMESTAMPTZ NOT NULL,
      block_id INT,
      hash TEXT,
      size INT,
      weight INT,
      is_coinbase BOOLEAN,
      output_total BIGINT,
      output_total_usd DOUBLE PRECISION,
      fee BIGINT,
      fee_usd DOUBLE PRECISION,
      details JSONB
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time',
      tsdb.segmentby='block_id',
      tsdb.orderby='time DESC'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

3. Create an index on the hash column to make queries for individual transactions faster:

   CREATE INDEX hash_idx ON public.transactions USING HASH (hash);
   

4. Create an index on the block_id column to make block-level queries faster:

When you create a hypertable, it is partitioned on the time column. TimescaleDB automatically creates an index on the time column. However, you'll often filter your time-series data on other columns as well. You use indexes to improve query performance.

```sql
CREATE INDEX block_idx ON public.transactions (block_id);
```

1. Create a unique index on the time and hash columns to make sure you don't accidentally insert duplicate records:

   CREATE UNIQUE INDEX time_hash_idx ON public.transactions (time, hash);
   

Load financial data

The dataset contains around 1.5 million Bitcoin transactions, the trades for five days. It includes information about each transaction, along with the value in satoshi. It also states if a trade is a coinbase transaction, and the reward a coin miner receives for mining the coin.

To ingest data into the tables that you created, you need to download the dataset and copy the data to your database.

1. Download the bitcoin_sample.zip file. The file contains a .csv file that contains Bitcoin transactions for the past five days. Download:

   bitcoin_sample.zip

2. In a new terminal window, run this command to unzip the .csv files:

   unzip bitcoin_sample.zip
   

3. In Terminal, navigate to the folder where you unzipped the Bitcoin transactions, then connect to your service using psql.

4. At the psql prompt, use the COPY command to transfer data into your Tiger Cloud service. If the .csv files aren't in your current directory, specify the file paths in these commands:

   \COPY transactions FROM 'tutorial_bitcoin_sample.csv' CSV HEADER;
   

   Because there is over a million rows of data, the COPY process could take a few minutes depending on your internet connection and local client resources.

===== PAGE: https://docs.tigerdata.com/tutorials/blockchain-query/beginner-blockchain-query/ =====

Query the Bitcoin blockchain - query data

When you have your dataset loaded, you can start constructing some queries to discover what your data tells you. In this section, you learn how to write queries that answer these questions:

• What are the five most recent coinbase transactions?
• What are the five most recent transactions?
• What are the five most recent blocks?

What are the five most recent coinbase transactions?

In the last procedure, you excluded coinbase transactions from the results. Coinbase transactions are the first transaction in a block, and they include the reward a coin miner receives for mining the coin. To find out the most recent coinbase transactions, you can use a similar SELECT statement, but search for transactions that are coinbase instead. If you include the transaction value in US Dollars again, you'll notice that the value is $0 for each. This is because the coin has not transferred ownership in coinbase transactions.

Finding the five most recent coinbase transactions

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to select the five most recent coinbase transactions:

   SELECT time, hash, block_id, fee_usd  FROM transactions
   WHERE is_coinbase IS TRUE
   ORDER BY time DESC
   LIMIT 5;
   

3. The data you get back looks a bit like this:

                time          |                               hash                               | block_id | fee_usd
   ------------------------+------------------------------------------------------------------+----------+---------
    2023-06-12 23:54:18+00 | 22e4610bc12d482bc49b7a1c5b27ad18df1a6f34256c16ee7e499b511e02d71e |   794111 |       0
    2023-06-12 23:53:08+00 | dde958bb96a302fd956ced32d7b98dd9860ff82d569163968ecfe29de457fedb |   794110 |       0
    2023-06-12 23:44:50+00 | 75ac1fa7febe1233ee57ca11180124c5ceb61b230cdbcbcba99aecc6a3e2a868 |   794109 |       0
    2023-06-12 23:44:14+00 | 1e941d66b92bf0384514ecb83231854246a94c86ff26270fbdd9bc396dbcdb7b |   794108 |       0
    2023-06-12 23:41:08+00 | 60ae50447254d5f4561e1c297ee8171bb999b6310d519a0d228786b36c9ffacf |   794107 |       0
   (5 rows)
   

What are the five most recent transactions?

This dataset contains Bitcoin transactions for the last five days. To find out the most recent transactions in the dataset, you can use a SELECT statement. In this case, you want to find transactions that are not coinbase transactions, sort them by time in descending order, and take the top five results. You also want to see the block ID, and the value of the transaction in US Dollars.

Finding the five most recent transactions

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to select the five most recent non-coinbase transactions:

   SELECT time, hash, block_id, fee_usd  FROM transactions
   WHERE is_coinbase IS NOT TRUE
   ORDER BY time DESC
   LIMIT 5;
   

3. The data you get back looks a bit like this:

                 time          |                               hash                               | block_id | fee_usd
   ------------------------+------------------------------------------------------------------+----------+---------
    2023-06-12 23:54:18+00 | 6f709d52e9aa7b2569a7f8c40e7686026ede6190d0532220a73fdac09deff973 |   794111 |   7.614
    2023-06-12 23:54:18+00 | ece5429f4a76b1603aecbee31bf3d05f74142a260e4023316250849fe49115ae |   794111 |   9.306
    2023-06-12 23:54:18+00 | 54a196398880a7e2e38312d4285fa66b9c7129f7d14dc68c715d783322544942 |   794111 | 13.1928
    2023-06-12 23:54:18+00 | 3e83e68735af556d9385427183e8160516fafe2f30f30405711c4d64bf0778a6 |   794111 |  3.5416
    2023-06-12 23:54:18+00 | ca20d073b1082d7700b3706fe2c20bc488d2fc4a9bb006eb4449efe3c3fc6b2b |   794111 |  8.6842
   (5 rows)
   

What are the five most recent blocks?

In this procedure, you use a more complicated query to return the five most recent blocks, and show some additional information about each, including the block weight, number of transactions in each block, and the total block value in US Dollars.

Finding the five most recent blocks

1. Connect to the Tiger Cloud service that contains the Bitcoin dataset.

2. At the psql prompt, use this query to select the five most recent coinbase transactions:

   WITH recent_blocks AS (
    SELECT block_id FROM transactions
    WHERE is_coinbase IS TRUE
    ORDER BY time DESC
    LIMIT 5
   )
   SELECT
    t.block_id, count(*) AS transaction_count,
    SUM(weight) AS block_weight,
    SUM(output_total_usd) AS block_value_usd
   FROM transactions t
   INNER JOIN recent_blocks b ON b.block_id = t.block_id
   WHERE is_coinbase IS NOT TRUE
   GROUP BY t.block_id;
   

3. The data you get back looks a bit like this:

    block_id | transaction_count | block_weight |  block_value_usd
   ----------+-------------------+--------------+--------------------
      794108 |              5625 |      3991408 |  65222453.36381342
      794111 |              5039 |      3991748 |  5966031.481099684
      794109 |              6325 |      3991923 |  5406755.801599815
      794110 |              2525 |      3995553 |  177249139.6457974
      794107 |              4464 |      3991838 | 107348519.36559173
   (5 rows)
   

===== PAGE: https://docs.tigerdata.com/tutorials/OLD-financial-candlestick-tick-data/create-candlestick-aggregates/ =====

Create candlestick aggregates

Turning raw, real-time tick data into aggregated candlestick views is a common task for users who work with financial data. If your data is not tick data, for example if you receive it in an already aggregated form such as 1-min buckets, you can still use these functions to help you create additional aggregates of your data into larger buckets, such as 1-hour or 1-day buckets. If you want to work with pre-aggregated stock and crypto data, see the Analyzing Intraday Stock Data tutorial for more examples.

TimescaleDB includes hyperfunctions that you can use to store and query your financial data more easily. Hyperfunctions are SQL functions within TimescaleDB that make it easier to manipulate and analyze time-series data in Postgres with fewer lines of code. There are three hyperfunctions that are essential for calculating candlestick values: time_bucket(), FIRST(), and LAST().

The time_bucket() hyperfunction helps you aggregate records into buckets of arbitrary time intervals based on the timestamp value. FIRST() and LAST() help you calculate the opening and closing prices. To calculate highest and lowest prices, you can use the standard Postgres aggregate functions MIN and MAX.

In this first SQL example, use the hyperfunctions to query the tick data, and turn it into 1-min candlestick values in the candlestick format:

-- Create the candlestick format
SELECT
    time_bucket('1 min', time) AS bucket,
    symbol,
    FIRST(price, time) AS "open",
    MAX(price) AS high,
    MIN(price) AS low,
    LAST(price, time) AS "close",
    LAST(day_volume, time) AS day_volume
FROM crypto_ticks
GROUP BY bucket, symbol

Hyperfunctions in this query:

• time_bucket('1 min', time): creates 1-minute buckets
• FIRST(price, time): selects the first price value in the bucket, ordered by time, which is the opening price of the candlestick.
• LAST(price, time) selects the last price value in the bucket, ordered by time, which is the closing price of the candlestick

Besides the hyperfunctions, you can see other common SQL aggregate functions like MIN and MAX, which calculate the lowest and highest prices in the candlestick.

This tutorial uses the LAST() hyperfunction to calculate the volume within a bucket, because the sample tick data already provides an incremental day_volume field which contains the total volume for the given day with each trade. Depending on the raw data you receive and whether you want to calculate volume in terms of trade count or the total value of the trades, you might need to use COUNT(*), SUM(price), or subtraction between the last and first values in the bucket to get the correct result.

Create continuous aggregates for candlestick data

In TimescaleDB, the most efficient way to create candlestick views is to use continuous aggregates. Continuous aggregates are very similar to Postgres materialized views but with three major advantages.

First, materialized views recreate all of the data any time the view is refreshed, which causes history to be lost. Continuous aggregates only refresh the buckets of aggregated data where the source, raw data has been changed or added.

Second, continuous aggregates can be automatically refreshed using built-in, user-configured policies. No special triggers or stored procedures are needed to refresh the data over time.

Finally, continuous aggregates are real-time by default. Any new raw tick data that is inserted between refreshes is automatically appended to the materialized data. This keeps your candlestick data up-to-date without having to write special SQL to UNION data from multiple views and tables.

Continuous aggregates are often used to power dashboards and other user-facing applications, like price charts, where query performance and timeliness of your data matter.

Let's see how to create different candlestick time buckets - 1 minute, 1 hour, and 1 day - using continuous aggregates with different refresh policies.

1-minute candlestick

To create a continuous aggregate of 1-minute candlestick data, use the same query that you previously used to get the 1-minute OHLCV values. But this time, put the query in a continuous aggregate definition:

/* 1-min candlestick view*/
CREATE MATERIALIZED VIEW one_min_candle
WITH (timescaledb.continuous) AS
    SELECT
        time_bucket('1 min', time) AS bucket,
        symbol,
        FIRST(price, time) AS "open",
        MAX(price) AS high,
        MIN(price) AS low,
        LAST(price, time) AS "close",
        LAST(day_volume, time) AS day_volume
    FROM crypto_ticks
    GROUP BY bucket, symbol

When you run this query, TimescaleDB queries 1-minute aggregate values of all your tick data, creating the continuous aggregate and materializing the results. But your candlestick data has only been materialized up to the last data point. If you want the continuous aggregate to stay up to date as new data comes in over time, you also need to add a continuous aggregate refresh policy. For example, to refresh the continuous aggregate every two minutes:

/* Refresh the continuous aggregate every two minutes */
SELECT add_continuous_aggregate_policy('one_min_candle',
    start_offset => INTERVAL '2 hour',
    end_offset => INTERVAL '10 sec',
    schedule_interval => INTERVAL '2 min');

The continuous aggregate refreshes every hour, so every hour new candlesticks are materialized, if there's new raw tick data in the hypertable.

When this job runs, it only refreshes the time period between start_offset and end_offset, and ignores modifications outside of this window.

In most cases, set end_offset to be the same or bigger as the time bucket in the continuous aggregate definition. This makes sure that only full buckets get materialized during the refresh process.

1-hour candlestick

To create a 1-hour candlestick view, follow the same process as in the previous step, except this time set the time bucket value to be one hour in the continuous aggregate definition:

/* 1-hour candlestick view */
CREATE MATERIALIZED VIEW one_hour_candle
WITH (timescaledb.continuous) AS
    SELECT
        time_bucket('1 hour', time) AS bucket,
        symbol,
        FIRST(price, time) AS "open",
        MAX(price) AS high,
        MIN(price) AS low,
        LAST(price, time) AS "close",
        LAST(day_volume, time) AS day_volume
    FROM crypto_ticks
    GROUP BY bucket, symbol

Add a refresh policy to refresh the continuous aggregate every hour:

/* Refresh the continuous aggregate every hour */
SELECT add_continuous_aggregate_policy('one_hour_candle',
    start_offset => INTERVAL '1 day',
    end_offset => INTERVAL '1 min',
    schedule_interval => INTERVAL '1 hour');

Notice how this example uses a different refresh policy with different parameter values to accommodate the 1-hour time bucket in the continuous aggregate definition. The continuous aggregate will refresh every hour, so every hour there will be new candlestick data materialized, if there's new raw tick data in the hypertable.

1-day candlestick

Create the final view in this tutorial for 1-day candlesticks using the same process as above, using a 1-day time bucket size:

/* 1-day candlestick */
CREATE MATERIALIZED VIEW one_day_candle
WITH (timescaledb.continuous) AS
    SELECT
        time_bucket('1 day', time) AS bucket,
        symbol,
        FIRST(price, time) AS "open",
        MAX(price) AS high,
        MIN(price) AS low,
        LAST(price, time) AS "close",
        LAST(day_volume, time) AS day_volume
    FROM crypto_ticks
    GROUP BY bucket, symbol

Add a refresh policy to refresh the continuous aggregate once a day:

/* Refresh the continuous aggregate every day */
SELECT add_continuous_aggregate_policy('one_day_candle',
    start_offset => INTERVAL '3 day',
    end_offset => INTERVAL '1 day',
    schedule_interval => INTERVAL '1 day');

The refresh job runs every day, and materializes two days' worth of candlesticks.

Optional: add price change (delta) column in the candlestick view

As an optional step, you can add an additional column in the continuous aggregate to calculate the price difference between the opening and closing price within the bucket.

In general, you can calculate the price difference with the formula:

(CLOSE PRICE - OPEN PRICE) / OPEN PRICE = delta

Calculate delta in SQL:

SELECT time_bucket('1 day', time) AS bucket, symbol, (LAST(price, time)-FIRST(price, time))/FIRST(price, time) AS change_pct
FROM crypto_ticks
WHERE price != 0
GROUP BY bucket, symbol

The full continuous aggregate definition for a 1-day candlestick with a price-change column:

/* 1-day candlestick with price change column*/
CREATE MATERIALIZED VIEW one_day_candle_delta
WITH (timescaledb.continuous) AS
    SELECT
        time_bucket('1 day', time) AS bucket,
        symbol,
        FIRST(price, time) AS "open",
        MAX(price) AS high,
        MIN(price) AS low,
        LAST(price, time) AS "close",
        LAST(day_volume, time) AS day_volume,
        (LAST(price, time)-FIRST(price, time))/FIRST(price, time) AS change_pct
    FROM crypto_ticks
    WHERE price != 0
    GROUP BY bucket, symbol

Using multiple continuous aggregates

You cannot currently create a continuous aggregate on top of another continuous aggregate. However, this is not necessary in most cases. You can get a similar result and performance by creating multiple continuous aggregates for the same hypertable. Due to the efficient materialization mechanism of continuous aggregates, both refresh and query performance should work well.

===== PAGE: https://docs.tigerdata.com/tutorials/OLD-financial-candlestick-tick-data/query-candlestick-views/ =====

Query candlestick views

So far in this tutorial, you have created the schema to store tick data, and set up multiple candlestick views. In this section, use some example candlestick queries and see how they can be represented in data visualizations.

The queries in this section are example queries. The sample data provided with this tutorial is updated on a regular basis to have near-time data, typically no more than a few days old. Our sample queries reflect time filters that might be longer than you would normally use, so feel free to modify the time filter in the WHERE clause as the data ages, or as you begin to insert updated tick readings.

1-min BTC/USD candlestick chart

Start with a one_min_candle continuous aggregate, which contains 1-min candlesticks:

SELECT * FROM one_min_candle
WHERE symbol = 'BTC/USD' AND bucket >= NOW() - INTERVAL '24 hour'
ORDER BY bucket

1-hour BTC/USD candlestick chart

If you find that 1-min candlesticks are too granular, you can query the one_hour_candle continuous aggregate containing 1-hour candlesticks:

SELECT * FROM one_hour_candle
WHERE symbol = 'BTC/USD' AND bucket >= NOW() - INTERVAL '2 day'
ORDER BY bucket

1-day BTC/USD candlestick chart

To zoom out even more, query the one_day_candle continuous aggregate, which has one-day candlesticks:

SELECT * FROM one_day_candle
WHERE symbol = 'BTC/USD' AND bucket >= NOW() - INTERVAL '14 days'
ORDER BY bucket

BTC vs. ETH 1-day price changes delta line chart

You can calculate and visualize the price change differences between two symbols. In a previous example, you saw how to do this by comparing the opening and closing prices. But what if you want to compare today's closing price with yesterday's closing price? Here's an example how you can achieve this by using the LAG() window function on an already existing candlestick view:

SELECT *, ("close" - LAG("close", 1) OVER (PARTITION BY symbol ORDER BY bucket)) / "close" AS change_pct
FROM one_day_candle
WHERE symbol IN ('BTC/USD', 'ETH/USD') AND bucket >= NOW() - INTERVAL '14 days'
ORDER BY bucket

===== PAGE: https://docs.tigerdata.com/tutorials/OLD-financial-candlestick-tick-data/design-tick-schema/ =====

Design schema and ingest tick data

This tutorial shows you how to store real-time cryptocurrency or stock tick data in TimescaleDB. The initial schema provides the foundation to store tick data only. Once you begin to store individual transactions, you can calculate the candlestick values using TimescaleDB continuous aggregates based on the raw tick data. This means that our initial schema doesn't need to specifically store candlestick data.

Schema

This schema uses two tables:

• crypto_assets: a relational table that stores the symbols to monitor. You can also include additional information about each symbol, such as social links.
• crypto_ticks: a time-series table that stores the real-time tick data.

crypto_assets:

Field	Description
symbol	The symbol of the crypto currency pair, such as BTC/USD
name	The name of the pair, such as Bitcoin USD

crypto_ticks:

Field	Description
time	Timestamp, in UTC time zone
symbol	Crypto pair symbol from the crypto_assets table
price	The price registered on the exchange at that time
day_volume	Total volume for the given day (incremental)

Create the tables:

CREATE TABLE crypto_assets (
    symbol TEXT UNIQUE,
    "name" TEXT
);

CREATE TABLE crypto_ticks (
    "time" TIMESTAMPTZ,
    symbol TEXT,
    price DOUBLE PRECISION,
    day_volume NUMERIC
);

You also need to turn the time-series table into a hypertable:

-- convert the regular 'crypto_ticks' table into a TimescaleDB hypertable with 7-day chunks
SELECT create_hypertable('crypto_ticks', 'time');

This is an important step in order to efficiently store your time-series data in TimescaleDB.

Using TIMESTAMP data types

It is best practice to store time values using the TIMESTAMP WITH TIME ZONE (TIMESTAMPTZ) data type. This makes it easier to query your data using different time zones. TimescaleDB stores TIMESTAMPTZ values in UTC internally and makes the necessary conversions for your queries.

Insert tick data

With the hypertable and relational table created, download the sample files containing crypto assets and tick data from the last three weeks. Insert the data into your TimescaleDB instance.

Inserting sample data

1. Download the sample .csv files (provided by Twelve Data): crypto_sample.csv

   wget https://assets.timescale.com/docs/downloads/candlestick/crypto_sample.zip
   

2. Unzip the file and change the directory if you need to:

   unzip crypto_sample.zip
   cd crypto_sample
   

3. At the psql prompt, insert the content of the .csv files into the database.

   psql -x "postgres://tsdbadmin:{YOUR_PASSWORD_HERE}@{YOUR_HOSTNAME_HERE}:{YOUR_PORT_HERE}/tsdb?sslmode=require"
   
   \COPY crypto_assets FROM 'crypto_assets.csv' CSV HEADER;
   \COPY crypto_ticks FROM 'crypto_ticks.csv' CSV HEADER;
   

If you want to ingest real-time market data, instead of sample data, check out our complementing tutorial Ingest real-time financial websocket data to ingest data directly from the Twelve Data financial API.

===== PAGE: https://docs.tigerdata.com/tutorials/OLD-financial-candlestick-tick-data/index/ =====

Store financial tick data in TimescaleDB using the OHLCV (candlestick) format

Candlestick charts are the standard way to analyze the price changes of financial assets. They can be used to examine trends in stock prices, cryptocurrency prices, or even NFT prices. To generate candlestick charts, you need candlestick data in the OHLCV format. That is, you need the Open, High, Low, Close, and Volume data for some financial assets.

This tutorial shows you how to efficiently store raw financial tick data, create different candlestick views, and query aggregated data in TimescaleDB using the OHLCV format. It also shows you how to download sample data containing real-world crypto tick transactions for cryptocurrencies like BTC, ETH, and other popular assets.

Prerequisites

Before you begin, make sure you have:

• A TimescaleDB instance running locally or on the cloud. For more information, see the Getting Started guide
• psql, DBeaver, or any other Postgres client

What's candlestick data and OHLCV?

Candlestick charts are used in the financial sector to visualize the price change of an asset. Each candlestick represents a time frame (for example, 1 minute, 5 minutes, 1 hour, or similar) and shows how the asset's price changed during that time.

Candlestick charts are generated from candlestick data, which is the collection of data points used in the chart. This is often abbreviated as OHLCV (open-high-low-close-volume):

• Open: opening price
• High: highest price
• Low: lowest price
• Close: closing price
• Volume: volume of transactions

These data points correspond to the bucket of time covered by the candlestick. For example, a 1-minute candlestick would need the open and close prices for that minute.

Many Tiger Data community members use TimescaleDB to store and analyze candlestick data. Here are some examples:

• How Trading Strategy built a data stack for crypto quant trading
• How Messari uses data to open the cryptoeconomy to everyone
• How I power a (successful) crypto trading bot with TimescaleDB

Follow this tutorial and see how to set up your TimescaleDB database to consume real-time tick or aggregated financial data and generate candlestick views efficiently.

• Design schema and ingest tick data
• Create candlestick (open-high-low-close-volume) aggregates
• Query candlestick views
• Advanced data management

===== PAGE: https://docs.tigerdata.com/tutorials/OLD-financial-candlestick-tick-data/advanced-data-management/ =====

Advanced data management

The final part of this tutorial shows you some more advanced techniques to efficiently manage your tick and candlestick data long-term. TimescaleDB is equipped with multiple features that help you manage your data lifecycle and reduce your disk storage needs as your data grows.

This section contains four examples of how you can set up automation policies on your tick data hypertable and your candlestick continuous aggregates. This can help you save on disk storage and improve the performance of long-range analytical queries by automatically:

• Deleting older tick data
• Deleting older candlestick data
• Compressing tick data
• Compressing candlestick data

Before you implement any of these automation policies, it's important to have a high-level understanding of chunk time intervals in TimescaleDB hypertables and continuous aggregates. The chunk time interval you set for your tick data table directly affects how these automation policies work. For more information, see the hypertables and chunks section.

Hypertable chunk time intervals and automation policies

TimescaleDB uses hypertables to provide a high-level and familiar abstraction layer to interact with Postgres tables. You just need to access one hypertable to access all of your time-series data.

Under the hood, TimescaleDB creates chunks based on the timestamp column. Each chunk size is determined by the chunk_time_interval parameter. You can provide this parameter when creating the hypertable, or you can change it afterwards. If you don't provide this optional parameter, the chunk time interval defaults to 7 days. This means that each of the chunks in the hypertable contains 7 days' worth of data.

Knowing your chunk time interval is important. All of the TimescaleDB automation policies described in this section depend on this information, and the chunk time interval fundamentally affects how these policies impact your data.

In this section, learn about these automation policies and how they work in the context of financial tick data.

Automatically delete older tick data

Usually, the older your time-series data, the less relevant and useful it is. This is often the case with tick data as well. As time passes, you might not need the raw tick data any more, because you only want to query the candlestick aggregations. In this scenario, you can decide to remove tick data automatically from your hypertable after it gets older than a certain time interval.

TimescaleDB has a built-in way to automatically remove raw data after a specific time. You can set up this automation using a data retention policy:

SELECT add_retention_policy('crypto_ticks', INTERVAL '7 days');

When you run this, it adds a data retention policy to the crypto_ticks hypertable that removes a chunk after all the data in the chunk becomes older than 7 days. All records in the chunk need to be older than 7 days before the chunk is dropped.

Knowledge of your hypertable's chunk time interval is crucial here. If you were to set a data retention policy with INTERVAL '3 days', the policy would not remove any data after three days, because your chunk time interval is seven days. Even after three days have passed, the most recent chunk still contains data that is newer than three days, and so cannot be removed by the data retention policy.

If you want to change this behavior, and drop chunks more often and sooner, experiment with different chunk time intervals. For example, if you set the chunk time interval to be two days only, you could create a retention policy with a 2-day interval that would drop a chunk every other day (assuming you're ingesting data in the meantime).

For more information, see the data retention section.

Make sure none of the continuous aggregate policies intersect with a data retention policy. It's possible to keep the candlestick data in the continuous aggregate and drop tick data from the underlying hypertable, but only if you materialize data in the continuous aggregate first, before the data is dropped from the underlying hypertable.

Automatically delete older candlestick data

Deleting older raw tick data from your hypertable while retaining aggregate views for longer periods is a common way of minimizing disk utilization. However, deleting older candlestick data from the continuous aggregates can provide another method for further control over long-term disk use. TimescaleDB allows you to create data retention policies on continuous aggregates as well.

Continuous aggregates also have chunk time intervals because they use hypertables in the background. By default, the continuous aggregate's chunk time interval is 10 times what the original hypertable's chunk time interval is. For example, if the original hypertable's chunk time interval is 7 days, the continuous aggregates that are on top of it will have a 70 day chunk time interval.

You can set up a data retention policy to remove old data from your one_min_candle continuous aggregate:

SELECT add_retention_policy('one_min_candle', INTERVAL '70 days');

This data retention policy removes chunks from the continuous aggregate that are older than 70 days. In TimescaleDB, this is determined by the range_end property of a hypertable, or in the case of a continuous aggregate, the materialized hypertable. In practice, this means that if you were to define a data retention policy of 30 days for a continuous aggregate that has a chunk_time_interval of 70 days, data would not be removed from the continuous aggregates until the range_end of a chunk is at least 70 days older than the current time, due to the chunk time interval of the original hypertable.

Automatically compress tick data

TimescaleDB allows you to keep your tick data in the hypertable but still save on storage costs with TimescaleDB's native compression. You need to enable compression on the hypertable and set up a compression policy to automatically compress old data.

Enable compression on crypto_ticks hypertable:

ALTER TABLE crypto_ticks SET (
 timescaledb.compress,
 timescaledb.compress_segmentby = 'symbol'
);

Set up compression policy to compress data that's older than 7 days:

SELECT add_compression_policy('crypto_ticks', INTERVAL '7 days');

Executing these two SQL scripts compresses chunks that are older than 7 days.

For more information, see the compression section.

Automatically compress candlestick data

Beginning with TimescaleDB 2.6, you can also set up a compression policy on your continuous aggregates. This is a useful feature if you store a lot of historical candlestick data that consumes significant disk space, but you still want to retain it for longer periods.

Enable compression on the one_min_candle view:

ALTER MATERIALIZED VIEW one_min_candle set (timescaledb.compress = true);

Add a compression policy to compress data after 70 days:

SELECT add_compression_policy('one_min_candle', compress_after=> INTERVAL '70 days');

Before setting a compression policy on any of the candlestick views, set a refresh policy first. The compression policy interval should be set so that actively refreshed time intervals are not compressed.

Read more about compressing continuous aggregates.

===== PAGE: https://docs.tigerdata.com/tutorials/energy-data/dataset-energy/ =====

Energy time-series data tutorial - set up dataset

This tutorial uses the energy consumption data for over a year in a hypertable named metrics.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Optimize time-series data in hypertables

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. To create a hypertable to store the energy consumption data, call CREATE TABLE.

   CREATE TABLE "metrics"(
       created timestamp with time zone default now() not null,
       type_id integer                                not null,
       value   double precision                       not null
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

Load energy consumption data

When you have your database set up, you can load the energy consumption data into the metrics hypertable.

This is a large dataset, so it might take a long time, depending on your network connection.

1. Download the dataset:

metrics.csv.gz

1. Use your file manager to decompress the downloaded dataset, and take a note of the path to the metrics.csv file.

2. At the psql prompt, copy the data from the metrics.csv file into your hypertable. Make sure you point to the correct path, if it is not in your current working directory:

   \COPY metrics FROM metrics.csv CSV;
   

3. You can check that the data has been copied successfully with this command:

   SELECT * FROM metrics LIMIT 5;
   

   You should get five records that look like this:

            created            | type_id | value
   -------------------------------+---------+-------
    2023-05-31 23:59:59.043264+00 |      13 |  1.78
    2023-05-31 23:59:59.042673+00 |       2 |   126
    2023-05-31 23:59:59.042667+00 |      11 |  1.79
    2023-05-31 23:59:59.042623+00 |      23 | 0.408
    2023-05-31 23:59:59.042603+00 |      12 |  0.96
   

Create continuous aggregates

In modern applications, data usually grows very quickly. This means that aggregating it into useful summaries can become very slow. If you are collecting data very frequently, you might want to aggregate your data into minutes or hours instead. For example, if an IoT device takes temperature readings every second, you might want to find the average temperature for each hour. Every time you run this query, the database needs to scan the entire table and recalculate the average. TimescaleDB makes aggregating data lightning fast, accurate, and easy with continuous aggregates.

Continuous aggregates in TimescaleDB are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background.

Continuous aggregates have a much lower maintenance burden than regular Postgres materialized views, because the whole view is not created from scratch on each refresh. This means that you can get on with working your data instead of maintaining your database.

Because continuous aggregates are based on hypertables, you can query them in exactly the same way as your other tables. This includes continuous aggregates in the rowstore, compressed into the columnstore, or tiered to object storage. You can even create continuous aggregates on top of your continuous aggregates, for an even more fine-tuned aggregation.

Real-time aggregation enables you to combine pre-aggregated data from the materialized view with the most recent raw data. This gives you up-to-date results on every query. In TimescaleDB v2.13 and later, real-time aggregates are DISABLED by default. In earlier versions, real-time aggregates are ENABLED by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data.

1. Monitor energy consumption on a day-to-day basis

   1. Create a continuous aggregate kwh_day_by_day for energy consumption:

      CREATE MATERIALIZED VIEW kwh_day_by_day(time, value)
         with (timescaledb.continuous) as
      SELECT time_bucket('1 day', created, 'Europe/Berlin') AS "time",
             round((last(value, created) - first(value, created)) * 100.) / 100. AS value
      FROM metrics
      WHERE type_id = 5
      GROUP BY 1;
      

   2. Add a refresh policy to keep kwh_day_by_day up-to-date:

      SELECT add_continuous_aggregate_policy('kwh_day_by_day',
         start_offset => NULL,
         end_offset => INTERVAL '1 hour',
         schedule_interval => INTERVAL '1 hour');
      

2. Monitor energy consumption on an hourly basis

   1. Create a continuous aggregate kwh_hour_by_hour for energy consumption:

      CREATE MATERIALIZED VIEW kwh_hour_by_hour(time, value)
        with (timescaledb.continuous) as
      SELECT time_bucket('01:00:00', metrics.created, 'Europe/Berlin') AS "time",
             round((last(value, created) - first(value, created)) * 100.) / 100. AS value
      FROM metrics
      WHERE type_id = 5
      GROUP BY 1;
      

   2. Add a refresh policy to keep the continuous aggregate up-to-date:

   SELECT add_continuous_aggregate_policy('kwh_hour_by_hour',
    start_offset => NULL,
       end_offset => INTERVAL '1 hour',
       schedule_interval => INTERVAL '1 hour');
   

3. Analyze your data

   Now you have made continuous aggregates, it could be a good idea to use them to perform analytics on your data. For example, to see how average energy consumption changes during weekdays over the last year, run the following query:

     WITH per_day AS (
      SELECT
        time,
        value
      FROM kwh_day_by_day
      WHERE "time" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year'
      ORDER BY 1
     ), daily AS (
         SELECT
            to_char(time, 'Dy') as day,
            value
         FROM per_day
     ), percentile AS (
         SELECT
             day,
             approx_percentile(0.50, percentile_agg(value)) as value
         FROM daily
         GROUP BY 1
         ORDER BY 1
     )
     SELECT
         d.day,
         d.ordinal,
         pd.value
     FROM unnest(array['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']) WITH ORDINALITY AS d(day, ordinal)
     LEFT JOIN percentile pd ON lower(pd.day) = lower(d.day);
   

   You see something like:

   day	ordinal	value
   Mon	2	23.08078714975423
   Sun	1	19.511430831944395
   Tue	3	25.003118897837307
   Wed	4	8.09300571759772

Connect Grafana to Tiger Cloud

To visualize the results of your queries, enable Grafana to read the data in your service:

1. Log in to Grafana

   In your browser, log in to either:

   • Self-hosted Grafana: at http://localhost:3000/. The default credentials are admin, admin.
   • Grafana Cloud: use the URL and credentials you set when you created your account.

2. Add your service as a data source

   1. Open Connections > Data sources, then click Add new data source.

   2. Select PostgreSQL from the list.

   3. Configure the connection:

      • Host URL, Database name, Username, and Password

        Configure using your connection details. Host URL is in the format <host>:<port>.

      • TLS/SSL Mode: select require.

      • PostgreSQL options: enable TimescaleDB.

      • Leave the default setting for all other fields.

   4. Click Save & test.

   Grafana checks that your details are set correctly.

===== PAGE: https://docs.tigerdata.com/tutorials/energy-data/query-energy/ =====

Energy consumption data tutorial - query the data

When you have your dataset loaded, you can start constructing some queries to discover what your data tells you. This tutorial uses TimescaleDB hyperfunctions to construct queries that are not possible in standard Postgres.

In this section, you learn how to construct queries, to answer these questions:

• Energy consumption by hour of day
• Energy consumption by weekday.
• Energy consumption by month.

What is the energy consumption by the hour of the day?

When you have your database set up for energy consumption data, you can construct a query to find the median and the maximum consumption of energy on an hourly basis in a typical day.

Finding how many kilowatts of energy is consumed on an hourly basis

1. Connect to the Tiger Cloud service that contains the energy consumption dataset.

2. At the psql prompt, use the TimescaleDB Toolkit functionality to get calculate the fiftieth percentile or the median. Then calculate the maximum energy consumed using the standard Postgres max function:

   WITH per_hour AS (
   SELECT
   time,
   value
   FROM kwh_hour_by_hour
   WHERE "time" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year'
   ORDER BY 1
   ), hourly AS (
    SELECT
         extract(HOUR FROM time) * interval '1 hour' as hour,
         value
    FROM per_hour
   )
   SELECT
       hour,
       approx_percentile(0.50, percentile_agg(value)) as median,
       max(value) as maximum
   FROM hourly
   GROUP BY 1
   ORDER BY 1;
   

3. The data you get back looks a bit like this:

         hour   |       median       | maximum
       ----------+--------------------+---------
        00:00:00 | 0.5998949812512439 |     0.6
        01:00:00 | 0.5998949812512439 |     0.6
        02:00:00 | 0.5998949812512439 |     0.6
        03:00:00 | 1.6015944383271534 |     1.9
        04:00:00 | 2.5986701108275327 |     2.7
        05:00:00 | 1.4007385207185301 |     3.4
        06:00:00 | 0.5998949812512439 |     2.7
        07:00:00 | 0.6997720645753496 |     0.8
        08:00:00 | 0.6997720645753496 |     0.8
        09:00:00 | 0.6997720645753496 |     0.8
        10:00:00 | 0.9003240409125329 |     1.1
        11:00:00 | 0.8001143897618259 |     0.9
   

What is the energy consumption by the day of the week?

You can also check how energy consumption varies between weekends and weekdays.

Finding energy consumption during the weekdays

1. Connect to the Tiger Cloud service that contains the energy consumption dataset.

2. At the psql prompt, use this query to find difference in consumption during the weekdays and the weekends:

   WITH per_day AS (
    SELECT
      time,
      value
    FROM kwh_day_by_day
    WHERE "time" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year'
    ORDER BY 1
   ), daily AS (
       SELECT
          to_char(time, 'Dy') as day,
          value
       FROM per_day
   ), percentile AS (
       SELECT
           day,
           approx_percentile(0.50, percentile_agg(value)) as value
       FROM daily
       GROUP BY 1
       ORDER BY 1
   )
   SELECT
       d.day,
       d.ordinal,
       pd.value
   FROM unnest(array['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']) WITH ORDINALITY AS d(day, ordinal)
   LEFT JOIN percentile pd ON lower(pd.day) = lower(d.day);
   
   

3. The data you get back looks a bit like this:

       day | ordinal |       value
   -----+---------+--------------------
    Mon |       2 |  23.08078714975423
    Sun |       1 | 19.511430831944395
    Tue |       3 | 25.003118897837307
    Wed |       4 |   8.09300571759772
    Sat |       7 |
    Fri |       6 |
    Thu |       5 |
   

What is the energy consumption on a monthly basis?

You may also want to check the energy consumption that occurs on a monthly basis.

Finding energy consumption for each month of the year

1. Connect to the Tiger Cloud service that contains the energy consumption dataset.

2. At the psql prompt, use this query to find consumption for each month of the year:

    WITH per_day AS (
    SELECT
      time,
      value
    FROM kwh_day_by_day
    WHERE "time" > now() - interval '1 year'
    ORDER BY 1
   ), per_month AS (
      SELECT
         to_char(time, 'Mon') as month,
          sum(value) as value
      FROM per_day
     GROUP BY 1
   )
   SELECT
      m.month,
      m.ordinal,
      pd.value
   FROM unnest(array['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']) WITH ORDINALITY AS m(month, ordinal)
   LEFT JOIN per_month pd ON lower(pd.month) = lower(m.month)
   ORDER BY ordinal;
   

3. The data you get back looks a bit like this:

       month | ordinal |       value
       -------+---------+-------------------
       Jan   |       1 |
       Feb   |       2 |
       Mar   |       3 |
       Apr   |       4 |
       May   |       5 | 75.69999999999999
       Jun   |       6 |
       Jul   |       7 |
       Aug   |       8 |
       Sep   |       9 |
       Oct   |      10 |
       Nov   |      11 |
       Dec   |      12 |
   

4. To visualize this in Grafana, create a new panel, and select the Bar Chart visualization. Select the energy consumption dataset as your data source, and type the query from the previous step. In the Format as section, select Table.

5. Select a color scheme so that different consumptions are shown in different colors. In the options panel, under Standard options, change the Color scheme to a useful by value range.

   

===== PAGE: https://docs.tigerdata.com/tutorials/energy-data/index/ =====

Energy consumption data tutorial

When you are planning to switch to a rooftop solar system, it isn't easy, even with a specialist at hand. You need details of your power consumption, typical usage hours, distribution over a year, and other information. Collecting consumption data at the granularity of a few seconds and then getting insights on it is key - and this is what TimescaleDB is best at.

This tutorial uses energy consumption data from a typical household for over a year. You construct queries that look at how many watts were consumed, and when. Additionally, you can visualize the energy consumption data in Grafana.

Prerequisites

Before you begin, make sure you have:

• Signed up for a free Tiger Data account.
• Signed up for a Grafana account to graph queries.

Steps in this tutorial

This tutorial covers:

1. Setting up your dataset: Set up and connect to a Tiger Cloud service, and load data into the database using psql.
2. Querying your dataset: Analyze a dataset containing energy consumption data using Tiger Cloud and Postgres, and visualize the results in Grafana.
3. Bonus: Store data efficiently: Learn how to store and query your energy consumption data more efficiently using compression feature of Timescale.

About querying data with Timescale

This tutorial uses sample energy consumption data to show you how to construct queries for time-series data. The analysis you do in this tutorial is similar to the kind of analysis households might use to do things like plan their solar installation, or optimize their energy use over time.

It starts by teaching you how to set up and connect to a Tiger Cloud service, create tables, and load data into the tables using psql.

You then learn how to conduct analysis and monitoring on your dataset. It also walks you through the steps to visualize the results in Grafana.

===== PAGE: https://docs.tigerdata.com/tutorials/energy-data/compress-energy/ =====

Energy consumption data tutorial - set up compression

You have now seen how to create a hypertable for your energy consumption dataset and query it. When ingesting a dataset like this is seldom necessary to update old data and over time the amount of data in the tables grows. Over time you end up with a lot of data and since this is mostly immutable you can compress it to save space and avoid incurring additional cost.

It is possible to use disk-oriented compression like the support offered by ZFS and Btrfs but since TimescaleDB is build for handling event-oriented data (such as time-series) it comes with support for compressing data in hypertables.

TimescaleDB compression allows you to store the data in a vastly more efficient format allowing up to 20x compression ratio compared to a normal Postgres table, but this is of course highly dependent on the data and configuration.

TimescaleDB compression is implemented natively in Postgres and does not require special storage formats. Instead it relies on features of Postgres to transform the data into columnar format before compression. The use of a columnar format allows better compression ratio since similar data is stored adjacently. For more details on how the compression format looks, you can look at the compression design section.

A beneficial side-effect of compressing data is that certain queries are significantly faster since less data has to be read into memory.

Compression setup

1. Connect to the Tiger Cloud service that contains the energy dataset using, for example psql.

2. Enable compression on the table and pick suitable segment-by and order-by column using the ALTER TABLE command:

   ALTER TABLE metrics
   SET (
       timescaledb.compress,
       timescaledb.compress_segmentby='type_id',
       timescaledb.compress_orderby='created DESC'
   );
   

   Depending on the choice if segment-by and order-by column you can get very different performance and compression ratio. To learn more about how to pick the correct columns, see here.

3. You can manually compress all the chunks of the hypertable using compress_chunk in this manner:

   SELECT compress_chunk(c) from show_chunks('metrics') c;
   

   You can also automate compression by adding a compression policy which will be covered below.

4. Now that you have compressed the table you can compare the size of the dataset before and after compression:

   SELECT
       pg_size_pretty(before_compression_total_bytes) as before,
       pg_size_pretty(after_compression_total_bytes) as after
    FROM hypertable_compression_stats('metrics');
   

   This shows a significant improvement in data usage:

    before | after
   --------+-------
    180 MB | 16 MB
   (1 row)
   

Add a compression policy

To avoid running the compression step each time you have some data to compress you can set up a compression policy. The compression policy allows you to compress data that is older than a particular age, for example, to compress all chunks that are older than 8 days:

SELECT add_compression_policy('metrics', INTERVAL '8 days');

Compression policies run on a regular schedule, by default once every day, which means that you might have up to 9 days of uncompressed data with the setting above.

You can find more information on compression policies in the add_compression_policy section.

Taking advantage of query speedups

Previously, compression was set up to be segmented by type_id column value. This means fetching data by filtering or grouping on that column will be more efficient. Ordering is also set to created descending so if you run queries which try to order data with that ordering, you should see performance benefits.

For instance, if you run the query example from previous section:

SELECT time_bucket('1 day', created, 'Europe/Berlin') AS "time",
        round((last(value, created) - first(value, created)) *
100.) / 100. AS value
FROM metrics
WHERE type_id = 5
GROUP BY 1;

You should see a decent performance difference when the dataset is compressed and when is decompressed. Try it yourself by running the previous query, decompressing the dataset and running it again while timing the execution time. You can enable timing query times in psql by running:

    \timing

To decompress the whole dataset, run:

    SELECT decompress_chunk(c) from show_chunks('metrics') c;

On an example setup, speedup performance observed was an order of magnitude, 30 ms when compressed vs 360 ms when decompressed.

Try it yourself and see what you get!

===== PAGE: https://docs.tigerdata.com/tutorials/financial-ingest-real-time/financial-ingest-dataset/ =====

Ingest real-time financial websocket data - Set up the dataset

This tutorial uses a dataset that contains second-by-second stock-trade data for the top 100 most-traded symbols, in a hypertable named stocks_real_time. It also includes a separate table of company symbols and company names, in a regular Postgres table named company.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Connect to the websocket server

When you connect to the Twelve Data API through a websocket, you create a persistent connection between your computer and the websocket server. You set up a Python environment, and pass two arguments to create a websocket object and establish the connection.

Set up a new Python environment

Create a new Python virtual environment for this project and activate it. All the packages you need to complete for this tutorial are installed in this environment.

1. Create and activate a Python virtual environment:

   virtualenv env
   source env/bin/activate
   

2. Install the Twelve Data Python wrapper library with websocket support. This library allows you to make requests to the API and maintain a stable websocket connection.

   pip install twelvedata websocket-client
   

3. Install Psycopg2 so that you can connect the TimescaleDB from your Python script:

   pip install psycopg2-binary
   

Create the websocket connection

A persistent connection between your computer and the websocket server is used to receive data for as long as the connection is maintained. You need to pass two arguments to create a websocket object and establish connection.

Websocket arguments

• on_event

  This argument needs to be a function that is invoked whenever there's a new data record is received from the websocket:

  def on_event(event):
      print(event) # prints out the data record (dictionary)
  

  This is where you want to implement the ingestion logic so whenever there's new data available you insert it into the database.

• symbols

  This argument needs to be a list of stock ticker symbols (for example, MSFT) or crypto trading pairs (for example, BTC/USD). When using a websocket connection you always need to subscribe to the events you want to receive. You can do this by using the symbols argument or if your connection is already created you can also use the subscribe() function to get data for additional symbols.

Connect to the websocket server

1. Create a new Python file called websocket_test.py and connect to the Twelve Data servers using the <YOUR_API_KEY>:

      import time
      from twelvedata import TDClient
   
       messages_history = []
   
       def on_event(event):
        print(event) # prints out the data record (dictionary)
        messages_history.append(event)
   
      td = TDClient(apikey="<YOUR_API_KEY>")
      ws = td.websocket(symbols=["BTC/USD", "ETH/USD"], on_event=on_event)
      ws.subscribe(['ETH/BTC', 'AAPL'])
      ws.connect()
      while True:
      print('messages received: ', len(messages_history))
      ws.heartbeat()
      time.sleep(10)
   

2. Run the Python script:

   python websocket_test.py
   

3. When you run the script, you receive a response from the server about the status of your connection:

   {'event': 'subscribe-status',
    'status': 'ok',
    'success': [
           {'symbol': 'BTC/USD', 'exchange': 'Coinbase Pro', 'mic_code': 'Coinbase Pro', 'country': '', 'type': 'Digital Currency'},
           {'symbol': 'ETH/USD', 'exchange': 'Huobi', 'mic_code': 'Huobi', 'country': '', 'type': 'Digital Currency'}
       ],
    'fails': None
   }
   

   When you have established a connection to the websocket server, wait a few seconds, and you can see data records, like this:

   {'event': 'price', 'symbol': 'BTC/USD', 'currency_base': 'Bitcoin', 'currency_quote': 'US Dollar', 'exchange': 'Coinbase Pro', 'type': 'Digital Currency', 'timestamp': 1652438893, 'price': 30361.2, 'bid': 30361.2, 'ask': 30361.2, 'day_volume': 49153}
   {'event': 'price', 'symbol': 'BTC/USD', 'currency_base': 'Bitcoin', 'currency_quote': 'US Dollar', 'exchange': 'Coinbase Pro', 'type': 'Digital Currency', 'timestamp': 1652438896, 'price': 30380.6, 'bid': 30380.6, 'ask': 30380.6, 'day_volume': 49157}
   {'event': 'heartbeat', 'status': 'ok'}
   {'event': 'price', 'symbol': 'ETH/USD', 'currency_base': 'Ethereum', 'currency_quote': 'US Dollar', 'exchange': 'Huobi', 'type': 'Digital Currency', 'timestamp': 1652438899, 'price': 2089.07, 'bid': 2089.02, 'ask': 2089.03, 'day_volume': 193818}
   {'event': 'price', 'symbol': 'BTC/USD', 'currency_base': 'Bitcoin', 'currency_quote': 'US Dollar', 'exchange': 'Coinbase Pro', 'type': 'Digital Currency', 'timestamp': 1652438900, 'price': 30346.0, 'bid': 30346.0, 'ask': 30346.0, 'day_volume': 49167}
   

   Each price event gives you multiple data points about the given trading pair such as the name of the exchange, and the current price. You can also occasionally see heartbeat events in the response; these events signal the health of the connection over time. At this point the websocket connection is working successfully to pass data.

Optimize time-series data in a hypertable

Hypertables are Postgres tables in TimescaleDB that automatically partition your time-series data by time. Time-series data represents the way a system, process, or behavior changes over time. Hypertables enable TimescaleDB to work efficiently with time-series data. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table.

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

Because TimescaleDB is 100% Postgres, you can use all the standard Postgres tables, indexes, stored procedures, and other objects alongside your hypertables. This makes creating and working with hypertables similar to standard Postgres.

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Create a hypertable to store the real-time cryptocurrency data

   Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data:

   CREATE TABLE crypto_ticks (
       "time" TIMESTAMPTZ,
       symbol TEXT,
       price DOUBLE PRECISION,
       day_volume NUMERIC
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='time',
      tsdb.segmentby='symbol',
      tsdb.orderby='time DESC'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

Create a standard Postgres table for relational data

When you have relational data that enhances your time-series data, store that data in standard Postgres relational tables.

1. Add a table to store the asset symbol and name in a relational table

   CREATE TABLE crypto_assets (
       symbol TEXT UNIQUE,
       "name" TEXT
   );
   

You now have two tables within your Tiger Cloud service. A hypertable named crypto_ticks, and a normal Postgres table named crypto_assets.

When you ingest data into a transactional database like Timescale, it is more efficient to insert data in batches rather than inserting data row-by-row. Using one transaction to insert multiple rows can significantly increase the overall ingest capacity and speed of your Tiger Cloud service.

Batching in memory

A common practice to implement batching is to store new records in memory first, then after the batch reaches a certain size, insert all the records from memory into the database in one transaction. The perfect batch size isn't universal, but you can experiment with different batch sizes (for example, 100, 1000, 10000, and so on) and see which one fits your use case better. Using batching is a fairly common pattern when ingesting data into TimescaleDB from Kafka, Kinesis, or websocket connections.

To ingest the data into your Tiger Cloud service, you need to implement the on_event function.

After the websocket connection is set up, you can use the on_event function to ingest data into the database. This is a data pipeline that ingests real-time financial data into your Tiger Cloud service.

You can implement a batching solution in Python with Psycopg2. You can implement the ingestion logic within the on_event function that you can then pass over to the websocket object.

This function needs to:

1. Check if the item is a data item, and not websocket metadata.
2. Adjust the data so that it fits the database schema, including the data types, and order of columns.
3. Add it to the in-memory batch, which is a list in Python.
4. If the batch reaches a certain size, insert the data, and reset or empty the list.

Ingest data in real-time

1. Update the Python script that prints out the current batch size, so you can follow when data gets ingested from memory into your database. Use the <HOST>, <PASSWORD>, and <PORT> details for the Tiger Cloud service where you want to ingest the data and your API key from Twelve Data:

   import time
   import psycopg2
   
   from twelvedata import TDClient
   from psycopg2.extras import execute_values
   from datetime import datetime
   
   class WebsocketPipeline():
       DB_TABLE = "stocks_real_time"
   
       DB_COLUMNS=["time", "symbol", "price", "day_volume"]
   
       MAX_BATCH_SIZE=100
   
       def __init__(self, conn):
           """Connect to the Twelve Data web socket server and stream
           data into the database.
   
           Args:
               conn: psycopg2 connection object
           """
           self.conn = conn
           self.current_batch = []
           self.insert_counter = 0
   
       def _insert_values(self, data):
           if self.conn is not None:
               cursor = self.conn.cursor()
               sql = f"""
               INSERT INTO {self.DB_TABLE} ({','.join(self.DB_COLUMNS)})
               VALUES %s;"""
               execute_values(cursor, sql, data)
               self.conn.commit()
   
       def _on_event(self, event):
           """This function gets called whenever there's a new data record coming
           back from the server.
   
           Args:
               event (dict): data record
           """
           if event["event"] == "price":
               timestamp = datetime.utcfromtimestamp(event["timestamp"])
               data = (timestamp, event["symbol"], event["price"], event.get("day_volume"))
   
               self.current_batch.append(data)
               print(f"Current batch size: {len(self.current_batch)}")
   
               if len(self.current_batch) == self.MAX_BATCH_SIZE:
                   self._insert_values(self.current_batch)
                   self.insert_counter += 1
                   print(f"Batch insert #{self.insert_counter}")
                   self.current_batch = []
           def start(self, symbols):
               """Connect to the web socket server and start streaming real-time data
               into the database.
   
               Args:
                   symbols (list of symbols): List of stock/crypto symbols
               """
               td = TDClient(apikey="<YOUR_API_KEY")
               ws = td.websocket(on_event=self._on_event)
               ws.subscribe(symbols)
               ws.connect()
               while True:
                  ws.heartbeat()
                  time.sleep(10)
       onn = psycopg2.connect(database="tsdb",
                           host="<HOST>",
                           user="tsdbadmin",
                           password="<PASSWORD>",
                           port="<PORT>")
   
       symbols = ["BTC/USD", "ETH/USD", "MSFT", "AAPL"]
       websocket = WebsocketPipeline(conn)
       websocket.start(symbols=symbols)
       ```
   
   

2. Run the script:

   python websocket_test.py
   

You can even create separate Python scripts to start multiple websocket connections for different types of symbols, for example, one for stock, and another one for cryptocurrency prices.

Troubleshooting

If you see an error message similar to this:

2022-05-13 18:51:41,976 - ws-twelvedata - ERROR - TDWebSocket ERROR: Handshake status 200 OK

Then check that you use a proper API key received from Twelve Data.

Connect Grafana to Tiger Cloud

To visualize the results of your queries, enable Grafana to read the data in your service:

1. Log in to Grafana

   In your browser, log in to either:

   • Self-hosted Grafana: at http://localhost:3000/. The default credentials are admin, admin.
   • Grafana Cloud: use the URL and credentials you set when you created your account.

2. Add your service as a data source

   1. Open Connections > Data sources, then click Add new data source.

   2. Select PostgreSQL from the list.

   3. Configure the connection:

      • Host URL, Database name, Username, and Password

        Configure using your connection details. Host URL is in the format <host>:<port>.

      • TLS/SSL Mode: select require.

      • PostgreSQL options: enable TimescaleDB.

      • Leave the default setting for all other fields.

   4. Click Save & test.

   Grafana checks that your details are set correctly.

===== PAGE: https://docs.tigerdata.com/tutorials/financial-ingest-real-time/financial-ingest-query/ =====

Ingest real-time financial websocket data - Query the data

To look at OHLCV values, the most effective way is to create a continuous aggregate. You can create a continuous aggregate to aggregate data for each hour, then set the aggregate to refresh every hour, and aggregate the last two hours' worth of data.

Creating a continuous aggregate

1. Connect to the Tiger Cloud service tsdb that contains the Twelve Data stocks dataset.

2. At the psql prompt, create the continuous aggregate to aggregate data every minute:

   CREATE MATERIALIZED VIEW one_hour_candle
   WITH (timescaledb.continuous) AS
       SELECT
           time_bucket('1 hour', time) AS bucket,
           symbol,
           FIRST(price, time) AS "open",
           MAX(price) AS high,
           MIN(price) AS low,
           LAST(price, time) AS "close",
           LAST(day_volume, time) AS day_volume
       FROM crypto_ticks
       GROUP BY bucket, symbol;
   

   When you create the continuous aggregate, it refreshes by default.

3. Set a refresh policy to update the continuous aggregate every hour, if there is new data available in the hypertable for the last two hours:

   SELECT add_continuous_aggregate_policy('one_hour_candle',
       start_offset => INTERVAL '3 hours',
       end_offset => INTERVAL '1 hour',
       schedule_interval => INTERVAL '1 hour');
   

Query the continuous aggregate

When you have your continuous aggregate set up, you can query it to get the OHLCV values.

Querying the continuous aggregate

1. Connect to the Tiger Cloud service that contains the Twelve Data stocks dataset.

2. At the psql prompt, use this query to select all AAPL OHLCV data for the past 5 hours, by time bucket:

   SELECT * FROM one_hour_candle
   WHERE symbol = 'AAPL' AND bucket >= NOW() - INTERVAL '5 hours'
   ORDER BY bucket;
   

   The result of the query looks like this:

            bucket         | symbol  |  open   |  high   |   low   |  close  | day_volume
   ------------------------+---------+---------+---------+---------+---------+------------
    2023-05-30 08:00:00+00 | AAPL   | 176.31 | 176.31 |    176 | 176.01 |
    2023-05-30 08:01:00+00 | AAPL   | 176.27 | 176.27 | 176.02 |  176.2 |
    2023-05-30 08:06:00+00 | AAPL   | 176.03 | 176.04 | 175.95 |    176 |
    2023-05-30 08:07:00+00 | AAPL   | 175.95 |    176 | 175.82 | 175.91 |
    2023-05-30 08:08:00+00 | AAPL   | 175.92 | 176.02 |  175.8 | 176.02 |
    2023-05-30 08:09:00+00 | AAPL   | 176.02 | 176.02 |  175.9 | 175.98 |
    2023-05-30 08:10:00+00 | AAPL   | 175.98 | 175.98 | 175.94 | 175.94 |
    2023-05-30 08:11:00+00 | AAPL   | 175.94 | 175.94 | 175.91 | 175.91 |
    2023-05-30 08:12:00+00 | AAPL   |  175.9 | 175.94 |  175.9 | 175.94 |
   

Graph OHLCV data

When you have extracted the raw OHLCV data, you can use it to graph the result in a candlestick chart, using Grafana. To do this, you need to have Grafana set up to connect to your self-hosted TimescaleDB instance.

Graphing OHLCV data

1. Ensure you have Grafana installed, and you are using the TimescaleDB database that contains the Twelve Data dataset set up as a data source.

2. In Grafana, from the Dashboards menu, click New Dashboard. In the New Dashboard page, click Add a new panel.

3. In the Visualizations menu in the top right corner, select Candlestick from the list. Ensure you have set the Twelve Data dataset as your data source.

4. Click Edit SQL and paste in the query you used to get the OHLCV values.

5. In the Format as section, select Table.

6. Adjust elements of the table as required, and click Apply to save your graph to the dashboard.

   

===== PAGE: https://docs.tigerdata.com/tutorials/nyc-taxi-geospatial/dataset-nyc/ =====

Plot geospatial time-series data tutorial - set up dataset

This tutorial uses a dataset that contains historical data from the New York City Taxi and Limousine Commission NYC TLC, in a hypertable named rides. It also includes a separate tables of payment types and rates, in a regular Postgres table named payment_types, and rates.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Optimize time-series data in hypertables

Time-series data represents how a system, process, or behavior changes over time. Hypertables are Postgres tables that help you improve insert and query performance by automatically partitioning your data by time. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range.

Hypertables exist alongside regular Postgres tables. You interact with hypertables and regular Postgres tables in the same way. You use regular Postgres tables for relational data.

1. Create a hypertable to store the taxi trip data

   CREATE TABLE "rides"(
       vendor_id TEXT,
       pickup_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
       dropoff_datetime TIMESTAMP WITHOUT TIME ZONE NOT NULL,
       passenger_count NUMERIC,
       trip_distance NUMERIC,
       pickup_longitude  NUMERIC,
       pickup_latitude   NUMERIC,
       rate_code         INTEGER,
       dropoff_longitude NUMERIC,
       dropoff_latitude  NUMERIC,
       payment_type INTEGER,
       fare_amount NUMERIC,
       extra NUMERIC,
       mta_tax NUMERIC,
       tip_amount NUMERIC,
       tolls_amount NUMERIC,
       improvement_surcharge NUMERIC,
       total_amount NUMERIC
   ) WITH (
      tsdb.hypertable,
      tsdb.partition_column='pickup_datetime',
      tsdb.create_default_indexes=false
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

2. Add another dimension to partition your hypertable more efficiently

   SELECT add_dimension('rides', by_hash('payment_type', 2));
   

3. Create an index to support efficient queries

   Index by vendor, rate code, and passenger count:

   CREATE INDEX ON rides (vendor_id, pickup_datetime DESC);
   CREATE INDEX ON rides (rate_code, pickup_datetime DESC);
   CREATE INDEX ON rides (passenger_count, pickup_datetime DESC);
   

Create standard Postgres tables for relational data

When you have other relational data that enhances your time-series data, you can create standard Postgres tables just as you would normally. For this dataset, there are two other tables of data, called payment_types and rates.

1. Add a relational table to store the payment types data

   CREATE TABLE IF NOT EXISTS "payment_types"(
       payment_type INTEGER,
       description TEXT
   );
   INSERT INTO payment_types(payment_type, description) VALUES
   (1, 'credit card'),
   (2, 'cash'),
   (3, 'no charge'),
   (4, 'dispute'),
   (5, 'unknown'),
   (6, 'voided trip');
   

2. Add a relational table to store the rates data

   CREATE TABLE IF NOT EXISTS "rates"(
       rate_code   INTEGER,
       description TEXT
   );
   INSERT INTO rates(rate_code, description) VALUES
   (1, 'standard rate'),
   (2, 'JFK'),
   (3, 'Newark'),
   (4, 'Nassau or Westchester'),
   (5, 'negotiated fare'),
   (6, 'group ride');
   

You can confirm that the scripts were successful by running the \dt command in the psql command line. You should see this:

           List of relations
 Schema |     Name      | Type  |  Owner
--------+---------------+-------+----------
 public | payment_types | table | tsdbadmin
 public | rates         | table | tsdbadmin
 public | rides         | table | tsdbadmin
(3 rows)

Load trip data

When you have your database set up, you can load the taxi trip data into the rides hypertable.

This is a large dataset, so it might take a long time, depending on your network connection.

1. Download the dataset:

nyc_data.tar.gz

1. Use your file manager to decompress the downloaded dataset, and take a note of the path to the nyc_data_rides.csv file.

2. At the psql prompt, copy the data from the nyc_data_rides.csv file into your hypertable. Make sure you point to the correct path, if it is not in your current working directory:

   \COPY rides FROM nyc_data_rides.csv CSV;
   

You can check that the data has been copied successfully with this command:

SELECT * FROM rides LIMIT 5;

You should get five records that look like this:

-[ RECORD 1 ]---------+--------------------
vendor_id             | 1
pickup_datetime       | 2016-01-01 00:00:01
dropoff_datetime      | 2016-01-01 00:11:55
passenger_count       | 1
trip_distance         | 1.20
pickup_longitude      | -73.979423522949219
pickup_latitude       | 40.744613647460938
rate_code             | 1
dropoff_longitude     | -73.992034912109375
dropoff_latitude      | 40.753944396972656
payment_type          | 2
fare_amount           | 9
extra                 | 0.5
mta_tax               | 0.5
tip_amount            | 0
tolls_amount          | 0
improvement_surcharge | 0.3
total_amount          | 10.3

Connect Grafana to Tiger Cloud

To visualize the results of your queries, enable Grafana to read the data in your service:

1. Log in to Grafana

   In your browser, log in to either:

   • Self-hosted Grafana: at http://localhost:3000/. The default credentials are admin, admin.
   • Grafana Cloud: use the URL and credentials you set when you created your account.

2. Add your service as a data source

   1. Open Connections > Data sources, then click Add new data source.

   2. Select PostgreSQL from the list.

   3. Configure the connection:

      • Host URL, Database name, Username, and Password

        Configure using your connection details. Host URL is in the format <host>:<port>.

      • TLS/SSL Mode: select require.

      • PostgreSQL options: enable TimescaleDB.

      • Leave the default setting for all other fields.

   4. Click Save & test.

   Grafana checks that your details are set correctly.

===== PAGE: https://docs.tigerdata.com/tutorials/nyc-taxi-geospatial/index/ =====

Plot geospatial time-series data tutorial

New York City is home to about 9 million people. This tutorial uses historical data from New York's yellow taxi network, provided by the New York City Taxi and Limousine Commission NYC TLC. The NYC TLC tracks over 200,000 vehicles making about 1 million trips each day. Because nearly all of this data is time-series data, proper analysis requires a purpose-built time-series database, like Timescale.

In the beginner NYC taxis tutorial, you looked at constructing queries that looked at how many rides were taken, and when. The NYC taxi cab dataset also contains information about where each ride was picked up. This is geospatial data, and you can use a Postgres extension called PostGIS to examine where rides are originating from. Additionally, you can visualize the data in Grafana, by overlaying it on a map.

Prerequisites

Before you begin, make sure you have:

• Signed up for a free Tiger Data account.
• If you want to graph your queries, signed up for a Grafana account.

Steps in this tutorial

This tutorial covers:

1. Setting up your dataset: Set up and connect to a Timescale service, and load data into your database using psql.
2. Querying your dataset: Analyze a dataset containing NYC taxi trip data using Tiger Cloud and Postgres, and plot the results in Grafana.

About querying data with Timescale

This tutorial uses the NYC taxi data to show you how to construct queries for geospatial time-series data. The analysis you do in this tutorial is similar to the kind of analysis civic organizations do to plan new roads and public services.

It starts by teaching you how to set up and connect to a Tiger Cloud service, create tables, and load data into the tables using psql. If you have already completed the first NYC taxis tutorial, then you already have the dataset loaded, and you can skip straight to the queries.

You then learn how to conduct analysis and monitoring on your dataset. It walks you through using Postgres queries with the PostGIS extension to obtain information, and plotting the results in Grafana.

===== PAGE: https://docs.tigerdata.com/tutorials/nyc-taxi-geospatial/plot-nyc/ =====

Plot geospatial time-series data tutorial - query the data

When you have your dataset loaded, you can start constructing some queries to discover what your data tells you. In this section, you learn how to combine the data in the NYC taxi dataset with geospatial data from PostGIS, to answer these questions:

• How many rides on New Year's Day 2016 originated from Times Square?
• Which rides traveled more than 5 miles in Manhattan?.

Set up your dataset for PostGIS

To answer these geospatial questions, you need the ride count data from the NYC taxi dataset, but you also need some geospatial data to work out which trips originated where. TimescaleDB is compatible with all other Postgres extensions, so you can use the PostGIS extension to slice the data by time and location.

With the extension loaded, you alter your hypertable so it's ready for geospatial queries. The rides table contains columns for pickup latitude and longitude, but it needs to be converted into geometry coordinates so that it works well with PostGIS.

Setting up your dataset for PostGIS

1. Connect to the Tiger Cloud service that contains the NYC taxi dataset.

2. At the psql prompt, add the PostGIS extension:

   CREATE EXTENSION postgis;
   

   You can check that PostGIS is installed properly by checking that it appears in the extension list when you run the \dx command.

3. Alter the hypertable to add geometry columns for ride pick up and drop off locations:

   ALTER TABLE rides ADD COLUMN pickup_geom geometry(POINT,2163);
   ALTER TABLE rides ADD COLUMN dropoff_geom geometry(POINT,2163);
   

4. Convert the latitude and longitude points into geometry coordinates, so that they work well with PostGIS. This could take a while, as it needs to update all the data in both columns:

   UPDATE rides SET pickup_geom = ST_Transform(ST_SetSRID(ST_MakePoint(pickup_longitude,pickup_latitude),4326),2163),
      dropoff_geom = ST_Transform(ST_SetSRID(ST_MakePoint(dropoff_longitude,dropoff_latitude),4326),2163);
   

How many rides on New Year's Day 2016 originated from Times Square?

When you have your database set up for PostGIS data, you can construct a query to return the number of rides on New Year's Day that originated in Times Square, in 30-minute buckets.

Finding how many rides on New Year's Day 2016 originated from Times Square

Times Square is located at (40.7589,-73.9851).

1. Connect to the Tiger Cloud service that contains the NYC taxi dataset.

2. At the psql prompt, use this query to select all rides taken in the first day of January 2016 that picked up within 400m of Times Square, and return a count of rides for each 30 minute interval:

   SELECT time_bucket('30 minutes', pickup_datetime) AS thirty_min,
       COUNT(*) AS near_times_sq
   FROM rides
   WHERE ST_Distance(pickup_geom, ST_Transform(ST_SetSRID(ST_MakePoint(-73.9851,40.7589),4326),2163)) < 400
   AND pickup_datetime < '2016-01-01 14:00'
   GROUP BY thirty_min
   ORDER BY thirty_min;
   

3. The data you get back looks a bit like this:

        thirty_min      | near_times_sq
   ---------------------+---------------
    2016-01-01 00:00:00 |            74
    2016-01-01 00:30:00 |           102
    2016-01-01 01:00:00 |           120
    2016-01-01 01:30:00 |            98
    2016-01-01 02:00:00 |           112
   

Which rides traveled more than 5 miles in Manhattan?

This query is especially well suited to plot on a map. It looks at rides that were longer than 5 miles, within the city of Manhattan.

In this query, you want to return rides longer than 5 miles, but also include the distance, so that you can visualize longer distances with different visual treatments. The query also includes a WHERE clause to apply a geospatial boundary, looking for trips within 2 km of Times Square. Finally, in the GROUP BY clause, supply the trip_distance and location variables so that Grafana can plot the data properly.

Finding rides that traveled more than 5 miles in Manhattan

1. Connect to the Tiger Cloud service that contains the NYC taxi dataset.

2. At the psql prompt, use this query to find rides longer than 5 miles in Manhattan:

   SELECT time_bucket('5m', rides.pickup_datetime) AS time,
          rides.trip_distance AS value,
          rides.pickup_latitude AS latitude,
          rides.pickup_longitude AS longitude
   FROM rides
   WHERE rides.pickup_datetime BETWEEN '2016-01-01T01:41:55.986Z' AND '2016-01-01T07:41:55.986Z' AND
     ST_Distance(pickup_geom,
                 ST_Transform(ST_SetSRID(ST_MakePoint(-73.9851,40.7589),4326),2163)
     ) < 2000
   GROUP BY time,
            rides.trip_distance,
            rides.pickup_latitude,
            rides.pickup_longitude
   ORDER BY time
   LIMIT 500;
   

3. The data you get back looks a bit like this:

           time         | value |      latitude      |      longitude
   ---------------------+-------+--------------------+---------------------
    2016-01-01 01:40:00 |  0.00 | 40.752281188964844 | -73.975021362304688
    2016-01-01 01:40:00 |  0.09 | 40.755722045898437 | -73.967872619628906
    2016-01-01 01:40:00 |  0.15 | 40.752742767333984 | -73.977737426757813
    2016-01-01 01:40:00 |  0.15 | 40.756877899169922 | -73.969779968261719
    2016-01-01 01:40:00 |  0.18 | 40.756717681884766 | -73.967330932617188
    ...
   

4. To visualize this in Grafana, create a new panel, and select the Geomap visualization. Select the NYC taxis dataset as your data source, and type the query from the previous step. In the Format as section, select Table. Your world map now shows a dot over New York, zoom in to see the visualization.

5. To make this visualization more useful, change the way that the rides are displayed. In the options panel, under Data layer, add a layer called Distance traveled and select the markers option. In the Color section, select value. You can also adjust the symbol and size here.

6. Select a color scheme so that different ride lengths are shown in different colors. In the options panel, under Standard options, change the Color scheme to a useful by value range. This example uses the Blue-Yellow-Red (by value) option.

   

===== PAGE: https://docs.tigerdata.com/api/configuration/tiger-postgres/ =====

TimescaleDB configuration and tuning

Just as you can tune settings in Postgres, TimescaleDB provides a number of configuration settings that may be useful to your specific installation and performance needs. These can also be set within the postgresql.conf file or as command-line parameters when starting Postgres.

Query Planning and Execution

timescaledb.enable_chunkwise_aggregation (bool)

If enabled, aggregations are converted into partial aggregations during query planning. The first part of the aggregation is executed on a per-chunk basis. Then, these partial results are combined and finalized. Splitting aggregations decreases the size of the created hash tables and increases data locality, which speeds up queries.

timescaledb.vectorized_aggregation (bool)

Enables or disables the vectorized optimizations in the query executor. For example, the sum() aggregation function on compressed chunks can be optimized in this way.

timescaledb.enable_merge_on_cagg_refresh (bool)

Set to ON to dramatically decrease the amount of data written on a continuous aggregate in the presence of a small number of changes, reduce the i/o cost of refreshing a continuous aggregate, and generate fewer Write-Ahead Logs (WAL). Only works for continuous aggregates that don't have compression enabled.

Please refer to the Grand Unified Configuration (GUC) parameters for a complete list.

Policies

timescaledb.max_background_workers (int)

Max background worker processes allocated to TimescaleDB. Set to at least 1 + the number of databases loaded with the TimescaleDB extension in a Postgres instance. Default value is 16.

Tiger Cloud service tuning

timescaledb.disable_load (bool)

Disable the loading of the actual extension

Administration

timescaledb.restoring (bool)

Set TimescaleDB in restoring mode. It is disabled by default.

timescaledb.license (string)

Change access to features based on the TimescaleDB license in use. For example, setting timescaledb.license to apache limits TimescaleDB to features that are implemented under the Apache 2 license. The default value is timescale, which allows access to all features.

timescaledb.telemetry_level (enum)

Telemetry settings level. Level used to determine which telemetry to send. Can be set to off or basic. Defaults to basic.

timescaledb.last_tuned (string)

Records last time timescaledb-tune ran.

timescaledb.last_tuned_version (string)

Version of timescaledb-tune used to tune when it runs.

===== PAGE: https://docs.tigerdata.com/api/configuration/gucs/ =====

Grand Unified Configuration (GUC) parameters

You use the following Grand Unified Configuration (GUC) parameters to optimize the behavior of your Tiger Cloud service.

The namespace of each GUC is timescaledb. To set a GUC you specify <namespace>.<GUC name>. For example:

SET timescaledb.enable_tiered_reads = true;

Name	Type	Default	Description
GUC_CAGG_HIGH_WORK_MEM_NAME	INTEGER	GUC_CAGG_HIGH_WORK_MEM_VALUE	The high working memory limit for the continuous aggregate invalidation processing. min: 64, max: MAX_KILOBYTES
GUC_CAGG_LOW_WORK_MEM_NAME	INTEGER	GUC_CAGG_LOW_WORK_MEM_VALUE	The low working memory limit for the continuous aggregate invalidation processing. min: 64, max: MAX_KILOBYTES
auto_sparse_indexes	BOOLEAN	true	The hypertable columns that are used as index keys will have suitable sparse indexes when compressed. Must be set at the moment of chunk compression, e.g. when the compress_chunk() is called.
bgw_log_level	ENUM	WARNING	Log level for the scheduler and workers of the background worker subsystem. Requires configuration reload to change.
cagg_processing_wal_batch_size	INTEGER	10000	Number of entries processed from the WAL at a go. Larger values take more memory but might be more efficient. min: 1000, max: 10000000
compress_truncate_behaviour	ENUM	COMPRESS_TRUNCATE_ONLY	Defines how truncate behaves at the end of compression. 'truncate_only' forces truncation. 'truncate_disabled' deletes rows instead of truncate. 'truncate_or_delete' allows falling back to deletion.
compression_batch_size_limit	INTEGER	1000	Setting this option to a number between 1 and 999 will force compression to limit the size of compressed batches to that amount of uncompressed tuples.Setting this to 0 defaults to the max batch size of 1000. min: 1, max: 1000
compression_orderby_default_function	STRING	"_timescaledb_functions.get_orderby_defaults"	Function to use for calculating default order_by setting for compression
compression_segmentby_default_function	STRING	"_timescaledb_functions.get_segmentby_defaults"	Function to use for calculating default segment_by setting for compression
current_timestamp_mock	STRING	NULL	this is for debugging purposes
debug_allow_cagg_with_deprecated_funcs	BOOLEAN	false	this is for debugging/testing purposes
debug_bgw_scheduler_exit_status	INTEGER	0	this is for debugging purposes min: 0, max: 255
debug_compression_path_info	BOOLEAN	false	this is for debugging/information purposes
debug_have_int128	BOOLEAN	#ifdef HAVE_INT128 true	this is for debugging purposes
debug_require_batch_sorted_merge	ENUM	DRO_Allow	this is for debugging purposes
debug_require_vector_agg	ENUM	DRO_Allow	this is for debugging purposes
debug_require_vector_qual	ENUM	DRO_Allow	this is for debugging purposes, to let us check if the vectorized quals are used or not. EXPLAIN differs after PG15 for custom nodes, and using the test templates is a pain
debug_skip_scan_info	BOOLEAN	false	Print debug info about SkipScan distinct columns
debug_toast_tuple_target	INTEGER	/* bootValue = */ 128	this is for debugging purposes min: /* minValue = */ 1, max: /* maxValue = */ 65535
enable_bool_compression	BOOLEAN	true	Enable bool compression
enable_bulk_decompression	BOOLEAN	true	Increases throughput of decompression, but might increase query memory usage
enable_cagg_reorder_groupby	BOOLEAN	true	Enable group by clause reordering for continuous aggregates
enable_cagg_sort_pushdown	BOOLEAN	true	Enable pushdown of ORDER BY clause for continuous aggregates
enable_cagg_watermark_constify	BOOLEAN	true	Enable constifying cagg watermark for real-time caggs
enable_cagg_window_functions	BOOLEAN	false	Allow window functions in continuous aggregate views
enable_chunk_append	BOOLEAN	true	Enable using chunk append node
enable_chunk_skipping	BOOLEAN	false	Enable using chunk column stats to filter chunks based on column filters
enable_chunkwise_aggregation	BOOLEAN	true	Enable the pushdown of aggregations to the chunk level
enable_columnarscan	BOOLEAN	true	A columnar scan replaces sequence scans for columnar-oriented storage and enables storage-specific optimizations like vectorized filters. Disabling columnar scan will make PostgreSQL fall back to regular sequence scans.
enable_compressed_direct_batch_delete	BOOLEAN	true	Enable direct batch deletion in compressed chunks
enable_compressed_skipscan	BOOLEAN	true	Enable SkipScan for distinct inputs over compressed chunks
enable_compression_indexscan	BOOLEAN	false	Enable indexscan during compression, if matching index is found
enable_compression_ratio_warnings	BOOLEAN	true	Enable warnings for poor compression ratio
enable_compression_wal_markers	BOOLEAN	true	Enable the generation of markers in the WAL stream which mark the start and end of compression operations
enable_compressor_batch_limit	BOOLEAN	false	Enable compressor batch limit for compressors which can go over the allocation limit (1 GB). This feature willlimit those compressors by reducing the size of the batch and thus avoid hitting the limit.
enable_constraint_aware_append	BOOLEAN	true	Enable constraint exclusion at execution time
enable_constraint_exclusion	BOOLEAN	true	Enable planner constraint exclusion
enable_custom_hashagg	BOOLEAN	false	Enable creating custom hash aggregation plans
enable_decompression_sorted_merge	BOOLEAN	true	Enable the merge of compressed batches to preserve the compression order by
enable_delete_after_compression	BOOLEAN	false	Delete all rows after compression instead of truncate
enable_deprecation_warnings	BOOLEAN	true	Enable warnings when using deprecated functionality
enable_direct_compress_copy	BOOLEAN	false	Enable experimental support for direct compression during COPY
enable_direct_compress_copy_client_sorted	BOOLEAN	false	Correct handling of data sorting by the user is required for this option.
enable_direct_compress_copy_sort_batches	BOOLEAN	true	Enable batch sorting during direct compress COPY
enable_dml_decompression	BOOLEAN	true	Enable DML decompression when modifying compressed hypertable
enable_dml_decompression_tuple_filtering	BOOLEAN	true	Recheck tuples during DML decompression to only decompress batches with matching tuples
enable_event_triggers	BOOLEAN	false	Enable event triggers for chunks creation
enable_exclusive_locking_recompression	BOOLEAN	false	Enable getting exclusive lock on chunk during segmentwise recompression
enable_foreign_key_propagation	BOOLEAN	true	Adjust foreign key lookup queries to target whole hypertable
enable_job_execution_logging	BOOLEAN	false	Retain job run status in logging table
enable_merge_on_cagg_refresh	BOOLEAN	false	Enable MERGE statement on cagg refresh
enable_multikey_skipscan	BOOLEAN	true	Enable SkipScan for multiple distinct inputs
enable_now_constify	BOOLEAN	true	Enable constifying now() in query constraints
enable_null_compression	BOOLEAN	true	Enable null compression
enable_optimizations	BOOLEAN	true	Enable TimescaleDB query optimizations
enable_ordered_append	BOOLEAN	true	Enable ordered append optimization for queries that are ordered by the time dimension
enable_parallel_chunk_append	BOOLEAN	true	Enable using parallel aware chunk append node
enable_qual_propagation	BOOLEAN	true	Enable propagation of qualifiers in JOINs
enable_rowlevel_compression_locking	BOOLEAN	false	Use only if you know what you are doing
enable_runtime_exclusion	BOOLEAN	true	Enable runtime chunk exclusion in ChunkAppend node
enable_segmentwise_recompression	BOOLEAN	true	Enable segmentwise recompression
enable_skipscan	BOOLEAN	true	Enable SkipScan for DISTINCT queries
enable_skipscan_for_distinct_aggregates	BOOLEAN	true	Enable SkipScan for DISTINCT aggregates
enable_sparse_index_bloom	BOOLEAN	true	This sparse index speeds up the equality queries on compressed columns, and can be disabled when not desired.
enable_tiered_reads	BOOLEAN	true	Enable reading of tiered data by including a foreign table representing the data in the object storage into the query plan
enable_transparent_decompression	BOOLEAN	true	Enable transparent decompression when querying hypertable
enable_tss_callbacks	BOOLEAN	true	Enable ts_stat_statements callbacks
enable_uuid_compression	BOOLEAN	false	Enable uuid compression
enable_vectorized_aggregation	BOOLEAN	true	Enable vectorized aggregation for compressed data
last_tuned	STRING	NULL	records last time timescaledb-tune ran
last_tuned_version	STRING	NULL	version of timescaledb-tune used to tune
license	STRING	TS_LICENSE_DEFAULT	Determines which features are enabled
materializations_per_refresh_window	INTEGER	10	The maximal number of individual refreshes per cagg refresh. If more refreshes need to be performed, they are merged into a larger single refresh. min: 0, max: INT_MAX
max_cached_chunks_per_hypertable	INTEGER	1024	Maximum number of chunks stored in the cache min: 0, max: 65536
max_open_chunks_per_insert	INTEGER	1024	Maximum number of open chunk tables per insert min: 0, max: PG_INT16_MAX
max_tuples_decompressed_per_dml_transaction	INTEGER	100000	If the number of tuples exceeds this value, an error will be thrown and transaction rolled back. Setting this to 0 sets this value to unlimited number of tuples decompressed. min: 0, max: 2147483647
restoring	BOOLEAN	false	In restoring mode all timescaledb internal hooks are disabled. This mode is required for restoring logical dumps of databases with timescaledb.
shutdown_bgw_scheduler	BOOLEAN	false	this is for debugging purposes
skip_scan_run_cost_multiplier	REAL	1.0	Default is 1.0 i.e. regularly estimated SkipScan run cost, 0.0 will make SkipScan to have run cost = 0 min: 0.0, max: 1.0
telemetry_level	ENUM	TELEMETRY_DEFAULT	Level used to determine which telemetry to send

Version: 2.22.1

===== PAGE: https://docs.tigerdata.com/api/uuid-functions/uuid_timestamp/ =====

uuid_timestamp()

Extract a Postgres timestamp with time zone from a UUIDv7 object.

uuid contains a millisecond unix timestamp and an optional sub-millisecond fraction. This fraction is used to construct the Postgres timestamp.

To include the sub-millisecond fraction in the returned timestamp, call uuid_timestamp_micros.

Samples

postgres=# SELECT uuid_timestamp('019913ce-f124-7835-96c7-a2df691caa98');

Returns something like:

uuid_timestamp
----------------------------
 2025-09-04 10:19:13.316+02

Arguments

Name	Type	Default	Required	Description
uuid	UUID	-	✔	The UUID object to extract the timestamp from

===== PAGE: https://docs.tigerdata.com/api/uuid-functions/uuid_version/ =====

uuid_version()

Extract the version number from a UUID object:

Samples

postgres=# SELECT uuid_version('019913ce-f124-7835-96c7-a2df691caa98');

Returns something like:

 uuid_version
--------------
            7

Arguments

Name	Type	Default	Required	Description
uuid	UUID	-	✔	The UUID object to extract the version number from

===== PAGE: https://docs.tigerdata.com/api/uuid-functions/generate_uuidv7/ =====

generate_uuidv7()

Generate a UUIDv7 object based on the current time.

The UUID contains a a UNIX timestamp split into millisecond and sub-millisecond parts, followed by random bits.

You can use this function to generate a time-ordered series of UUIDs suitable for use in a time-partitioned column in TimescaleDB.

Samples

• Generate a UUIDv7 object based on the current time

  postgres=# SELECT generate_uuidv7();
             generate_uuidv7
  --------------------------------------
   019913ce-f124-7835-96c7-a2df691caa98
  

• Insert a generated UUIDv7 object

  INSERT INTO alerts VALUES (generate_uuidv7(), 'high CPU');
  

===== PAGE: https://docs.tigerdata.com/api/uuid-functions/to_uuidv7/ =====

to_uuidv7()

Create a UUIDv7 object from a Postgres timestamp and random bits.

ts is converted to a UNIX timestamp split into millisecond and sub-millisecond parts.

Samples

SELECT to_uuidv7(ts)
FROM generate_series('2025-01-01:00:00:00'::timestamptz, '2025-01-01:00:00:03'::timestamptz, '1 microsecond'::interval) ts;

Arguments

Name	Type	Default	Required	Description
ts	TIMESTAMPTZ	-	✔	The timestamp used to return a UUIDv7 object

===== PAGE: https://docs.tigerdata.com/api/uuid-functions/uuid_timestamp_micros/ =====

uuid_timestamp_micros()

Extract a Postgres timestamp with time zone from a UUIDv7 object. uuid contains a millisecond unix timestamp and an optional sub-millisecond fraction.

Unlike uuid_timestamp, the microsecond part of uuid is used to construct a Postgres timestamp with microsecond precision.

Unless uuid is known to encode a valid sub-millisecond fraction, use uuid_timestamp.

Samples

postgres=# SELECT uuid_timestamp_micros('019913ce-f124-7835-96c7-a2df691caa98');

Returns something like:

uuid_timestamp_micros
-------------------------------
 2025-09-04 10:19:13.316512+02

Arguments

Name	Type	Default	Required	Description
uuid	UUID	-	✔	The UUID object to extract the timestamp from

===== PAGE: https://docs.tigerdata.com/api/uuid-functions/to_uuidv7_boundary/ =====

to_uuidv7_boundary()

Create a UUIDv7 object from a Postgres timestamp for use in range queries.

ts is converted to a UNIX timestamp split into millisecond and sub-millisecond parts.

The random bits of the UUID are set to zero in order to create a "lower" boundary UUID.

For example, you can use the returned UUIDvs to find all rows with UUIDs where the timestamp is less than the boundary UUID's timestamp.

Samples

• Create a boundary UUID from a timestamp:

  postgres=# SELECT to_uuidv7_boundary('2025-09-04 11:01');
  

  Returns something like:

            to_uuidv7_boundary
  --------------------------------------
   019913f5-30e0-7000-8000-000000000000
  

• Use a boundary UUID to find all UUIDs with a timestamp below '2025-09-04 10:00':

  SELECT * FROM uuid_events WHERE event_id < to_uuidv7_boundary('2025-09-04 10:00');
  

Arguments

Name	Type	Default	Required	Description
ts	TIMESTAMPTZ	-	✔	The timestamp used to return a UUIDv7 object

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/cleanup_copy_chunk_operation_experimental/ =====

cleanup_copy_chunk_operation()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

You can copy or move a chunk to a new location within a multi-node environment. The operation happens over multiple transactions so, if it fails, it is manually cleaned up using this function. Without cleanup, the failed operation might hold a replication slot open, which in turn prevents storage from being reclaimed. The operation ID is logged in case of a failed copy or move operation and is required as input to the cleanup function.

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

Required arguments

Name	Type	Description
operation_id	NAME	ID of the failed operation

Sample usage

Clean up a failed operation:

CALL timescaledb_experimental.cleanup_copy_chunk_operation('ts_copy_1_31');

Get a list of running copy or move operations:

SELECT * FROM _timescaledb_catalog.chunk_copy_operation;

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/create_distributed_restore_point/ =====

create_distributed_restore_point()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Creates a same-named marker record, for example restore point, in the write-ahead logs of all nodes in a multi-node TimescaleDB cluster.

The restore point can be used as a recovery target on each node, ensuring the entire multi-node cluster can be restored to a consistent state. The function returns the write-ahead log locations for all nodes where the marker record was written.

This function is similar to the Postgres function pg_create_restore_point, but it has been modified to work with a distributed database.

This function can only be run on the access node, and requires superuser privileges.

Required arguments

Name	Description
name	The restore point name

Returns

Column	Type	Description
node_name	NAME	Node name, or NULL for access node
node_type	TEXT	Node type name: access_node or data_node
restore_point	PG_LSN	Restore point log sequence number

Errors

An error is given if:

• The restore point name is more than 64 characters
• A recovery is in progress
• The current WAL level is not set to replica or logical
• The current user is not a superuser
• The current server is not the access node
• TimescaleDB's 2PC transactions are not enabled

Sample usage

This example create a restore point called pitr across three data nodes and the access node:

SELECT * FROM create_distributed_restore_point('pitr');
 node_name |  node_type  | restore_point
-----------+-------------+---------------
           | access_node | 0/3694A30
 dn1       | data_node   | 0/3694A98
 dn2       | data_node   | 0/3694B00
 dn3       | data_node   | 0/3694B68
(4 rows)

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/copy_chunk_experimental/ =====

copy_chunk()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

TimescaleDB allows you to copy existing chunks to a new location within a multi-node environment. This allows each data node to work both as a primary for some chunks and backup for others. If a data node fails, its chunks already exist on other nodes that can take over the responsibility of serving them.

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

Required arguments

Name	Type	Description
chunk	REGCLASS	Name of chunk to be copied
source_node	NAME	Data node where the chunk currently resides
destination_node	NAME	Data node where the chunk is to be copied

Required settings

When copying a chunk, the destination data node needs a way to authenticate with the data node that holds the source chunk. It is currently recommended to use a password file on the data node.

The wal_level setting must also be set to logical or higher on data nodes from which chunks are copied. If you are copying or moving many chunks in parallel, you can increase max_wal_senders and max_replication_slots.

Failures

When a copy operation fails, it sometimes creates objects and metadata on the destination data node. It can also hold a replication slot open on the source data node. To clean up these objects and metadata, use cleanup_copy_chunk_operation.

Sample usage

CALL timescaledb_experimental.copy_chunk('_timescaledb_internal._dist_hyper_1_1_chunk', 'data_node_2', 'data_node_3');

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/alter_data_node/ =====

alter_data_node()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Change the configuration of a data node that was originally set up with add_data_node on the access node.

Only users with certain privileges can alter data nodes. When you alter the connection details for a data node, make sure that the altered configuration is reachable and can be authenticated by the access node.

Required arguments

Name	Description
node_name	Name for the data node

Optional arguments

Name	Description
host	Host name for the remote data node
database	Database name where remote hypertables are created. The default is the database name that was provided in add_data_node
port	Port to use on the remote data node. The default is the Postgres port that was provided in add_data_node
available	Configure availability of the remote data node. The default is true meaning that the data node is available for read/write queries

Returns

Column	Description
node_name	Local name to use for the data node
host	Host name for the remote data node
port	Port for the remote data node
database	Database name used on the remote data node
available	Availability of the remote data node for read/write queries

Errors

An error is given if:

• A remote data node with the provided node_name argument does not exist.

Privileges

To alter a data node, you must have the correct permissions, or be the owner of the remote server. Additionally, you must have the USAGE privilege on the timescaledb_fdw foreign data wrapper.

Sample usage

To change the port number and host information for an existing data node dn1:

SELECT alter_data_node('dn1', host => 'dn1.example.com', port => 6999);

Data nodes are available for read/write queries by default. If the data node becomes unavailable for some reason, the read/write query gives an error. This API provides an optional argument, available, to mark an existing data node as available or unavailable for read/write queries. By marking a data node as unavailable you can allow read/write queries to proceed in the cluster. For more information, see the multi-node HA section

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/move_chunk_experimental/ =====

move_chunk()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

TimescaleDB allows you to move chunks to other data nodes. Moving chunks is useful in order to rebalance a multi-node cluster or remove a data node from the cluster.

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

Required arguments

Name	Type	Description
chunk	REGCLASS	Name of chunk to be copied
source_node	NAME	Data node where the chunk currently resides
destination_node	NAME	Data node where the chunk is to be copied

Required settings

When moving a chunk, the destination data node needs a way to authenticate with the data node that holds the source chunk. It is currently recommended to use a password file on the data node.

The wal_level setting must also be set to logical or higher on data nodes from which chunks are moved. If you are copying or moving many chunks in parallel, you can increase max_wal_senders and max_replication_slots.

Failures

When a move operation fails, it sometimes creates objects and metadata on the destination data node. It can also hold a replication slot open on the source data node. To clean up these objects and metadata, use cleanup_copy_chunk_operation.

Sample usage

CALL timescaledb_experimental.move_chunk('_timescaledb_internal._dist_hyper_1_1_chunk', 'data_node_2', 'data_node_3');

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/distributed_exec/ =====

distributed_exec()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

This procedure is used on an access node to execute a SQL command across the data nodes of a distributed database. For instance, one use case is to create the roles and permissions needed in a distributed database.

The procedure can run distributed commands transactionally, so a command is executed either everywhere or nowhere. However, not all SQL commands can run in a transaction. This can be toggled with the argument transactional. Note if the execution is not transactional, a failure on one of the data node requires manual dealing with any introduced inconsistency.

Note that the command is not executed on the access node itself and it is not possible to chain multiple commands together in one call.

You cannot run distributed_exec with some SQL commands. For example, ALTER EXTENSION doesn't work because it can't be called after the TimescaleDB extension is already loaded.

Required arguments

Name	Type	Description
query	TEXT	The command to execute on data nodes.

Optional arguments

Name	Type	Description
node_list	ARRAY	An array of data nodes where the command should be executed. Defaults to all data nodes if not specified.
transactional	BOOLEAN	Allows to specify if the execution of the statement should be transactional or not. Defaults to TRUE.

Sample usage

Create the role testrole across all data nodes in a distributed database:

CALL distributed_exec($$ CREATE USER testrole WITH LOGIN $$);

Create the role testrole on two specific data nodes:

CALL distributed_exec($$ CREATE USER testrole WITH LOGIN $$, node_list => '{ "dn1", "dn2" }');

Create the table example on all data nodes:

CALL distributed_exec($$ CREATE TABLE example (ts TIMESTAMPTZ, value INTEGER) $$);

Create new databases dist_database on data nodes, which requires setting transactional to FALSE:

CALL distributed_exec('CREATE DATABASE dist_database', transactional => FALSE);

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/create_distributed_hypertable/ =====

create_distributed_hypertable()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Create a TimescaleDB hypertable distributed across a multinode environment.

create_distributed_hypertable() replaces create_hypertable() (old interface). Distributed tables use the old API. The new generalized create_hypertable API was introduced in TimescaleDB v2.13.

Required arguments

Name	Type	Description
relation	REGCLASS	Identifier of the table you want to convert to a hypertable.
time_column_name	TEXT	Name of the column that contains time values, as well as the primary column to partition by.

Optional arguments

Name	Type	Description
partitioning_column	TEXT	Name of an additional column to partition by.
number_partitions	INTEGER	Number of hash partitions to use for partitioning_column. Must be > 0. Default is the number of data_nodes.
associated_schema_name	TEXT	Name of the schema for internal hypertable tables. Default is _timescaledb_internal.
associated_table_prefix	TEXT	Prefix for internal hypertable chunk names. Default is _hyper.
chunk_time_interval	INTERVAL	Interval in event time that each chunk covers. Must be > 0. Default is 7 days.
create_default_indexes	BOOLEAN	Boolean whether to create default indexes on time/partitioning columns. Default is TRUE.
if_not_exists	BOOLEAN	Boolean whether to print warning if table already converted to hypertable or raise exception. Default is FALSE.
partitioning_func	REGCLASS	The function to use for calculating a value's partition.
migrate_data	BOOLEAN	Set to TRUE to migrate any existing data from the relation table to chunks in the new hypertable. A non-empty table generates an error without this option. Large tables may take significant time to migrate. Default is FALSE.
time_partitioning_func	REGCLASS	Function to convert incompatible primary time column values to compatible ones. The function must be IMMUTABLE.
replication_factor	INTEGER	The number of data nodes to which the same data is written to. This is done by creating chunk copies on this amount of data nodes. Must be >= 1; If not set, the default value is determined by the timescaledb.hypertable_replication_factor_default GUC. Read the best practices before changing the default.
data_nodes	ARRAY	The set of data nodes used for the distributed hypertable. If not present, defaults to all data nodes known by the access node (the node on which the distributed hypertable is created).

Returns

Column	Type	Description
hypertable_id	INTEGER	ID of the hypertable in TimescaleDB.
schema_name	TEXT	Schema name of the table converted to hypertable.
table_name	TEXT	Table name of the table converted to hypertable.
created	BOOLEAN	TRUE if the hypertable was created, FALSE when if_not_exists is TRUE and no hypertable was created.

Sample usage

Create a table conditions which is partitioned across data nodes by the 'location' column. Note that the number of space partitions is automatically equal to the number of data nodes assigned to this hypertable (all configured data nodes in this case, as data_nodes is not specified).

SELECT create_distributed_hypertable('conditions', 'time', 'location');

Create a table conditions using a specific set of data nodes.

SELECT create_distributed_hypertable('conditions', 'time', 'location',
    data_nodes => '{ "data_node_1", "data_node_2", "data_node_4", "data_node_7" }');

Best practices

• Hash partitions: Best practice for distributed hypertables is to enable hash partitions. With hash partitions, incoming data is divided between the data nodes. Without hash partition, all data for each time slice is written to a single data node.

• Time intervals: Follow the guidelines for chunk_time_interval defined in [create_hypertable] create-hypertable-old.

  When you enable hash partitioning, the hypertable is evenly distributed across the data nodes. This means you can set a larger time interval. For example, you ingest 10 GB of data per day shared over five data nodes, each node has 64 GB of memory. If this is the only table being served by these data nodes, use a time interval of 1 week:

   7 days * 10 GB             70
   --------------------  ==  ---  ~= 22% of main memory used for the most recent chunks
   5 data nodes * 64 GB      320
  

  If you do not enable hash partitioning, use the same chunk_time_interval settings as a non-distributed instance. This is because all incoming data is handled by a single node.

• Replication factor: replication_factor defines the number of data nodes a newly created chunk is replicated in. For example, when you set replication_factor to 3, each chunk exists on 3 separate data nodes. Rows written to a chunk are inserted into all data notes in a two-phase commit protocol.

  If a data node fails or is removed, no data is lost. Writes succeed on the other data nodes. However, the chunks on the lost data node are now under-replicated. When the failed data node becomes available, rebalance the chunks with a call to copy_chunk.

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/attach_data_node/ =====

attach_data_node()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Attach a data node to a hypertable. The data node should have been previously created using add_data_node.

When a distributed hypertable is created, by default it uses all available data nodes for the hypertable, but if a data node is added after a hypertable is created, the data node is not automatically used by existing distributed hypertables.

If you want a hypertable to use a data node that was created later, you must attach the data node to the hypertable using this function.

Required arguments

Name	Description
node_name	Name of data node to attach
hypertable	Name of distributed hypertable to attach node to

Optional arguments

Name	Description
if_not_attached	Prevents error if the data node is already attached to the hypertable. A notice is printed that the data node is attached. Defaults to FALSE.
repartition	Change the partitioning configuration so that all the attached data nodes are used. Defaults to TRUE.

Returns

Column	Description
hypertable_id	Hypertable id of the modified hypertable
node_hypertable_id	Hypertable id on the remote data node
node_name	Name of the attached data node

Sample usage

Attach a data node dn3 to a distributed hypertable conditions previously created with create_distributed_hypertable.

SELECT * FROM attach_data_node('dn3','conditions');

hypertable_id | node_hypertable_id |  node_name
--------------+--------------------+-------------
            5 |                  3 | dn3

(1 row)

You must add a data node to your distributed database first with add_data_node first before attaching it.

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/set_number_partitions/ =====

set_number_partitions()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Sets the number of partitions (slices) of a space dimension on a hypertable. The new partitioning only affects new chunks.

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable to update the number of partitions for.
number_partitions	INTEGER	The new number of partitions for the dimension. Must be greater than 0 and less than 32,768.

Optional arguments

Name	Type	Description
dimension_name	REGCLASS	The name of the space dimension to set the number of partitions for.

The dimension_name needs to be explicitly specified only if the hypertable has more than one space dimension. An error is thrown otherwise.

Sample usage

For a table with a single space dimension:

SELECT set_number_partitions('conditions', 2);

For a table with more than one space dimension:

SELECT set_number_partitions('conditions', 2, 'device_id');

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/add_data_node/ =====

add_data_node()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Add a new data node on the access node to be used by distributed hypertables. The data node is automatically used by distributed hypertables that are created after the data node has been added, while existing distributed hypertables require an additional attach_data_node.

If the data node already exists, the command aborts with either an error or a notice depending on the value of if_not_exists.

For security purposes, only superusers or users with necessary privileges can add data nodes (see below for details). When adding a data node, the access node also tries to connect to the data node and therefore needs a way to authenticate with it. TimescaleDB currently supports several different such authentication methods for flexibility (including trust, user mappings, password, and certificate methods). Refer to Setting up Multi-Node TimescaleDB for more information about node-to-node authentication.

Unless bootstrap is false, the function attempts to bootstrap the data node by:

1. Creating the database given in database that serve as the new data node.
2. Loading the TimescaleDB extension in the new database.
3. Setting metadata to make the data node part of the distributed database.

Note that user roles are not automatically created on the new data node during bootstrapping. The distributed_exec procedure can be used to create additional roles on the data node after it is added.

Required arguments

Name	Description
node_name	Name for the data node.
host	Host name for the remote data node.

Optional arguments

Name	Description
database	Database name where remote hypertables are created. The default is the current database name.
port	Port to use on the remote data node. The default is the Postgres port used by the access node on which the function is executed.
if_not_exists	Do not fail if the data node already exists. The default is FALSE.
bootstrap	Bootstrap the remote data node. The default is TRUE.
password	Password for authenticating with the remote data node during bootstrapping or validation. A password only needs to be provided if the data node requires password authentication and a password for the user does not exist in a local password file on the access node. If password authentication is not used, the specified password is ignored.

Returns

Column	Description
node_name	Local name to use for the data node
host	Host name for the remote data node
port	Port for the remote data node
database	Database name used on the remote data node
node_created	Was the data node created locally
database_created	Was the database created on the remote data node
extension_created	Was the extension created on the remote data node

Errors

An error is given if:

• The function is executed inside a transaction.
• The function is executed in a database that is already a data node.
• The data node already exists and if_not_exists is FALSE.
• The access node cannot connect to the data node due to a network failure or invalid configuration (for example, wrong port, or there is no way to authenticate the user).
• If bootstrap is FALSE and the database was not previously bootstrapped.

Privileges

To add a data node, you must be a superuser or have the USAGE privilege on the timescaledb_fdw foreign data wrapper. To grant such privileges to a regular user role, do:

GRANT USAGE ON FOREIGN DATA WRAPPER timescaledb_fdw TO <newrole>;

Note, however, that superuser privileges might still be necessary on the data node in order to bootstrap it, including creating the TimescaleDB extension on the data node unless it is already installed.

Sample usage

If you have an existing hypertable conditions and want to use time as the range partitioning column and location as the hash partitioning column. You also want to distribute the chunks of the hypertable on two data nodes dn1.example.com and dn2.example.com:

SELECT add_data_node('dn1', host => 'dn1.example.com');
SELECT add_data_node('dn2', host => 'dn2.example.com');
SELECT create_distributed_hypertable('conditions', 'time', 'location');

If you want to create a distributed database with the two data nodes local to this instance, you can write:

SELECT add_data_node('dn1', host => 'localhost', database => 'dn1');
SELECT add_data_node('dn2', host => 'localhost', database => 'dn2');
SELECT create_distributed_hypertable('conditions', 'time', 'location');

Note that this does not offer any performance advantages over using a regular hypertable, but it can be useful for testing.

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/detach_data_node/ =====

detach_data_node()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Detach a data node from one hypertable or from all hypertables.

Reasons for detaching a data node include:

• A data node should no longer be used by a hypertable and needs to be removed from all hypertables that use it
• You want to have fewer data nodes for a distributed hypertable to partition across

Required arguments

Name	Type	Description
node_name	TEXT	Name of data node to detach from the distributed hypertable

Optional arguments

Name	Type	Description
hypertable	REGCLASS	Name of the distributed hypertable where the data node should be detached. If NULL, the data node is detached from all hypertables.
if_attached	BOOLEAN	Prevent error if the data node is not attached. Defaults to false.
force	BOOLEAN	Force detach of the data node even if that means that the replication factor is reduced below what was set. Note that it is never allowed to reduce the replication factor below 1 since that would cause data loss.
repartition	BOOLEAN	Make the number of hash partitions equal to the new number of data nodes (if such partitioning exists). This ensures that the remaining data nodes are used evenly. Defaults to true.

Returns

The number of hypertables the data node was detached from.

Errors

Detaching a node is not permitted:

• If it would result in data loss for the hypertable due to the data node containing chunks that are not replicated on other data nodes
• If it would result in under-replicated chunks for the distributed hypertable (without the force argument)

Replication is currently experimental, and not a supported feature

Detaching a data node is under no circumstances possible if that would mean data loss for the hypertable. Nor is it possible to detach a data node, unless forced, if that would mean that the distributed hypertable would end up with under-replicated chunks.

The only safe way to detach a data node is to first safely delete any data on it or replicate it to another data node.

Sample usage

Detach data node dn3 from conditions:

SELECT detach_data_node('dn3', 'conditions');

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/set_replication_factor/ =====

set_replication_factor()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Sets the replication factor of a distributed hypertable to the given value. Changing the replication factor does not affect the number of replicas for existing chunks. Chunks created after changing the replication factor are replicated in accordance with new value of the replication factor. If the replication factor cannot be satisfied, since the amount of attached data nodes is less than new replication factor, the command aborts with an error.

If existing chunks have less replicas than new value of the replication factor, the function prints a warning.

Required arguments

Name	Type	Description
hypertable	REGCLASS	Distributed hypertable to update the replication factor for.
replication_factor	INTEGER	The new value of the replication factor. Must be greater than 0, and smaller than or equal to the number of attached data nodes.

Errors

An error is given if:

• hypertable is not a distributed hypertable.
• replication_factor is less than 1, which cannot be set on a distributed hypertable.
• replication_factor is bigger than the number of attached data nodes.

If a bigger replication factor is desired, it is necessary to attach more data nodes by using attach_data_node.

Sample usage

Update the replication factor for a distributed hypertable to 2:

SELECT set_replication_factor('conditions', 2);

Example of the warning if any existing chunk of the distributed hypertable has less than 2 replicas:

WARNING:  hypertable "conditions" is under-replicated
DETAIL:  Some chunks have less than 2 replicas.

Example of providing too big of a replication factor for a hypertable with 2 attached data nodes:

SELECT set_replication_factor('conditions', 3);
ERROR:  too big replication factor for hypertable "conditions"
DETAIL:  The hypertable has 2 data nodes attached, while the replication factor is 3.
HINT:  Decrease the replication factor or attach more data nodes to the hypertable.

===== PAGE: https://docs.tigerdata.com/api/distributed-hypertables/delete_data_node/ =====

delete_data_node()

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

This function is executed on an access node to remove a data node from the local database. As part of the deletion, the data node is detached from all hypertables that are using it, if permissions and data integrity requirements are satisfied. For more information, see detach_data_node.

Deleting a data node is strictly a local operation; the data node itself is not affected and the corresponding remote database on the data node is left intact, including all its data. The operation is local to ensure it can complete even if the remote data node is not responding and to avoid unintentional data loss on the data node.

It is not possible to use add_data_node to add the same data node again without first deleting the database on the data node or using another database. This is to prevent adding a data node that was previously part of the same or another distributed database but is no longer synchronized.

Errors

An error is generated if the data node cannot be detached from all attached hypertables.

Required arguments

Name	Type	Description
node_name	TEXT	Name of the data node.

Optional arguments

Name	Type	Description
if_exists	BOOLEAN	Prevent error if the data node does not exist. Defaults to false.
force	BOOLEAN	Force removal of data nodes from hypertables unless that would result in data loss. Defaults to false.
repartition	BOOLEAN	Make the number of hash partitions equal to the new number of data nodes (if such partitioning exists). This ensures that the remaining data nodes are used evenly. Defaults to true.

Returns

A boolean indicating if the operation was successful or not.

Sample usage

To delete a data node named dn1:

SELECT delete_data_node('dn1');

===== PAGE: https://docs.tigerdata.com/api/informational-views/chunk_compression_settings/ =====

timescaledb_information.chunk_compression_settings

Shows information about compression settings for each chunk that has compression enabled on it.

Samples

Show compression settings for all chunks:

SELECT * FROM timescaledb_information.chunk_compression_settings'
hypertable               | measurements
chunk					 | _timescaledb_internal._hyper_1_1_chunk
segmentby                |
orderby                  | "time" DESC

Find all chunk compression settings for a specific hypertable:

SELECT * FROM timescaledb_information.chunk_compression_settings WHERE hypertable::TEXT LIKE 'metrics';
hypertable               | metrics
chunk					 | _timescaledb_internal._hyper_2_3_chunk
segmentby                | metric_id
orderby                  | "time"

Arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable which has compression enabled
chunk	REGCLASS	Chunk which has compression enabled
segmentby	TEXT	List of columns used for segmenting the compressed data
orderby	TEXT	List of columns used for ordering compressed data along with ordering and NULL ordering information

===== PAGE: https://docs.tigerdata.com/api/informational-views/jobs/ =====

timescaledb_information.jobs

Shows information about all jobs registered with the automation framework.

Samples

Shows a job associated with the refresh policy for continuous aggregates:

SELECT * FROM timescaledb_information.jobs;
job_id            | 1001
application_name  | Refresh Continuous Aggregate Policy [1001]
schedule_interval | 01:00:00
max_runtime       | 00:00:00
max_retries       | -1
retry_period      | 01:00:00
proc_schema       | _timescaledb_internal
proc_name         | policy_refresh_continuous_aggregate
owner             | postgres
scheduled         | t
config            | {"start_offset": "20 days", "end_offset": "10
days", "mat_hypertable_id": 2}
next_start        | 2020-10-02 12:38:07.014042-04
hypertable_schema | _timescaledb_internal
hypertable_name   | _materialized_hypertable_2
check_schema      | _timescaledb_internal
check_name       | policy_refresh_continuous_aggregate_check

Find all jobs related to compression policies (before TimescaleDB v2.20):

SELECT * FROM timescaledb_information.jobs where application_name like 'Compression%';
-[ RECORD 1 ]-----+--------------------------------------------------
job_id            | 1002
application_name  | Compression Policy [1002]
schedule_interval | 15 days 12:00:00
max_runtime       | 00:00:00
max_retries       | -1
retry_period      | 01:00:00
proc_schema       | _timescaledb_internal
proc_name         | policy_compression
owner             | postgres
scheduled         | t
config            | {"hypertable_id": 3, "compress_after": "60 days"}
next_start        | 2020-10-18 01:31:40.493764-04
hypertable_schema | public
hypertable_name   | conditions
check_schema      | _timescaledb_internal
check_name        | policy_compression_check

Find all jobs related to columnstore policies (TimescaleDB v2.20 and later):

SELECT * FROM timescaledb_information.jobs where application_name like 'Columnstore%';
-[ RECORD 1 ]-----+--------------------------------------------------
job_id            | 1002
application_name  | Columnstore Policy [1002]
schedule_interval | 15 days 12:00:00
max_runtime       | 00:00:00
max_retries       | -1
retry_period      | 01:00:00
proc_schema       | _timescaledb_internal
proc_name         | policy_compression
owner             | postgres
scheduled         | t
config            | {"hypertable_id": 3, "compress_after": "60 days"}
next_start        | 2025-10-18 01:31:40.493764-04
hypertable_schema | public
hypertable_name   | conditions
check_schema      | _timescaledb_internal
check_name        | policy_compression_check

Find custom jobs:

SELECT * FROM timescaledb_information.jobs where application_name like 'User-Define%';
-[ RECORD 1 ]-----+------------------------------
job_id            | 1003
application_name  | User-Defined Action [1003]
schedule_interval | 01:00:00
max_runtime       | 00:00:00
max_retries       | -1
retry_period      | 00:05:00
proc_schema       | public
proc_name         | custom_aggregation_func
owner             | postgres
scheduled         | t
config            | {"type": "function"}
next_start        | 2020-10-02 14:45:33.339885-04
hypertable_schema |
hypertable_name   |
check_schema      | NULL
check_name        | NULL
-[ RECORD 2 ]-----+------------------------------
job_id            | 1004
application_name  | User-Defined Action [1004]
schedule_interval | 01:00:00
max_runtime       | 00:00:00
max_retries       | -1
retry_period      | 00:05:00
proc_schema       | public
proc_name         | custom_retention_func
owner             | postgres
scheduled         | t
config            | {"type": "function"}
next_start        | 2020-10-02 14:45:33.353733-04
hypertable_schema |
hypertable_name   |
check_schema      | NULL
check_name        | NULL

Arguments

Name	Type	Description
job_id	INTEGER	The ID of the background job
application_name	TEXT	Name of the policy or job
schedule_interval	INTERVAL	The interval at which the job runs. Defaults to 24 hours
max_runtime	INTERVAL	The maximum amount of time the job is allowed to run by the background worker scheduler before it is stopped
max_retries	INTEGER	The number of times the job is retried if it fails
retry_period	INTERVAL	The amount of time the scheduler waits between retries of the job on failure
proc_schema	TEXT	Schema name of the function or procedure executed by the job
proc_name	TEXT	Name of the function or procedure executed by the job
owner	TEXT	Owner of the job
scheduled	BOOLEAN	Set to true to run the job automatically
fixed_schedule	BOOLEAN	Set to true for jobs executing at fixed times according to a schedule interval and initial start
config	JSONB	Configuration passed to the function specified by proc_name at execution time
next_start	TIMESTAMP WITH TIME ZONE	Next start time for the job, if it is scheduled to run automatically
initial_start	TIMESTAMP WITH TIME ZONE	Time the job is first run and also the time on which execution times are aligned for jobs with fixed schedules
hypertable_schema	TEXT	Schema name of the hypertable. Set to NULL for a job
hypertable_name	TEXT	Table name of the hypertable. Set to NULL for a job
check_schema	TEXT	Schema name of the optional configuration validation function, set when the job is created or updated
check_name	TEXT	Name of the optional configuration validation function, set when the job is created or updated

===== PAGE: https://docs.tigerdata.com/api/informational-views/hypertables/ =====

timescaledb_information.hypertables

Get metadata information about hypertables.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get information about a hypertable.

CREATE TABLE metrics(time timestamptz, device int, temp float);
SELECT create_hypertable('metrics','time');

SELECT * from timescaledb_information.hypertables WHERE hypertable_name = 'metrics';

-[ RECORD 1 ]-------+--------
hypertable_schema   | public
hypertable_name     | metrics
owner               | sven
num_dimensions      | 1
num_chunks          | 0
compression_enabled | f
tablespaces         | NULL

Available columns

Name	Type	Description
hypertable_schema	TEXT	Schema name of the hypertable
hypertable_name	TEXT	Table name of the hypertable
owner	TEXT	Owner of the hypertable
num_dimensions	SMALLINT	Number of dimensions
num_chunks	BIGINT	Number of chunks
compression_enabled	BOOLEAN	Is compression enabled on the hypertable?
is_distributed	BOOLEAN	Sunsetted since TimescaleDB v2.14.0 Is the hypertable distributed?
replication_factor	SMALLINT	Sunsetted since TimescaleDB v2.14.0 Replication factor for a distributed hypertable
data_nodes	TEXT	Sunsetted since TimescaleDB v2.14.0 Nodes on which hypertable is distributed
tablespaces	TEXT	Tablespaces attached to the hypertable

===== PAGE: https://docs.tigerdata.com/api/informational-views/policies/ =====

timescaledb_experimental.policies

The policies view provides information on all policies set on continuous aggregates.

Only policies applying to continuous aggregates are shown in this view. Policies applying to regular hypertables or regular materialized views are not displayed.

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

Samples

Select from the timescaledb_experimental.policies table to view it:

SELECT * FROM timescaledb_experimental.policies;

Example of the returned output:

-[ RECORD 1 ]--------------------------------------------------------------------
relation_name     | mat_m1
relation_schema   | public
schedule_interval | @ 1 hour
proc_schema       | _timescaledb_internal
proc_name         | policy_refresh_continuous_aggregate
config            | {"end_offset": 1, "start_offset", 10, "mat_hypertable_id": 2}
hypertable_schema | _timescaledb_internal
hypertable_name   | _materialized_hypertable_2
-[ RECORD 2 ]--------------------------------------------------------------------
relation_name     | mat_m1
relation_schema   | public
schedule_interval | @ 1 day
proc_schema       | _timescaledb_internal
proc_name         | policy_compression
config            | {"hypertable_id": 2, "compress_after", 11}
hypertable_schema | _timescaledb_internal
hypertable_name   | _materialized_hypertable_2
-[ RECORD 3 ]--------------------------------------------------------------------
relation_name     | mat_m1
relation_schema   | public
schedule_interval | @ 1 day
proc_schema       | _timescaledb_internal
proc_name         | policy_retention
config            | {"drop_after": 20, "hypertable_id": 2}
hypertable_schema | _timescaledb_internal
hypertable_name   | _materialized_hypertable_2

Available columns

Column	Type	Description
relation_name	Name of the continuous aggregate
relation_schema	Schema of the continuous aggregate
schedule_interval	How often the policy job runs
proc_schema	Schema of the policy job
proc_name	Name of the policy job
config	Configuration details for the policy job
hypertable_schema	Schema of the hypertable that contains the actual data for the continuous aggregate view
hypertable_name	Name of the hypertable that contains the actual data for the continuous aggregate view

===== PAGE: https://docs.tigerdata.com/api/informational-views/chunks/ =====

timescaledb_information.chunks

Get metadata about the chunks of hypertables.

This view shows metadata for the chunk's primary time-based dimension. For information about a hypertable's secondary dimensions, the dimensions view should be used instead.

If the chunk's primary dimension is of a time datatype, range_start and range_end are set. Otherwise, if the primary dimension type is integer based, range_start_integer and range_end_integer are set.

Samples

Get information about the chunks of a hypertable.

Dimension builder by_range was introduced in TimescaleDB 2.13. The chunk_creation_time metadata was introduced in TimescaleDB 2.13.

CREATE TABLESPACE tablespace1 location '/usr/local/pgsql/data1';

CREATE TABLE hyper_int (a_col integer, b_col integer, c integer);
SELECT table_name from create_hypertable('hyper_int', by_range('a_col', 10));
CREATE OR REPLACE FUNCTION integer_now_hyper_int() returns int LANGUAGE SQL STABLE as $$ SELECT coalesce(max(a_col), 0) FROM hyper_int $$;
SELECT set_integer_now_func('hyper_int', 'integer_now_hyper_int');

INSERT INTO hyper_int SELECT generate_series(1,5,1), 10, 50;

SELECT attach_tablespace('tablespace1', 'hyper_int');
INSERT INTO hyper_int VALUES( 25 , 14 , 20), ( 25, 15, 20), (25, 16, 20);

SELECT * FROM timescaledb_information.chunks WHERE hypertable_name = 'hyper_int';

-[ RECORD 1 ]----------+----------------------
hypertable_schema      | public
hypertable_name        | hyper_int
chunk_schema           | _timescaledb_internal
chunk_name             | _hyper_7_10_chunk
primary_dimension      | a_col
primary_dimension_type | integer
range_start            |
range_end              |
range_start_integer    | 0
range_end_integer      | 10
is_compressed          | f
chunk_tablespace       |
data_nodes             |
-[ RECORD 2 ]----------+----------------------
hypertable_schema      | public
hypertable_name        | hyper_int
chunk_schema           | _timescaledb_internal
chunk_name             | _hyper_7_11_chunk
primary_dimension      | a_col
primary_dimension_type | integer
range_start            |
range_end              |
range_start_integer    | 20
range_end_integer      | 30
is_compressed          | f
chunk_tablespace       | tablespace1
data_nodes             |

Available columns

Name	Type	Description
hypertable_schema	TEXT	Schema name of the hypertable
hypertable_name	TEXT	Table name of the hypertable
chunk_schema	TEXT	Schema name of the chunk
chunk_name	TEXT	Name of the chunk
primary_dimension	TEXT	Name of the column that is the primary dimension
primary_dimension_type	REGTYPE	Type of the column that is the primary dimension
range_start	TIMESTAMP WITH TIME ZONE	Start of the range for the chunk's dimension
range_end	TIMESTAMP WITH TIME ZONE	End of the range for the chunk's dimension
range_start_integer	BIGINT	Start of the range for the chunk's dimension, if the dimension type is integer based
range_end_integer	BIGINT	End of the range for the chunk's dimension, if the dimension type is integer based
is_compressed	BOOLEAN	Is the data in the chunk compressed? Note that for distributed hypertables, this is the cached compression status of the chunk on the access node. The cached status on the access node and data node is not in sync in some scenarios. For example, if a user compresses or decompresses the chunk on the data node instead of the access node, or sets up compression policies directly on data nodes. Use chunk_compression_stats() function to get real-time compression status for distributed chunks.
chunk_tablespace	TEXT	Tablespace used by the chunk
data_nodes	ARRAY	Nodes on which the chunk is replicated. This is applicable only to chunks for distributed hypertables
chunk_creation_time	TIMESTAMP WITH TIME ZONE	The time when this chunk was created for data addition

===== PAGE: https://docs.tigerdata.com/api/informational-views/data_nodes/ =====

timescaledb_information.data_nodes

Get information on data nodes. This function is specific to running TimescaleDB in a multi-node setup.

Multi-node support is sunsetted.

TimescaleDB v2.13 is the last release that includes multi-node support for Postgres versions 13, 14, and 15.

Samples

Get metadata related to data nodes.

SELECT * FROM timescaledb_information.data_nodes;

 node_name    | owner      | options
--------------+------------+--------------------------------
 dn1         | postgres   | {host=localhost,port=15431,dbname=test}
 dn2         | postgres   | {host=localhost,port=15432,dbname=test}
(2 rows)

Available columns

Name	Type	Description
node_name	TEXT	Data node name.
owner	REGCLASS	Oid of the user, who added the data node.
options	JSONB	Options used when creating the data node.

===== PAGE: https://docs.tigerdata.com/api/informational-views/hypertable_compression_settings/ =====

timescaledb_information.hypertable_compression_settings

Shows information about compression settings for each hypertable chunk that has compression enabled on it.

Samples

Show compression settings for all hypertables:

SELECT * FROM timescaledb_information.hypertable_compression_settings;
hypertable               | measurements
chunk                    | _timescaledb_internal._hyper_2_97_chunk
segmentby                |
orderby                  | time DESC

Find compression settings for a specific hypertable:

SELECT * FROM timescaledb_information.hypertable_compression_settings WHERE hypertable::TEXT LIKE 'metrics';
hypertable               | metrics
chunk                    | _timescaledb_internal._hyper_1_12_chunk
segmentby                | metric_id
orderby                  | time DESC

Arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable which has compression enabled
chunk	REGCLASS	Hypertable chunk which has compression enabled
segmentby	TEXT	List of columns used for segmenting the compressed data
orderby	TEXT	List of columns used for ordering compressed data along with ordering and NULL ordering information

===== PAGE: https://docs.tigerdata.com/api/informational-views/compression_settings/ =====

timescaledb_information.compression_settings

This view exists for backwards compatibility. The supported views to retrieve information about compression are:

• timescaledb_information.hypertable_compression_settings
• timescaledb_information.chunk_compression_settings.

This section describes a feature that is deprecated. We strongly recommend that you do not use this feature in a production environment. If you need more information, contact us.

Get information about compression-related settings for hypertables. Each row of the view provides information about individual orderby and segmentby columns used by compression.

How you use segmentby is the single most important thing for compression. It affects compresion rates, query performance, and what is compressed or decompressed by mutable compression.

Samples

CREATE TABLE hypertab (a_col integer, b_col integer, c_col integer, d_col integer, e_col integer);
SELECT table_name FROM create_hypertable('hypertab', by_range('a_col', 864000000));

ALTER TABLE hypertab SET (timescaledb.compress, timescaledb.compress_segmentby = 'a_col,b_col',
  timescaledb.compress_orderby = 'c_col desc, d_col asc nulls last');

SELECT * FROM timescaledb_information.compression_settings WHERE hypertable_name = 'hypertab';

-[ RECORD 1 ]----------+---------
hypertable_schema      | public
hypertable_name        | hypertab
attname                | a_col
segmentby_column_index | 1
orderby_column_index   |
orderby_asc            |
orderby_nullsfirst     |
-[ RECORD 2 ]----------+---------
hypertable_schema      | public
hypertable_name        | hypertab
attname                | b_col
segmentby_column_index | 2
orderby_column_index   |
orderby_asc            |
orderby_nullsfirst     |
-[ RECORD 3 ]----------+---------
hypertable_schema      | public
hypertable_name        | hypertab
attname                | c_col
segmentby_column_index |
orderby_column_index   | 1
orderby_asc            | f
orderby_nullsfirst     | t
-[ RECORD 4 ]----------+---------
hypertable_schema      | public
hypertable_name        | hypertab
attname                | d_col
segmentby_column_index |
orderby_column_index   | 2
orderby_asc            | t
orderby_nullsfirst     | f

The by_range dimension builder is an addition to TimescaleDB 2.13.

Available columns

Name	Type	Description
hypertable_schema	TEXT	Schema name of the hypertable
hypertable_name	TEXT	Table name of the hypertable
attname	TEXT	Name of the column used in the compression settings
segmentby_column_index	SMALLINT	Position of attname in the compress_segmentby list
orderby_column_index	SMALLINT	Position of attname in the compress_orderby list
orderby_asc	BOOLEAN	True if this is used for order by ASC, False for order by DESC
orderby_nullsfirst	BOOLEAN	True if nulls are ordered first for this column, False if nulls are ordered last

===== PAGE: https://docs.tigerdata.com/api/informational-views/dimensions/ =====

timescaledb_information.dimensions

Returns information about the dimensions of a hypertable. Hypertables can be partitioned on a range of different dimensions. By default, all hypertables are partitioned on time, but it is also possible to partition on other dimensions in addition to time.

For hypertables that are partitioned solely on time, timescaledb_information.dimensions returns a single row of metadata. For hypertables that are partitioned on more than one dimension, the call returns a row for each dimension.

For time-based dimensions, the metadata returned indicates the integer datatype, such as BIGINT, INTEGER, or SMALLINT, and the time-related datatype, such as TIMESTAMPTZ, TIMESTAMP, or DATE. For space-based dimension, the metadata returned specifies the number of num_partitions.

If the hypertable uses time data types, the time_interval column is defined. Alternatively, if the hypertable uses integer data types, the integer_interval and integer_now_func columns are defined.

Samples

Get information about the dimensions of hypertables.

-- Create a range and hash partitioned hypertable
CREATE TABLE dist_table(time timestamptz, device int, temp float);
SELECT create_hypertable('dist_table', by_range('time', INTERVAL '7 days'));
SELECT add_dimension('dist_table', by_hash('device', 3));

SELECT * from timescaledb_information.dimensions
  ORDER BY hypertable_name, dimension_number;

-[ RECORD 1 ]-----+-------------------------
hypertable_schema | public
hypertable_name   | dist_table
dimension_number  | 1
column_name       | time
column_type       | timestamp with time zone
dimension_type    | Time
time_interval     | 7 days
integer_interval  |
integer_now_func  |
num_partitions    |
-[ RECORD 2 ]-----+-------------------------
hypertable_schema | public
hypertable_name   | dist_table
dimension_number  | 2
column_name       | device
column_type       | integer
dimension_type    | Space
time_interval     |
integer_interval  |
integer_now_func  |
num_partitions    | 2

The by_range and by_hash dimension builders are an addition to TimescaleDB 2.13.

Get information about dimensions of a hypertable that has two time-based dimensions.

CREATE TABLE hyper_2dim (a_col date, b_col timestamp, c_col integer);
SELECT table_name from create_hypertable('hyper_2dim', by_range('a_col'));
SELECT add_dimension('hyper_2dim', by_range('b_col', INTERVAL '7 days'));

SELECT * FROM timescaledb_information.dimensions WHERE hypertable_name = 'hyper_2dim';

-[ RECORD 1 ]-----+----------------------------
hypertable_schema | public
hypertable_name   | hyper_2dim
dimension_number  | 1
column_name       | a_col
column_type       | date
dimension_type    | Time
time_interval     | 7 days
integer_interval  |
integer_now_func  |
num_partitions    |
-[ RECORD 2 ]-----+----------------------------
hypertable_schema | public
hypertable_name   | hyper_2dim
dimension_number  | 2
column_name       | b_col
column_type       | timestamp without time zone
dimension_type    | Time
time_interval     | 7 days
integer_interval  |
integer_now_func  |
num_partitions    |

Available columns

Name	Type	Description
hypertable_schema	TEXT	Schema name of the hypertable
hypertable_name	TEXT	Table name of the hypertable
dimension_number	BIGINT	Dimension number of the hypertable, starting from 1
column_name	TEXT	Name of the column used to create this dimension
column_type	REGTYPE	Type of the column used to create this dimension
dimension_type	TEXT	Is this a time based or space based dimension
time_interval	INTERVAL	Time interval for primary dimension if the column type is a time datatype
integer_interval	BIGINT	Integer interval for primary dimension if the column type is an integer datatype
integer_now_func	TEXT	`integer_now`` function for primary dimension if the column type is an integer datatype
num_partitions	SMALLINT	Number of partitions for the dimension

The time_interval and integer_interval columns are not applicable for space based dimensions.

===== PAGE: https://docs.tigerdata.com/api/informational-views/job_errors/ =====

timescaledb_information.job_errors

Shows information about runtime errors encountered by jobs run by the automation framework. This includes custom jobs and jobs run by policies created to manage data retention, continuous aggregates, columnstore, and other automation policies. For more information about automation policies, see the policies section.

Samples

See information about recent job failures:

SELECT job_id, proc_schema, proc_name, pid, sqlerrcode, err_message from timescaledb_information.job_errors ;

 job_id | proc_schema |  proc_name   |  pid  | sqlerrcode |                     err_message
--------+-------------+--------------+-------+------------+-----------------------------------------------------
   1001 | public      | custom_proc2 | 83111 | 40001      | could not serialize access due to concurrent update
   1003 | public      | job_fail     | 83134 | 57014      | canceling statement due to user request
   1005 | public      | job_fail     |       |            | job crash detected, see server logs
(3 rows)

Available columns

Name	Type	Description
job_id	INTEGER	The ID of the background job created to implement the policy
proc_schema	TEXT	Schema name of the function or procedure executed by the job
proc_name	TEXT	Name of the function or procedure executed by the job
pid	INTEGER	The process ID of the background worker executing the job. This is NULL in the case of a job crash
start_time	TIMESTAMP WITH TIME ZONE	Start time of the job
finish_time	TIMESTAMP WITH TIME ZONE	Time when error was reported
sqlerrcode	TEXT	The error code associated with this error, if any. See the official Postgres documentation for a full list of error codes
err_message	TEXT	The detailed error message

Error retention policy

The informational view timescaledb_information.job_errors is defined on top of the table _timescaledb_internal.job_errors in the internal schema. To prevent this table from growing too large, a system background job Error Log Retention Policy [2] is enabled by default, with this configuration:

id                | 2
application_name  | Error Log Retention Policy [2]
schedule_interval | 1 mon
max_runtime       | 01:00:00
max_retries       | -1
retry_period      | 01:00:00
proc_schema       | _timescaledb_internal
proc_name         | policy_job_error_retention
owner             | owner must be a user with WRITE privilege on the table `_timescaledb_internal.job_errors`
scheduled         | t
fixed_schedule    | t
initial_start     | 2000-01-01 02:00:00+02
hypertable_id     |
config            | {"drop_after": "1 month"}
check_schema      | _timescaledb_internal
check_name        | policy_job_error_retention_check
timezone          |

On TimescaleDB and Managed Service for TimescaleDB, the owner of the error retention job is tsdbadmin. In an on-premise installation, the owner of the job is the same as the extension owner. The owner of the retention job can alter it and delete it. For example, the owner can change the retention interval like this:

SELECT alter_job(id,config:=jsonb_set(config,'{drop_after}', '"2 weeks"')) FROM _timescaledb_config.bgw_job WHERE id = 2;

===== PAGE: https://docs.tigerdata.com/api/informational-views/job_history/ =====

timescaledb_information.history

Shows information about the jobs run by the automation framework. This includes custom jobs and jobs run by policies created to manage data retention, continuous aggregates, columnstore, and other automation policies. For more information about automation policies, see jobs.

Samples

To retrieve information about recent jobs:

SELECT job_id, pid, proc_schema, proc_name, succeeded, config, sqlerrcode, err_message
FROM timescaledb_information.job_history
ORDER BY id, job_id;
 job_id |   pid   | proc_schema |    proc_name     | succeeded |   config   | sqlerrcode |   err_message
--------+---------+-------------+------------------+-----------+------------+------------+------------------
   1001 | 1779278 | public      | custom_job_error | f         |            | 22012      | division by zero
   1000 | 1779407 | public      | custom_job_ok    | t         |            |            |
   1001 | 1779408 | public      | custom_job_error | f         |            | 22012      | division by zero
   1000 | 1779467 | public      | custom_job_ok    | t         | {"foo": 1} |            |
   1001 | 1779468 | public      | custom_job_error | f         | {"bar": 1} | 22012      | division by zero
(5 rows)

Available columns

Name	Type	Description
id	INTEGER	The sequencial ID to identify the job execution
job_id	INTEGER	The ID of the background job created to implement the policy
succeeded	BOOLEAN	TRUE when the job ran successfully, FALSE for failed executions
proc_schema	TEXT	The schema name of the function or procedure executed by the job
proc_name	TEXT	The name of the function or procedure executed by the job
pid	INTEGER	The process ID of the background worker executing the job. This is NULL in the case of a job crash
start_time	TIMESTAMP WITH TIME ZONE	The time the job started
finish_time	TIMESTAMP WITH TIME ZONE	The time when the error was reported
config	JSONB	The job configuration at the moment of execution
sqlerrcode	TEXT	The error code associated with this error, if any. See the official Postgres documentation for a full list of error codes
err_message	TEXT	The detailed error message

Error retention policy

The timescaledb_information.job_history informational view is defined on top of the _timescaledb_internal.bgw_job_stat_history table in the internal schema. To prevent this table from growing too large, the Job History Log Retention Policy [3] system background job is enabled by default, with this configuration:

job_id            | 3
application_name  | Job History Log Retention Policy [3]
schedule_interval | 1 mon
max_runtime       | 01:00:00
max_retries       | -1
retry_period      | 01:00:00
proc_schema       | _timescaledb_functions
proc_name         | policy_job_stat_history_retention
owner             | owner must be a user with WRITE privilege on the table `_timescaledb_internal.bgw_job_stat_history`
scheduled         | t
fixed_schedule    | t
config            | {"drop_after": "1 month"}
next_start        | 2024-06-01 01:00:00+00
initial_start     | 2000-01-01 00:00:00+00
hypertable_schema |
hypertable_name   |
check_schema      | _timescaledb_functions
check_name        | policy_job_stat_history_retention_check

On TimescaleDB and Managed Service for TimescaleDB, the owner of the job history retention job is tsdbadmin. In an on-premise installation, the owner of the job is the same as the extension owner. The owner of the retention job can alter it and delete it. For example, the owner can change the retention interval like this:

SELECT alter_job(id,config:=jsonb_set(config,'{drop_after}', '"2 weeks"')) FROM _timescaledb_config.bgw_job WHERE id = 3;

===== PAGE: https://docs.tigerdata.com/api/informational-views/job_stats/ =====

timescaledb_information.job_stats

Shows information and statistics about jobs run by the automation framework. This includes jobs set up for user defined actions and jobs run by policies created to manage data retention, continuous aggregates, columnstore, and other automation policies. (See policies). The statistics include information useful for administering jobs and determining whether they ought be rescheduled, such as: when and whether the background job used to implement the policy succeeded and when it is scheduled to run next.

Samples

Get job success/failure information for a specific hypertable.

SELECT job_id, total_runs, total_failures, total_successes
  FROM timescaledb_information.job_stats
  WHERE hypertable_name = 'test_table';

 job_id | total_runs | total_failures | total_successes
--------+------------+----------------+-----------------
   1001 |          1 |              0 |               1
   1004 |          1 |              0 |               1
(2 rows)

Get information about continuous aggregate policy related statistics

SELECT  js.* FROM
  timescaledb_information.job_stats js, timescaledb_information.continuous_aggregates cagg
  WHERE cagg.view_name = 'max_mat_view_timestamp'
  and cagg.materialization_hypertable_name = js.hypertable_name;

-[ RECORD 1 ]----------+------------------------------
hypertable_schema      | _timescaledb_internal
hypertable_name        | _materialized_hypertable_2
job_id                 | 1001
last_run_started_at    | 2020-10-02 09:38:06.871953-04
last_successful_finish | 2020-10-02 09:38:06.932675-04
last_run_status        | Success
job_status             | Scheduled
last_run_duration      | 00:00:00.060722
next_start             | 2020-10-02 10:38:06.932675-04
total_runs             | 1
total_successes        | 1
total_failures         | 0

Available columns

Name	Type	Description
hypertable_schema	TEXT	Schema name of the hypertable
hypertable_name	TEXT	Table name of the hypertable
job_id	INTEGER	The id of the background job created to implement the policy
last_run_started_at	TIMESTAMP WITH TIME ZONE	Start time of the last job
last_successful_finish	TIMESTAMP WITH TIME ZONE	Time when the job completed successfully
last_run_status	TEXT	Whether the last run succeeded or failed
job_status	TEXT	Status of the job. Valid values are 'Running', 'Scheduled' and 'Paused'
last_run_duration	INTERVAL	Duration of last run of the job
next_start	TIMESTAMP WITH TIME ZONE	Start time of the next run
total_runs	BIGINT	The total number of runs of this job
total_successes	BIGINT	The total number of times this job succeeded
total_failures	BIGINT	The total number of times this job failed

===== PAGE: https://docs.tigerdata.com/api/informational-views/continuous_aggregates/ =====

timescaledb_information.continuous_aggregates

Get metadata and settings information for continuous aggregates.

Samples

SELECT * FROM timescaledb_information.continuous_aggregates;

-[ RECORD 1 ]---------------------+-------------------------------------------------
hypertable_schema                 | public
hypertable_name                   | foo
view_schema                       | public
view_name                         | contagg_view
view_owner                        | postgres
materialized_only                 | f
compression_enabled               | f
materialization_hypertable_schema | _timescaledb_internal
materialization_hypertable_name   | _materialized_hypertable_2
view_definition                   |  SELECT foo.a,                                  +
                                  |     COUNT(foo.b) AS countb                      +
                                  |    FROM foo                                     +
                                  |   GROUP BY (time_bucket('1 day', foo.a)), foo.a;
finalized                         | t

Available columns

Name	Type	Description
hypertable_schema	TEXT	Schema of the hypertable from the continuous aggregate view
hypertable_name	TEXT	Name of the hypertable from the continuous aggregate view
view_schema	TEXT	Schema for continuous aggregate view
view_name	TEXT	User supplied name for continuous aggregate view
view_owner	TEXT	Owner of the continuous aggregate view
materialized_only	BOOLEAN	Return only materialized data when querying the continuous aggregate view
compression_enabled	BOOLEAN	Is compression enabled for the continuous aggregate view?
materialization_hypertable_schema	TEXT	Schema of the underlying materialization table
materialization_hypertable_name	TEXT	Name of the underlying materialization table
view_definition	TEXT	SELECT query for continuous aggregate view
finalized	BOOLEAN	Whether the continuous aggregate stores data in finalized or partial form. Since TimescaleDB 2.7, the default is finalized.

===== PAGE: https://docs.tigerdata.com/api/jobs-automation/alter_job/ =====

alter_job()

Jobs scheduled using the TimescaleDB automation framework run periodically in a background worker. You can change the schedule of these jobs with the alter_job function. To alter an existing job, refer to it by job_id. The job_id runs a given job, and its current schedule can be found in the timescaledb_information.jobs view, which lists information about every scheduled jobs, as well as in timescaledb_information.job_stats. The job_stats view also gives information about when each job was last run and other useful statistics for deciding what the new schedule should be.

Samples

Reschedules job ID 1000 so that it runs every two days:

SELECT alter_job(1000, schedule_interval => INTERVAL '2 days');

Disables scheduling of the compression policy on the conditions hypertable:

SELECT alter_job(job_id, scheduled => false)
FROM timescaledb_information.jobs
WHERE proc_name = 'policy_compression' AND hypertable_name = 'conditions'

Reschedules continuous aggregate job ID 1000 so that it next runs at 9:00:00 on 15 March, 2020:

SELECT alter_job(1000, next_start => '2020-03-15 09:00:00.0+00');

Required arguments

Name	Type	Description
job_id	INTEGER	The ID of the policy job being modified

Optional arguments

Name	Type	Description
schedule_interval	INTERVAL	The interval at which the job runs. Defaults to 24 hours.
max_runtime	INTERVAL	The maximum amount of time the job is allowed to run by the background worker scheduler before it is stopped.
max_retries	INTEGER	The number of times the job is retried if it fails.
retry_period	INTERVAL	The amount of time the scheduler waits between retries of the job on failure.
scheduled	BOOLEAN	Set to FALSE to exclude this job from being run as background job.
config	JSONB	Job-specific configuration, passed to the function when it runs. This includes: verbose_log: boolean, defaults to false. Enable verbose logging output when running the compression policy. maxchunks_to_compress: integer, defaults to 0 (no limit). The maximum number of chunks to compress during a policy run. recompress: boolean, defaults to true. Recompress partially compressed chunks. compress_after: see add_compression_policy. compress_created_before: see add_compression_policy.
next_start	TIMESTAMPTZ	The next time at which to run the job. The job can be paused by setting this value to infinity, and restarted with a value of now().
if_exists	BOOLEAN	Set to trueto issue a notice instead of an error if the job does not exist. Defaults to false.
check_config	REGPROC	A function that takes a single argument, the JSONB config structure. The function is expected to raise an error if the configuration is not valid, and return nothing otherwise. Can be used to validate the configuration when updating a job. Only functions, not procedures, are allowed as values for check_config.
fixed_schedule	BOOLEAN	To enable fixed scheduled job runs, set to TRUE.
initial_start	TIMESTAMPTZ	Set the time when the fixed_schedule job run starts. For example, 19:10:25-07.
timezone	TEXT	Address the 1-hour shift in start time when clocks change from Daylight Saving Time to Standard Time. For example, America/Sao_Paulo.

When a job begins, the next_start parameter is set to infinity. This prevents the job from attempting to be started again while it is running. When the job completes, whether or not the job is successful, the parameter is automatically updated to the next computed start time.

Note that altering the next_start value is only effective for the next execution of the job in case of fixed schedules. On the next execution, it will automatically return to the schedule.

Returns

Column	Type	Description
job_id	INTEGER	The ID of the job being modified
schedule_interval	INTERVAL	The interval at which the job runs. Defaults to 24 hours
max_runtime	INTERVAL	The maximum amount of time the job is allowed to run by the background worker scheduler before it is stopped
max_retries	INTEGER	The number of times the job is retried if it fails
retry_period	INTERVAL	The amount of time the scheduler waits between retries of the job on failure
scheduled	BOOLEAN	Returns true if the job is executed by the TimescaleDB scheduler
config	JSONB	Jobs-specific configuration, passed to the function when it runs
next_start	TIMESTAMPTZ	The next time to run the job
check_config	TEXT	The function used to validate updated job configurations

Calculation of next start on failure

When a job run results in a runtime failure, the next start of the job is calculated taking into account both its retry_period and schedule_interval. The next_start time is calculated using the following formula:

next_start = finish_time + consecutive_failures * retry_period ± jitter

where jitter (± 13%) is added to avoid the "thundering herds" effect.

To ensure that the next_start time is not put off indefinitely or produce timestamps so large they end up out of range, it is capped at 5*schedule_interval. Also, more than 20 consecutive failures are not considered, so if the number of consecutive failures is higher, then it multiplies by 20.

Additionally, for jobs with fixed schedules, the system ensures that if the next start ( calculated as specified), surpasses the next scheduled execution, the job is executed again at the next scheduled slot and not after that. This ensures that the job does not miss scheduled executions.

There is a distinction between runtime failures that do not cause the job to crash and job crashes. In the event of a job crash, the next start calculation follows the same formula, but it is always at least 5 minutes after the job's last finish, to give an operator enough time to disable it before another crash.

===== PAGE: https://docs.tigerdata.com/api/jobs-automation/delete_job/ =====

delete_job()

Delete a job registered with the automation framework. This works for jobs as well as policies.

If the job is currently running, the process is terminated.

Samples

Delete the job with the job id 1000:

SELECT delete_job(1000);

Required arguments

Name	Type	Description
job_id	INTEGER	TimescaleDB background job id

===== PAGE: https://docs.tigerdata.com/api/jobs-automation/run_job/ =====

run_job()

Run a previously registered job in the current session. This works for job as well as policies. Since run_job is implemented as stored procedure it cannot be executed inside a SELECT query but has to be executed with CALL.

Any background worker job can be run in the foreground when executed with run_job. You can use this with an increased log level to help debug problems.

Samples

Set log level shown to client to DEBUG1 and run the job with the job ID 1000:

SET client_min_messages TO DEBUG1;
CALL run_job(1000);

Required arguments

Name	Description
job_id	(INTEGER) TimescaleDB background job ID

===== PAGE: https://docs.tigerdata.com/api/jobs-automation/add_job/ =====

add_job()

Register a job for scheduling by the automation framework. For more information about scheduling, including example jobs, see the jobs documentation section.

Samples

Register the user_defined_action procedure to run every hour:

CREATE OR REPLACE PROCEDURE user_defined_action(job_id int, config jsonb) LANGUAGE PLPGSQL AS
$$
BEGIN
  RAISE NOTICE 'Executing action % with config %', job_id, config;
END
$$;

SELECT add_job('user_defined_action','1h');
SELECT add_job('user_defined_action','1h', fixed_schedule => false);

Register the user_defined_action procedure to run at midnight every Sunday. The initial_start provided must satisfy these requirements, so it must be a Sunday midnight:

-- December 4, 2022 is a Sunday
SELECT add_job('user_defined_action','1 week', initial_start => '2022-12-04 00:00:00+00'::timestamptz);
-- if subject to DST
SELECT add_job('user_defined_action','1 week', initial_start => '2022-12-04 00:00:00+00'::timestamptz, timezone => 'Europe/Berlin');

Required arguments

Name	Type	Description
proc	REGPROC	Name of the function or procedure to register as a job.
schedule_interval	INTERVAL	Interval between executions of this job. Defaults to 24 hours

Optional arguments

Name	Type	Description
config	JSONB	Jobs-specific configuration, passed to the function when it runs
initial_start	TIMESTAMPTZ	Time the job is first run. In the case of fixed schedules, this also serves as the origin on which job executions are aligned. If omitted, the current time is used as origin in the case of fixed schedules.
scheduled	BOOLEAN	Set to FALSE to exclude this job from scheduling. Defaults to TRUE.
check_config	REGPROC	A function that takes a single argument, the JSONB config structure. The function is expected to raise an error if the configuration is not valid, and return nothing otherwise. Can be used to validate the configuration when adding a job. Only functions, not procedures, are allowed as values for check_config.
fixed_schedule	BOOLEAN	Set to FALSE if you want the next start of a job to be determined as its last finish time plus the schedule interval. Set to TRUE if you want the next start of a job to begin schedule_interval after the last start. Defaults to TRUE
timezone	TEXT	A valid time zone. If fixed_schedule is TRUE, subsequent executions of the job are aligned on its initial start. However, daylight savings time (DST) changes may shift this alignment. Set to a valid time zone if you want to mitigate this issue. Defaults to NULL.

Returns

Column	Type	Description
job_id	INTEGER	TimescaleDB background job ID

===== PAGE: https://docs.tigerdata.com/api/data-retention/add_retention_policy/ =====

add_retention_policy()

Create a policy to drop chunks older than a given interval of a particular hypertable or continuous aggregate on a schedule in the background. For more information, see the drop_chunks section. This implements a data retention policy and removes data on a schedule. Only one retention policy may exist per hypertable.

When you create a retention policy on a hypertable with an integer based time column, you must set the integer_now_func to match your data. If you are seeing invalid value issues when you call add_retention_policy, set VERBOSITY verbose to see the full context.

Samples

• Create a data retention policy to discard chunks greater than 6 months old:

  SELECT add_retention_policy('conditions', drop_after => INTERVAL '6 months');
  

  When you call drop_after, the time data range present in the partitioning time column is used to select the target chunks.

• Create a data retention policy with an integer-based time column:

  SELECT add_retention_policy('conditions', drop_after => BIGINT '600000');
  

• Create a data retention policy to discard chunks created before 6 months:

  SELECT add_retention_policy('conditions', drop_created_before => INTERVAL '6 months');
  

  When you call drop_created_before, chunks created 3 months ago are selected.

Arguments

Name	Type	Default	Required	Description
relation	REGCLASS	-	✔	Name of the hypertable or continuous aggregate to create the policy for
drop_after	INTERVAL or INTEGER	-	✔	Chunks fully older than this interval when the policy is run are dropped. You specify drop_after differently depending on the hypertable time column type: TIMESTAMP, TIMESTAMPTZ, and DATE: use INTERVAL type Integer-based timestamps: use INTEGER type. You must set integer_now_func to match your data
schedule_interval	INTERVAL	NULL	✖	The interval between the finish time of the last execution and the next start.
initial_start	TIMESTAMPTZ	NULL	✖	Time the policy is first run. If omitted, then the schedule interval is the interval between the finish time of the last execution and the next start. If provided, it serves as the origin with respect to which the next_start is calculated.
timezone	TEXT	NULL	✖	A valid time zone. If initial_start is also specified, subsequent executions of the retention policy are aligned on its initial start. However, daylight savings time (DST) changes may shift this alignment. Set to a valid time zone if this is an issue you want to mitigate. If omitted, UTC bucketing is performed.
if_not_exists	BOOLEAN	false	✖	Set to true to avoid an error if the drop_chunks_policy already exists. A notice is issued instead.
drop_created_before	INTERVAL	NULL	✖	Chunks with creation time older than this cut-off point are dropped. The cut-off point is computed as now() - drop_created_before. Not supported for continuous aggregates yet.

You specify drop_after differently depending on the hypertable time column type:

• TIMESTAMP, TIMESTAMPTZ, and DATE time columns: the time interval should be an INTERVAL type.
• Integer-based timestamps: the time interval should be an integer type. You must set the integer_now_func.

Returns

Column	Type	Description
job_id	INTEGER	TimescaleDB background job ID created to implement this policy

===== PAGE: https://docs.tigerdata.com/api/data-retention/remove_retention_policy/ =====

remove_retention_policy()

Remove a policy to drop chunks of a particular hypertable.

Samples

SELECT remove_retention_policy('conditions');

Removes the existing data retention policy for the conditions table.

Required arguments

Name	Type	Description
relation	REGCLASS	Name of the hypertable or continuous aggregate from which to remove the policy

Optional arguments

Name	Type	Description
if_exists	BOOLEAN	Set to true to avoid throwing an error if the policy does not exist. Defaults to false.

===== PAGE: https://docs.tigerdata.com/api/hypertable/create_table/ =====

CREATE TABLE

Create a hypertable partitioned on a single dimension with columnstore enabled, or create a standard Postgres relational table.

A hypertable is a specialized Postgres table that automatically partitions your data by time. All actions that work on a Postgres table, work on hypertables. For example, ALTER TABLE and SELECT. By default, a hypertable is partitioned on the time dimension. To add secondary dimensions to a hypertable, call add_dimension. To convert an existing relational table into a hypertable, call create_hypertable.

As the data cools and becomes more suited for analytics, add a columnstore policy so your data is automatically converted to the columnstore after a specific time interval. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads. You can also manually convert chunks in a hypertable to the columnstore.

Hypertable to hypertable foreign keys are not allowed, all other combinations are permitted.

The columnstore settings are applied on a per-chunk basis. You can change the settings by calling ALTER TABLE without first converting the entire hypertable back to the rowstore. The new settings apply only to the chunks that have not yet been converted to columnstore, the existing chunks in the columnstore do not change. Similarly, if you remove an existing columnstore policy and then add a new one, the new policy applies only to the unconverted chunks. This means that chunks with different columnstore settings can co-exist in the same hypertable.

TimescaleDB calculates default columnstore settings for each chunk when it is created. These settings apply to each chunk, and not the entire hypertable. To explicitly disable the defaults, set a setting to an empty string.

CREATE TABLE extends the standard Postgres CREATE TABLE. This page explains the features and arguments specific to TimescaleDB.

Since TimescaleDB v2.20.0

Samples

• Create a hypertable partitioned on the time dimension and enable columnstore:

  1. Create the hypertable:

  CREATE TABLE crypto_ticks (
     "time" TIMESTAMPTZ,
     symbol TEXT,
     price DOUBLE PRECISION,
     day_volume NUMERIC
  ) WITH (
    tsdb.hypertable,
    tsdb.partition_column='time',
    tsdb.segmentby='symbol',
    tsdb.orderby='time DESC'
  );
  

  1. Enable hypercore by adding a columnstore policy:

     CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '1d');
     

• Create a hypertable partitioned on the time with fewer chunks based on time interval:

  CREATE TABLE IF NOT EXISTS hypertable_control_chunk_interval(
   time int4 NOT NULL,
   device text,
   value float
  ) WITH (
   tsdb.hypertable,
   tsdb.partition_column='time',
   tsdb.chunk_interval=3453
  );
  

• Create a hypertable partitioned using UUIDv7:

   -- For optimal compression on the ID column, first enable UUIDv7 compression
   SET enable_uuid_compression=true;
   -- Then create your table
   CREATE TABLE events (
      id  uuid PRIMARY KEY DEFAULT generate_uuidv7(),
      payload jsonb
   ) WITH (tsdb.hypertable, tsdb.partition_column = 'id');
  

  -- For optimal compression on the ID column, first enable UUIDv7 compression
  SET enable_uuid_compression=true;
  -- Then create your table
  CREATE TABLE events (
     id  uuid PRIMARY KEY DEFAULT uuidv7(),
     payload jsonb
  ) WITH (tsdb.hypertable, tsdb.partition_column = 'id');
  

• Enable data compression during ingestion:

  When you set timescaledb.enable_direct_compress_copy your data gets compressed in memory during ingestion with COPY statements. By writing the compressed batches immediately in the columnstore, the IO footprint is significantly lower. Also, the columnstore policy you set is less important, INSERT already produces compressed chunks.

Please note that this feature is a tech preview and not production-ready. Using this feature could lead to regressed query performance and/or storage ratio, if the ingested batches are not correctly ordered or are of too high cardinality.

To enable in-memory data compression during ingestion:

SET timescaledb.enable_direct_compress_copy=on;

Important facts

• High cardinality use cases do not produce good batches and lead to degreaded query performance.

• The columnstore is optimized to store 1000 records per batch, which is the optimal format for ingestion per segment by.

• WAL records are written for the compressed batches rather than the individual tuples.

• Currently only COPY is support, INSERT will eventually follow.

• Best results are achieved for batch ingestion with 1000 records or more, upper boundary is 10.000 records.

• Continous Aggregates are not supported at the moment.

  1. Create a hypertable:

  CREATE TABLE t(time timestamptz, device text, value float) WITH (tsdb.hypertable,tsdb.partition_column='time');
  

  1. Copy data into the hypertable: You achieve the highest insert rate using binary format. CSV and text format are also supported.

  COPY t FROM '/tmp/t.binary' WITH (format binary);
  

• Create a Postgres relational table:

  CREATE TABLE IF NOT EXISTS relational_table(
   device text,
   value float
  );
  

Arguments

The syntax is:

CREATE TABLE  (
   -- Standard Postgres syntax for CREATE TABLE
)
WITH (
   tsdb.hypertable = true | false
   tsdb.partition_column = '<column_name> ',
   tsdb.chunk_interval = '<interval>'
   tsdb.create_default_indexes =  true | false
   tsdb.associated_schema = '<schema_name>',
   tsdb.associated_table_prefix = '<prefix>'
   tsdb.orderby = '<column_name> [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]',
   tsdb.segmentby = '<column_name> [, ...]',
   tsdb.sparse_index = '<index>(<column_name>), index(<column_name>)'
)

Name	Type	Default	Required	Description
tsdb.hypertable	BOOLEAN	true	✖	Create a new hypertable for time-series data rather than a standard Postgres relational table.
tsdb.partition_column	TEXT	true	✖	Set the time column to automatically partition your time-series data by.
tsdb.chunk_interval	TEXT	7 days	✖	Change this to better suit your needs. For example, if you set chunk_interval to 1 day, each chunk stores data from the same day. Data from different days is stored in different chunks.
tsdb.create_default_indexes	BOOLEAN	true	✖	Set to false to not automatically create indexes. The default indexes are: On all hypertables, a descending index on partition_column On hypertables with space partitions, an index on the space parameter and partition_column
tsdb.associated_schema	REGCLASS	_timescaledb_internal	✖	Set the schema name for internal hypertable tables.
tsdb.associated_table_prefix	TEXT	_hyper	✖	Set the prefix for the names of internal hypertable chunks.
tsdb.orderby	TEXT	Descending order on the time column in table_name.	✖	The order in which items are used in the columnstore. Specified in the same way as an ORDER BY clause in a SELECT query. Setting tsdb.orderby automatically creates an implicit min/max sparse index on the orderby column.
tsdb.segmentby	TEXT	TimescaleDB looks at pg_stats and determines an appropriate column based on the data cardinality and distribution. If pg_stats is not available, TimescaleDB looks for an appropriate column from the existing indexes.	✖	Set the list of columns used to segment data in the columnstore for table. An identifier representing the source of the data such as device_id or tags_id is usually a good candidate.
tsdb.sparse_index	TEXT	TimescaleDB evaluates the columns you already have indexed, checks which data types are a good fit for sparse indexing, then creates a sparse index as an optimization.	✖	Configure the sparse indexes for compressed chunks. Requires setting tsdb.orderby. Supported index types include: bloom(<column_name>): a probabilistic index, effective for = filters. Cannot be applied to tsdb.orderby columns. minmax(<column_name>): stores min/max values for each compressed chunk. Setting tsdb.orderby automatically creates an implicit min/max sparse index on the orderby column. Define multiple indexes using a comma-separated list. You can set only one index per column. Set to an empty string to avoid using sparse indexes and explicitly disable the default behavior.

Returns

TimescaleDB returns a simple message indicating success or failure.

===== PAGE: https://docs.tigerdata.com/api/hypertable/drop_chunks/ =====

drop_chunks()

Removes data chunks whose time range falls completely before (or after) a specified time. Shows a list of the chunks that were dropped, in the same style as the show_chunks function.

Chunks are constrained by a start and end time and the start time is always before the end time. A chunk is dropped if its end time is older than the older_than timestamp or, if newer_than is given, its start time is newer than the newer_than timestamp.

Note that, because chunks are removed if and only if their time range falls fully before (or after) the specified timestamp, the remaining data may still contain timestamps that are before (or after) the specified one.

Chunks can only be dropped based on their time intervals. They cannot be dropped based on a hash partition.

Samples

Drop all chunks from hypertable conditions older than 3 months:

SELECT drop_chunks('conditions', INTERVAL '3 months');

Example output:

              drop_chunks
----------------------------------------
 _timescaledb_internal._hyper_3_5_chunk
 _timescaledb_internal._hyper_3_6_chunk
 _timescaledb_internal._hyper_3_7_chunk
 _timescaledb_internal._hyper_3_8_chunk
 _timescaledb_internal._hyper_3_9_chunk
(5 rows)

Drop all chunks from hypertable conditions created before 3 months:

SELECT drop_chunks('conditions', created_before => now() -  INTERVAL '3 months');

Drop all chunks more than 3 months in the future from hypertable conditions. This is useful for correcting data ingested with incorrect clocks:

SELECT drop_chunks('conditions', newer_than => now() + interval '3 months');

Drop all chunks from hypertable conditions before 2017:

SELECT drop_chunks('conditions', '2017-01-01'::date);

Drop all chunks from hypertable conditions before 2017, where time column is given in milliseconds from the UNIX epoch:

SELECT drop_chunks('conditions', 1483228800000);

Drop all chunks older than 3 months ago and newer than 4 months ago from hypertable conditions:

SELECT drop_chunks('conditions', older_than => INTERVAL '3 months', newer_than => INTERVAL '4 months')

Drop all chunks created 3 months ago and created 4 months before from hypertable conditions:

SELECT drop_chunks('conditions', created_before => INTERVAL '3 months', created_after => INTERVAL '4 months')

Drop all chunks older than 3 months ago across all hypertables:

SELECT drop_chunks(format('%I.%I', hypertable_schema, hypertable_name)::regclass, INTERVAL '3 months')
  FROM timescaledb_information.hypertables;

Required arguments

Name	Type	Description
relation	REGCLASS	Hypertable or continuous aggregate from which to drop chunks.

Optional arguments

Name	Type	Description
older_than	ANY	Specification of cut-off point where any chunks older than this timestamp should be removed.
newer_than	ANY	Specification of cut-off point where any chunks newer than this timestamp should be removed.
verbose	BOOLEAN	Setting to true displays messages about the progress of the reorder command. Defaults to false.
created_before	ANY	Specification of cut-off point where any chunks created before this timestamp should be removed.
created_after	ANY	Specification of cut-off point where any chunks created after this timestamp should be removed.

The older_than and newer_than parameters can be specified in two ways:

• interval type: The cut-off point is computed as now() - older_than and similarly now() - newer_than. An error is returned if an INTERVAL is supplied and the time column is not one of a TIMESTAMP, TIMESTAMPTZ, or DATE.

• timestamp, date, or integer type: The cut-off point is explicitly given as a TIMESTAMP / TIMESTAMPTZ / DATE or as a SMALLINT / INT / BIGINT. The choice of timestamp or integer must follow the type of the hypertable's time column.

The created_before and created_after parameters can be specified in two ways:

• interval type: The cut-off point is computed as now() - created_before and similarly now() - created_after. This uses the chunk creation time relative to the current time for the filtering.

• timestamp, date, or integer type: The cut-off point is explicitly given as a TIMESTAMP / TIMESTAMPTZ / DATE or as a SMALLINT / INT / BIGINT. The choice of integer value must follow the type of the hypertable's partitioning column. Otherwise the chunk creation time is used for the filtering.

When using just an interval type, the function assumes that you are removing things in the past. If you want to remove data in the future, for example to delete erroneous entries, use a timestamp.

When both older_than and newer_than arguments are used, the function returns the intersection of the resulting two ranges. For example, specifying newer_than => 4 months and older_than => 3 months drops all chunks between 3 and 4 months old. Similarly, specifying newer_than => '2017-01-01' and older_than => '2017-02-01' drops all chunks between '2017-01-01' and '2017-02-01'. Specifying parameters that do not result in an overlapping intersection between two ranges results in an error.

When both created_before and created_after arguments are used, the function returns the intersection of the resulting two ranges. For example, specifying created_after => 4 monthsandcreated_before=> 3 months drops all chunks created between 3 and 4 months from now. Similarly, specifying created_after=> '2017-01-01'andcreated_before => '2017-02-01' drops all chunks created between '2017-01-01' and '2017-02-01'. Specifying parameters that do not result in an overlapping intersection between two ranges results in an error.

The created_before/created_after parameters cannot be used together with older_than/newer_than.

===== PAGE: https://docs.tigerdata.com/api/hypertable/detach_chunk/ =====

detach_chunk()

Separate a chunk from a hypertable.

chunk becomes a standalone hypertable with the same name and schema. All existing constraints and indexes on chunk are preserved after detaching. Foreign keys are dropped.

In this initial release, you cannot detach a chunk that has been converted to the columnstore.

Since TimescaleDB v2.21.0

Samples

Detach a chunk from a hypertable:

CALL detach_chunk('_timescaledb_internal._hyper_1_2_chunk');

Arguments

Name	Type	Description
chunk	REGCLASS	Name of the chunk to detach.

Returns

This function returns void.

===== PAGE: https://docs.tigerdata.com/api/hypertable/attach_tablespace/ =====

attach_tablespace()

Attach a tablespace to a hypertable and use it to store chunks. A tablespace is a directory on the filesystem that allows control over where individual tables and indexes are stored on the filesystem. A common use case is to create a tablespace for a particular storage disk, allowing tables to be stored there. To learn more, see the Postgres documentation on tablespaces.

TimescaleDB can manage a set of tablespaces for each hypertable, automatically spreading chunks across the set of tablespaces attached to a hypertable. If a hypertable is hash partitioned, TimescaleDB tries to place chunks that belong to the same partition in the same tablespace. Changing the set of tablespaces attached to a hypertable may also change the placement behavior. A hypertable with no attached tablespaces has its chunks placed in the database's default tablespace.

Samples

Attach the tablespace disk1 to the hypertable conditions:

SELECT attach_tablespace('disk1', 'conditions');
SELECT attach_tablespace('disk2', 'conditions', if_not_attached => true);

Required arguments

Name	Type	Description
tablespace	TEXT	Name of the tablespace to attach.
hypertable	REGCLASS	Hypertable to attach the tablespace to.

Tablespaces need to be created before being attached to a hypertable. Once created, tablespaces can be attached to multiple hypertables simultaneously to share the underlying disk storage. Associating a regular table with a tablespace using the TABLESPACE option to CREATE TABLE, prior to calling create_hypertable, has the same effect as calling attach_tablespace immediately following create_hypertable.

Optional arguments

Name	Type	Description
if_not_attached	BOOLEAN	Set to true to avoid throwing an error if the tablespace is already attached to the table. A notice is issued instead. Defaults to false.

===== PAGE: https://docs.tigerdata.com/api/hypertable/hypertable_size/ =====

hypertable_size()

hypertable_size()

Get the total disk space used by a hypertable or continuous aggregate, that is, the sum of the size for the table itself including chunks, any indexes on the table, and any toast tables. The size is reported in bytes. This is equivalent to computing the sum of total_bytes column from the output of hypertable_detailed_size function.

When a continuous aggregate name is provided, the function transparently looks up the backing hypertable and returns its statistics instead.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get the size information for a hypertable.

SELECT hypertable_size('devices');

 hypertable_size
-----------------
           73728

Get the size information for all hypertables.

SELECT hypertable_name, hypertable_size(format('%I.%I', hypertable_schema, hypertable_name)::regclass)
  FROM timescaledb_information.hypertables;

Get the size information for a continuous aggregate.

SELECT hypertable_size('device_stats_15m');

 hypertable_size
-----------------
           73728

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable or continuous aggregate to show size of.

Returns

Name	Type	Description
hypertable_size	BIGINT	Total disk space used by the specified hypertable, including all indexes and TOAST data

NULL is returned if the function is executed on a non-hypertable relation.

===== PAGE: https://docs.tigerdata.com/api/hypertable/hypertable_approximate_size/ =====

hypertable_approximate_size()

Get the approximate total disk space used by a hypertable or continuous aggregate, that is, the sum of the size for the table itself including chunks, any indexes on the table, and any toast tables. The size is reported in bytes. This is equivalent to computing the sum of total_bytes column from the output of hypertable_approximate_detailed_size function.

When a continuous aggregate name is provided, the function transparently looks up the backing hypertable and returns its statistics instead.

This function relies on the per backend caching using the in-built Postgres storage manager layer to compute the approximate size cheaply. The PG cache invalidation clears off the cached size for a chunk when DML happens into it. That size cache is thus able to get the latest size in a matter of minutes. Also, due to the backend caching, any long running session will only fetch latest data for new or modified chunks and can use the cached data (which is calculated afresh the first time around) effectively for older chunks. Thus it is recommended to use a single connected Postgres backend session to compute the approximate sizes of hypertables to get faster results.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get the approximate size information for a hypertable.

SELECT * FROM hypertable_approximate_size('devices');
 hypertable_approximate_size
-----------------------------
                        8192

Get the approximate size information for all hypertables.

SELECT hypertable_name, hypertable_approximate_size(format('%I.%I', hypertable_schema, hypertable_name)::regclass)
  FROM timescaledb_information.hypertables;

Get the approximate size information for a continuous aggregate.

SELECT hypertable_approximate_size('device_stats_15m');

 hypertable_approximate_size
-----------------------------
                        8192

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable or continuous aggregate to show size of.

Returns

Name	Type	Description
hypertable_approximate_size	BIGINT	Total approximate disk space used by the specified hypertable, including all indexes and TOAST data

NULL is returned if the function is executed on a non-hypertable relation.

===== PAGE: https://docs.tigerdata.com/api/hypertable/split_chunk/ =====

split_chunk()

Split a large chunk at a specific point in time. If you do not specify the timestamp to split at, chunk is split equally.

Samples

• Split a chunk at a specific time:

  CALL split_chunk('chunk_1', split_at => '2025-03-01 00:00');
  

• Split a chunk in two:

  For example, If the chunk duration is, 24 hours, the following command splits chunk_1 into two chunks of 12 hours each.

  CALL split_chunk('chunk_1');
  

Required arguments

Name	Type	Required	Description
chunk	REGCLASS	✔	Name of the chunk to split.
split_at	TIMESTAMPTZ	✖	Timestamp to split the chunk at.

Returns

This function returns void.

===== PAGE: https://docs.tigerdata.com/api/hypertable/attach_chunk/ =====

attach_chunk()

Attach a hypertable as a chunk in another hypertable at a given slice in a dimension.

The schema, name, existing constraints, and indexes of chunk do not change, even if a constraint conflicts with a chunk constraint in hypertable.

The hypertable you attach chunk to does not need to have the same dimension columns as the hypertable you previously detached chunk from.

While attaching chunk to hypertable:

• Dimension columns in chunk are set as NOT NULL.
• Any foreign keys in hypertable are created in chunk.

You cannot:

• Attaching a chunk that is still attached to another hypertable. First call detach_chunk.
• Attaching foreign tables are not supported.

Since TimescaleDB v2.21.0

Samples

Attach a hypertable as a chunk in another hypertable for a specific slice in a dimension:

CALL attach_chunk('ht', '_timescaledb_internal._hyper_1_2_chunk', '{"device_id": [0, 1000]}');

Arguments

Name	Type	Description
hypertable	REGCLASS	Name of the hypertable to attach chunk to.
chunk	REGCLASS	Name of the chunk to attach.
slices	JSONB	The slice chunk will occupy in hypertable. slices cannot clash with the slice already occupied by an existing chunk in hypertable.

Returns

This function returns void.

===== PAGE: https://docs.tigerdata.com/api/hypertable/detach_tablespaces/ =====

detach_tablespaces()

Detach all tablespaces from a hypertable. After issuing this command on a hypertable, it no longer has any tablespaces attached to it. New chunks are instead placed in the database's default tablespace.

Samples

Detach all tablespaces from the hypertable conditions:

SELECT detach_tablespaces('conditions');

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable to detach a the tablespace from.

===== PAGE: https://docs.tigerdata.com/api/hypertable/create_hypertable/ =====

create_hypertable()

Replace a standard Postgres relational table with a hypertable that is partitioned on a single dimension. To create a new hypertable, best practice is to call CREATE TABLE.

A hypertable is a Postgres table that automatically partitions your data by time. A dimension defines the way your data is partitioned. All actions work on the resulting hypertable. For example, ALTER TABLE, and SELECT.

If the table to convert already contains data, set migrate_data to TRUE. However, this may take a long time and there are limitations when the table contains foreign key constraints.

You cannot run create_hypertable() on a table that is already partitioned using declarative partitioning or inheritance. The time column must be defined as NOT NULL. If this is not already specified on table creation, create_hypertable automatically adds this constraint on the table when it is executed.

This page describes the generalized hypertable API introduced in TimescaleDB v2.13. The old interface for create_hypertable is also available.

Samples

Before you call create_hypertable, you create a standard Postgres relational table. For example:

CREATE TABLE conditions (
   time        TIMESTAMPTZ         NOT NULL,
   location    text                NOT NULL,
   temperature DOUBLE PRECISION    NULL
);

The following examples show you how to create a hypertable from an existing table or a function:

• Time partition a hypertable by time range
• Time partition a hypertable using composite columns and immutable functions
• Time partition a hypertable using ISO formatting
• Time partition a hypertable using UUIDv7

Time partition a hypertable by time range

The following examples show different ways to create a hypertable:

• Convert with range partitioning on the time column:

  SELECT create_hypertable('conditions', by_range('time'));
  

• Convert with a set_chunk_time_interval of 24 hours: Either:

  SELECT create_hypertable('conditions', by_range('time', 86400000000));
  

  or:

  SELECT create_hypertable('conditions', by_range('time', INTERVAL '1 day'));
  

• with range partitioning on the time column, do not raise a warning if conditions is already a hypertable:

  SELECT create_hypertable('conditions', by_range('time'), if_not_exists => TRUE);
  

If you call SELECT * FROM create_hypertable(...) the return value is formatted as a table with column headings.

Time partition a hypertable using composite columns and immutable functions

The following example shows how to time partition the measurements relational table on a composite column type using a range partitioning function.

1. Create the report type, then an immutable function that converts the column value into a supported column value:

   CREATE TYPE report AS (reported timestamp with time zone, contents jsonb);
   
   CREATE FUNCTION report_reported(report)
     RETURNS timestamptz
     LANGUAGE SQL
     IMMUTABLE AS
     'SELECT $1.reported';
   

2. Create the hypertable using the immutable function:

   SELECT create_hypertable('measurements', by_range('report', partition_func => 'report_reported'));
   

Time partition a hypertable using ISO formatting

The following example shows how to time partition the events table on a jsonb (event) column type, which has a top level started key that contains an ISO 8601 formatted timestamp:

CREATE FUNCTION event_started(jsonb)
    RETURNS timestamptz
    LANGUAGE SQL
    IMMUTABLE AS
  $func$SELECT ($1->>'started')::timestamptz$func$;

SELECT create_hypertable('events', by_range('event', partition_func => 'event_started'));

Time partition a hypertable using UUIDv7:

1. Create a table with a UUIDv7 column:

   CREATE TABLE events (
       id  uuid PRIMARY KEY DEFAULT generate_uuidv7(),
       payload jsonb
   );
   

   CREATE TABLE events (
       id  uuid PRIMARY KEY DEFAULT uuidv7(),
       payload jsonb
   );
   

2. Partition the table based on the timestamps embedded within the UUID values:

   SELECT create_hypertable(
       'events',
       by_range('id', INTERVAL '1 month')
   );
   

Subsequent data insertion and queries automatically leverage the UUIDv7-based partitioning.

Arguments

Name	Type	Default	Required	Description
create_default_indexes	BOOLEAN	TRUE	✖	Create default indexes on time/partitioning columns.
dimension	DIMENSION_INFO	-	✔	To create a _timescaledb_internal.dimension_info instance to partition a hypertable, you call by_range and by_hash.
if_not_exists	BOOLEAN	FALSE	✖	Set to TRUE to print a warning if relation is already a hypertable. By default, an exception is raised.
migrate_data	BOOLEAN	FALSE	✖	Set to TRUE to migrate any existing data in relation in to chunks in the new hypertable. Depending on the amount of data to be migrated, setting migrate_data can lock the table for a significant amount of time. If there are foreign key constraints to other tables in the data to be migrated, create_hypertable() can run into deadlock. A hypertable can only contain foreign keys to another hypertable. UNIQUE and PRIMARY constraints must include the partitioning key. Deadlock may happen when concurrent transactions simultaneously try to insert data into tables that are referenced in the foreign key constraints, and into the converting table itself. To avoid deadlock, manually obtain a SHARE ROW EXCLUSIVE lock on the referenced tables before you call create_hypertable in the same transaction. If you leave migrate_data set to the default, non-empty tables generate an error when you call create_hypertable.
relation	REGCLASS	-	✔	Identifier of the table to convert to a hypertable.

Dimension info

To create a _timescaledb_internal.dimension_info instance, you call add_dimension to an existing hypertable.

Samples

Hypertables must always have a primary range dimension, followed by an arbitrary number of additional dimensions that can be either range or hash, Typically this is just one hash. For example:

SELECT add_dimension('conditions', by_range('time'));
SELECT add_dimension('conditions', by_hash('location', 2));

For incompatible data types such as jsonb, you can specify a function to the partition_func argument of the dimension build to extract a compatible data type. Look in the example section below.

Custom partitioning

By default, TimescaleDB calls Postgres's internal hash function for the given type. You use a custom partitioning function for value types that do not have a native Postgres hash function.

You can specify a custom partitioning function for both range and hash partitioning. A partitioning function should take a anyelement argument as the only parameter and return a positive integer hash value. This hash value is not a partition identifier, but rather the inserted value's position in the dimension's key space, which is then divided across the partitions.

by_range()

Create a by-range dimension builder. You can partition by_range on it's own.

Samples

• Partition on time using CREATE TABLE

  The simplest usage is to partition on a time column:

  CREATE TABLE conditions (
     time        TIMESTAMPTZ       NOT NULL,
     location    TEXT              NOT NULL,
     device      TEXT              NOT NULL,
     temperature DOUBLE PRECISION  NULL,
     humidity    DOUBLE PRECISION  NULL
  ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='time'
  );
  

  If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

  This is the default partition, you do not need to add it explicitly.

• Extract time from a non-time column using create_hypertable

  If you have a table with a non-time column containing the time, such as a JSON column, add a partition function to extract the time:

  CREATE TABLE my_table (
     metric_id serial not null,
     data jsonb,
  );
  
  CREATE FUNCTION get_time(jsonb) RETURNS timestamptz AS $$
    SELECT ($1->>'time')::timestamptz
  $$ LANGUAGE sql IMMUTABLE;
  
  SELECT create_hypertable('my_table', by_range('data', '1 day', 'get_time'));
  

Arguments

Name	Type	Default	Required	Description
column_name	NAME	-	✔	Name of column to partition on.
partition_func	REGPROC	-	✖	The function to use for calculating the partition of a value.
partition_interval	ANYELEMENT	-	✖	Interval to partition column on.

If the column to be partitioned is a:

• TIMESTAMP, TIMESTAMPTZ, or DATE: specify partition_interval either as an INTERVAL type or an integer value in microseconds.

• Another integer type: specify partition_interval as an integer that reflects the column's underlying semantics. For example, if this column is in UNIX time, specify partition_interval in milliseconds.

The partition type and default value depending on column type is:

Column Type	Partition Type	Default value
TIMESTAMP WITHOUT TIMEZONE	INTERVAL/INTEGER	1 week
TIMESTAMP WITH TIMEZONE	INTERVAL/INTEGER	1 week
DATE	INTERVAL/INTEGER	1 week
SMALLINT	SMALLINT	10000
INT	INT	100000
BIGINT	BIGINT	1000000

by_hash()

The main purpose of hash partitioning is to enable parallelization across multiple disks within the same time interval. Every distinct item in hash partitioning is hashed to one of N buckets. By default, TimescaleDB uses flexible range intervals to manage chunk sizes.

Parallelizing disk I/O

You use Parallel I/O in the following scenarios:

• Two or more concurrent queries should be able to read from different disks in parallel.
• A single query should be able to use query parallelization to read from multiple disks in parallel.

For the following options:

• RAID: use a RAID setup across multiple physical disks, and expose a single logical disk to the hypertable. That is, using a single tablespace.

  Best practice is to use RAID when possible, as you do not need to manually manage tablespaces in the database.

• Multiple tablespaces: for each physical disk, add a separate tablespace to the database. TimescaleDB allows you to add multiple tablespaces to a single hypertable. However, although under the hood, a hypertable's chunks are spread across the tablespaces associated with that hypertable.

  When using multiple tablespaces, a best practice is to also add a second hash-partitioned dimension to your hypertable and to have at least one hash partition per disk. While a single time dimension would also work, it would mean that the first chunk is written to one tablespace, the second to another, and so on, and thus would parallelize only if a query's time range exceeds a single chunk.

When adding a hash partitioned dimension, set the number of partitions to a multiple of number of disks. For example, the number of partitions P=N*Pd where N is the number of disks and Pd is the number of partitions per disk. This enables you to add more disks later and move partitions to the new disk from other disks.

TimescaleDB does not benefit from a very large number of hash partitions, such as the number of unique items you expect in partition field. A very large number of hash partitions leads both to poorer per-partition load balancing (the mapping of items to partitions using hashing), as well as much increased planning latency for some types of queries.

Samples

CREATE TABLE conditions (
   "time"      TIMESTAMPTZ       NOT NULL,
   location    TEXT              NOT NULL,
   device      TEXT              NOT NULL,
   temperature DOUBLE PRECISION  NULL,
   humidity    DOUBLE PRECISION  NULL
) WITH (
   tsdb.hypertable,
   tsdb.partition_column='time',
   tsdb.chunk_interval='1 day'
);

SELECT add_dimension('conditions', by_hash('location', 2));

Arguments

Name	Type	Default	Required	Description
column_name	NAME	-	✔	Name of column to partition on.
partition_func	REGPROC	-	✖	The function to use to calcule the partition of a value.
number_partitions	ANYELEMENT	-	✔	Number of hash partitions to use for partitioning_column. Must be greater than 0.

Returns

by_range and by-hash return an opaque _timescaledb_internal.dimension_info instance, holding the dimension information used by this function.

Returns

Column	Type	Description
hypertable_id	INTEGER	The ID of the hypertable you created.
created	BOOLEAN	TRUE when the hypertable is created. FALSE when if_not_exists is true and no hypertable was created.

===== PAGE: https://docs.tigerdata.com/api/hypertable/move_chunk/ =====

move_chunk()

TimescaleDB allows you to move data and indexes to different tablespaces. This allows you to move data to more cost-effective storage as it ages.

The move_chunk function acts like a combination of the Postgres CLUSTER command and Postgres ALTER TABLE...SET TABLESPACE commands. Unlike these Postgres commands, however, the move_chunk function uses lower lock levels so that the chunk and hypertable are able to be read for most of the process. This comes at a cost of slightly higher disk usage during the operation. For a more detailed discussion of this capability, see the documentation on managing storage with tablespaces.

You must be logged in as a super user, such as the postgres user, to use the move_chunk() call.

Samples

SELECT move_chunk(
  chunk => '_timescaledb_internal._hyper_1_4_chunk',
  destination_tablespace => 'tablespace_2',
  index_destination_tablespace => 'tablespace_3',
  reorder_index => 'conditions_device_id_time_idx',
  verbose => TRUE
);

Required arguments

Name	Type	Description
chunk	REGCLASS	Name of chunk to be moved
destination_tablespace	NAME	Target tablespace for chunk being moved
index_destination_tablespace	NAME	Target tablespace for index associated with the chunk you are moving

Optional arguments

Name	Type	Description
reorder_index	REGCLASS	The name of the index (on either the hypertable or chunk) to order by
verbose	BOOLEAN	Setting to true displays messages about the progress of the move_chunk command. Defaults to false.

===== PAGE: https://docs.tigerdata.com/api/hypertable/hypertable_index_size/ =====

hypertable_index_size()

Get the disk space used by an index on a hypertable, including the disk space needed to provide the index on all chunks. The size is reported in bytes.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get size of a specific index on a hypertable.

\d conditions_table
                     Table "public.conditions_table"
 Column |           Type           | Collation | Nullable | Default
--------+--------------------------+-----------+----------+---------
 time   | timestamp with time zone |           | not null |
 device | integer                  |           |          |
 volume | integer                  |           |          |
Indexes:
    "second_index" btree ("time")
    "test_table_time_idx" btree ("time" DESC)
    "third_index" btree ("time")

SELECT hypertable_index_size('second_index');

 hypertable_index_size
-----------------------
                163840

SELECT pg_size_pretty(hypertable_index_size('second_index'));

 pg_size_pretty
----------------
 160 kB

Required arguments

Name	Type	Description
index_name	REGCLASS	Name of the index on a hypertable

Returns

Column	Type	Description
hypertable_index_size	BIGINT	Returns the disk space used by the index

NULL is returned if the function is executed on a non-hypertable relation.

===== PAGE: https://docs.tigerdata.com/api/hypertable/enable_chunk_skipping/ =====

enable_chunk_skipping()

Early access: TimescaleDB v2.17.1

Enable range statistics for a specific column in a compressed hypertable. This tracks a range of values for that column per chunk. Used for chunk skipping during query optimization and applies only to the chunks created after chunk skipping is enabled.

Best practice is to enable range tracking on columns that are correlated to the partitioning column. In other words, enable tracking on secondary columns which are referenced in the WHERE clauses in your queries.

TimescaleDB supports min/max range tracking for the smallint, int, bigint, serial, bigserial, date, timestamp, and timestamptz data types. The min/max ranges are calculated when a chunk belonging to this hypertable is compressed using the compress_chunk function. The range is stored in start (inclusive) and end (exclusive) form in the chunk_column_stats catalog table.

This way you store the min/max values for such columns in this catalog table at the per-chunk level. These min/max range values do not participate in partitioning of the data. These ranges are used for chunk skipping when the WHERE clause of an SQL query specifies ranges on the column.

A DROP COLUMN on a column with statistics tracking enabled on it ends up removing all relevant entries from the catalog table.

A decompress_chunk invocation on a compressed chunk resets its entries from the chunk_column_stats catalog table since now it's available for DML and the min/max range values can change on any further data manipulation in the chunk.

By default, this feature is disabled. To enable chunk skipping, set timescaledb.enable_chunk_skipping = on in postgresql.conf. When you upgrade from a database instance that uses compression but does not support chunk skipping, you need to recompress the previously compressed chunks for chunk skipping to work.

Samples

In this sample, you create the conditions hypertable with partitioning on the time column. You then specify and enable additional columns to track ranges for.

CREATE TABLE conditions (
   time        TIMESTAMPTZ       NOT NULL,
   location    TEXT              NOT NULL,
   device      TEXT              NOT NULL,
   temperature DOUBLE PRECISION  NULL,
   humidity    DOUBLE PRECISION  NULL
) WITH (
   tsdb.hypertable,
   tsdb.partition_column='time'
);

SELECT enable_chunk_skipping('conditions', 'device_id');

If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

Arguments

Name	Type	Default	Required	Description
column_name	TEXT	-	✔	Column to track range statistics for
hypertable	REGCLASS	-	✔	Hypertable that the column belongs to
if_not_exists	BOOLEAN	false	✖	Set to true so that a notice is sent when ranges are not being tracked for a column. By default, an error is thrown

Returns

Column	Type	Description
column_stats_id	INTEGER	ID of the entry in the TimescaleDB internal catalog
enabled	BOOLEAN	Returns true when tracking is enabled, if_not_exists is true, and when a new entry is not added

===== PAGE: https://docs.tigerdata.com/api/hypertable/detach_tablespace/ =====

detach_tablespace()

Detach a tablespace from one or more hypertables. This only means that new chunks are not placed on the detached tablespace. This is useful, for instance, when a tablespace is running low on disk space and one would like to prevent new chunks from being created in the tablespace. The detached tablespace itself and any existing chunks with data on it remains unchanged and continue to work as before, including being available for queries. Note that newly inserted data rows may still be inserted into an existing chunk on the detached tablespace since existing data is not cleared from a detached tablespace. A detached tablespace can be reattached if desired to once again be considered for chunk placement.

Samples

Detach the tablespace disk1 from the hypertable conditions:

SELECT detach_tablespace('disk1', 'conditions');
SELECT detach_tablespace('disk2', 'conditions', if_attached => true);

Detach the tablespace disk1 from all hypertables that the current user has permissions for:

SELECT detach_tablespace('disk1');

Required arguments

Name	Type	Description
tablespace	TEXT	Tablespace to detach.

When giving only the tablespace name as argument, the given tablespace is detached from all hypertables that the current role has the appropriate permissions for. Therefore, without proper permissions, the tablespace may still receive new chunks after this command is issued.

Optional arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable to detach a the tablespace from.
if_attached	BOOLEAN	Set to true to avoid throwing an error if the tablespace is not attached to the given table. A notice is issued instead. Defaults to false.

When specifying a specific hypertable, the tablespace is only detached from the given hypertable and thus may remain attached to other hypertables.

===== PAGE: https://docs.tigerdata.com/api/hypertable/chunks_detailed_size/ =====

chunks_detailed_size()

Get information about the disk space used by the chunks belonging to a hypertable, returning size information for each chunk table, any indexes on the chunk, any toast tables, and the total size associated with the chunk. All sizes are reported in bytes.

If the function is executed on a distributed hypertable, it returns disk space usage information as a separate row per node. The access node is not included since it doesn't have any local chunk data.

Additional metadata associated with a chunk can be accessed via the timescaledb_information.chunks view.

Samples

SELECT * FROM chunks_detailed_size('dist_table')
  ORDER BY chunk_name, node_name;

     chunk_schema      |      chunk_name       | table_bytes | index_bytes | toast_bytes | total_bytes |       node_name
-----------------------+-----------------------+-------------+-------------+-------------+-------------+-----------------------
 _timescaledb_internal | _dist_hyper_1_1_chunk |        8192 |       32768 |           0 |       40960 | data_node_1
 _timescaledb_internal | _dist_hyper_1_2_chunk |        8192 |       32768 |           0 |       40960 | data_node_2
 _timescaledb_internal | _dist_hyper_1_3_chunk |        8192 |       32768 |           0 |       40960 | data_node_3

Required arguments

Name	Type	Description
hypertable	REGCLASS	Name of the hypertable

Returns

Column	Type	Description
chunk_schema	TEXT	Schema name of the chunk
chunk_name	TEXT	Name of the chunk
table_bytes	BIGINT	Disk space used by the chunk table
index_bytes	BIGINT	Disk space used by indexes
toast_bytes	BIGINT	Disk space of toast tables
total_bytes	BIGINT	Total disk space used by the chunk, including all indexes and TOAST data
node_name	TEXT	Node for which size is reported, applicable only to distributed hypertables

If executed on a relation that is not a hypertable, the function returns NULL.

===== PAGE: https://docs.tigerdata.com/api/hypertable/create_hypertable_old/ =====

create_hypertable()

This page describes the hypertable API supported prior to TimescaleDB v2.13. Best practice is to use the new create_hypertable interface.

Creates a TimescaleDB hypertable from a Postgres table (replacing the latter), partitioned on time and with the option to partition on one or more other columns. The Postgres table cannot be an already partitioned table (declarative partitioning or inheritance). In case of a non-empty table, it is possible to migrate the data during hypertable creation using the migrate_data option, although this might take a long time and has certain limitations when the table contains foreign key constraints (see below).

After creation, all actions, such as ALTER TABLE, SELECT, etc., still work on the resulting hypertable.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Convert table conditions to hypertable with just time partitioning on column time:

SELECT create_hypertable('conditions', 'time');

Convert table conditions to hypertable, setting chunk_time_interval to 24 hours.

SELECT create_hypertable('conditions', 'time', chunk_time_interval => 86400000000);
SELECT create_hypertable('conditions', 'time', chunk_time_interval => INTERVAL '1 day');

Convert table conditions to hypertable. Do not raise a warning if conditions is already a hypertable:

SELECT create_hypertable('conditions', 'time', if_not_exists => TRUE);

Time partition table measurements on a composite column type report using a time partitioning function. Requires an immutable function that can convert the column value into a supported column value:

CREATE TYPE report AS (reported timestamp with time zone, contents jsonb);

CREATE FUNCTION report_reported(report)
  RETURNS timestamptz
  LANGUAGE SQL
  IMMUTABLE AS
  'SELECT $1.reported';

SELECT create_hypertable('measurements', 'report', time_partitioning_func => 'report_reported');

Time partition table events, on a column type jsonb (event), which has a top level key (started) containing an ISO 8601 formatted timestamp:

CREATE FUNCTION event_started(jsonb)
  RETURNS timestamptz
  LANGUAGE SQL
  IMMUTABLE AS
  $func$SELECT ($1->>'started')::timestamptz$func$;

SELECT create_hypertable('events', 'event', time_partitioning_func => 'event_started');

Required arguments

Name	Type	Description
relation	REGCLASS	Identifier of table to convert to hypertable.
time_column_name	REGCLASS	Name of the column containing time values as well as the primary column to partition by.

Optional arguments

Name	Type	Description
partitioning_column	REGCLASS	Name of an additional column to partition by. If provided, the number_partitions argument must also be provided.
number_partitions	INTEGER	Number of hash partitions to use for partitioning_column. Must be > 0.
chunk_time_interval	INTERVAL	Event time that each chunk covers. Must be > 0. Default is 7 days.
create_default_indexes	BOOLEAN	Whether to create default indexes on time/partitioning columns. Default is TRUE.
if_not_exists	BOOLEAN	Whether to print warning if table already converted to hypertable or raise exception. Default is FALSE.
partitioning_func	REGCLASS	The function to use for calculating a value's partition.
associated_schema_name	REGCLASS	Name of the schema for internal hypertable tables. Default is _timescaledb_internal.
associated_table_prefix	TEXT	Prefix for internal hypertable chunk names. Default is _hyper.
migrate_data	BOOLEAN	Set to TRUE to migrate any existing data from the relation table to chunks in the new hypertable. A non-empty table generates an error without this option. Large tables may take significant time to migrate. Defaults to FALSE.
time_partitioning_func	REGCLASS	Function to convert incompatible primary time column values to compatible ones. The function must be IMMUTABLE.
replication_factor	INTEGER	Replication factor to use with distributed hypertable. If not provided, value is determined by the timescaledb.hypertable_replication_factor_default GUC.
data_nodes	ARRAY	This is the set of data nodes that are used for this table if it is distributed. This has no impact on non-distributed hypertables. If no data nodes are specified, a distributed hypertable uses all data nodes known by this instance.
distributed	BOOLEAN	Set to TRUE to create distributed hypertable. If not provided, value is determined by the timescaledb.hypertable_distributed_default GUC. When creating a distributed hypertable, consider using create_distributed_hypertable in place of create_hypertable. Default is NULL.

Returns

Column	Type	Description
hypertable_id	INTEGER	ID of the hypertable in TimescaleDB.
schema_name	TEXT	Schema name of the table converted to hypertable.
table_name	TEXT	Table name of the table converted to hypertable.
created	BOOLEAN	TRUE if the hypertable was created, FALSE when if_not_exists is true and no hypertable was created.

If you use SELECT * FROM create_hypertable(...) you get the return value formatted as a table with column headings.

The use of the migrate_data argument to convert a non-empty table can lock the table for a significant amount of time, depending on how much data is in the table. It can also run into deadlock if foreign key constraints exist to other tables.

When converting a normal SQL table to a hypertable, pay attention to how you handle constraints. A hypertable can contain foreign keys to normal SQL table columns, but the reverse is not allowed. UNIQUE and PRIMARY constraints must include the partitioning key.

The deadlock is likely to happen when concurrent transactions simultaneously try to insert data into tables that are referenced in the foreign key constraints and into the converting table itself. The deadlock can be prevented by manually obtaining SHARE ROW EXCLUSIVE lock on the referenced tables before calling create_hypertable in the same transaction, see Postgres documentation for the syntax.

Units

The time column supports the following data types:

Description	Types
Timestamp	TIMESTAMP, TIMESTAMPTZ
Date	DATE
Integer	SMALLINT, INT, BIGINT

The type flexibility of the 'time' column allows the use of non-time-based values as the primary chunk partitioning column, as long as those values can increment.

For incompatible data types (for example, jsonb) you can specify a function to the time_partitioning_func argument which can extract a compatible data type.

The units of chunk_time_interval should be set as follows:

• For time columns having timestamp or DATE types, the chunk_time_interval should be specified either as an interval type or an integral value in microseconds.
• For integer types, the chunk_time_interval must be set explicitly, as the database does not otherwise understand the semantics of what each integer value represents (a second, millisecond, nanosecond, etc.). So if your time column is the number of milliseconds since the UNIX epoch, and you wish to have each chunk cover 1 day, you should specify chunk_time_interval => 86400000.

In case of hash partitioning (in other words, if number_partitions is greater than zero), it is possible to optionally specify a custom partitioning function. If no custom partitioning function is specified, the default partitioning function is used. The default partitioning function calls Postgres's internal hash function for the given type, if one exists. Thus, a custom partitioning function can be used for value types that do not have a native Postgres hash function. A partitioning function should take a single anyelement type argument and return a positive integer hash value. Note that this hash value is not a partition ID, but rather the inserted value's position in the dimension's key space, which is then divided across the partitions.

The time column in create_hypertable must be defined as NOT NULL. If this is not already specified on table creation, create_hypertable automatically adds this constraint on the table when it is executed.

===== PAGE: https://docs.tigerdata.com/api/hypertable/set_chunk_time_interval/ =====

set_chunk_time_interval()

Sets the chunk_time_interval on a hypertable. The new interval is used when new chunks are created, and time intervals on existing chunks are not changed.

Samples

For a TIMESTAMP column, set chunk_time_interval to 24 hours:

SELECT set_chunk_time_interval('conditions', INTERVAL '24 hours');
SELECT set_chunk_time_interval('conditions', 86400000000);

For a time column expressed as the number of milliseconds since the UNIX epoch, set chunk_time_interval to 24 hours:

SELECT set_chunk_time_interval('conditions', 86400000);

Arguments

Name	Type	Default	Required	Description
hypertable	REGCLASS	-	✔	Hypertable or continuous aggregate to update interval for.
chunk_time_interval	See note	-	✔	Event time that each new chunk covers.
dimension_name	REGCLASS	-	✖	The name of the time dimension to set the number of partitions for. Only use dimension_name when your hypertable has multiple time dimensions.

If you change chunk time interval you may see a chunk that is smaller than the new interval. For example, if you have two 7-day chunks that cover 14 days, then change chunk_time_interval to 3 days, you may end up with a transition chunk covering one day. This happens because the start and end of the new chunk is calculated based on dividing the timeline by the chunk_time_interval starting at epoch 0. This leads to the following chunks [0, 3), [3, 6), [6, 9), [9, 12), [12, 15), [15, 18) and so on. The two 7-day chunks covered data up to day 14: [0, 7), [8, 14), so the 3-day chunk for [12, 15) is reduced to a one day chunk. The following chunk [15, 18) is created as a full 3 day chunk.

The valid types for the chunk_time_interval depend on the type used for the hypertable time column:

time column type	chunk_time_interval type	Time unit
TIMESTAMP	INTERVAL	days, hours, minutes, etc
	INTEGER or BIGINT	microseconds
TIMESTAMPTZ	INTERVAL	days, hours, minutes, etc
	INTEGER or BIGINT	microseconds
DATE	INTERVAL	days, hours, minutes, etc
	INTEGER or BIGINT	microseconds
SMALLINT	SMALLINT	The same time unit as the time column
INT	INT	The same time unit as the time column
BIGINT	BIGINT	The same time unit as the time column

For more information, see hypertable partitioning.

===== PAGE: https://docs.tigerdata.com/api/hypertable/show_tablespaces/ =====

show_tablespaces()

Show the tablespaces attached to a hypertable.

Samples

SELECT * FROM show_tablespaces('conditions');

 show_tablespaces
------------------
 disk1
 disk2

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable to show attached tablespaces for.

===== PAGE: https://docs.tigerdata.com/api/hypertable/disable_chunk_skipping/ =====

disable_chunk_skipping()

Disable range tracking for a specific column in a hypertable in the columnstore.

Samples

In this sample, you convert the conditions table to a hypertable with partitioning on the time column. You then specify and enable additional columns to track ranges for. You then disable range tracking:

SELECT create_hypertable('conditions', 'time');
SELECT enable_chunk_skipping('conditions', 'device_id');
SELECT disable_chunk_skipping('conditions', 'device_id');

Best practice is to enable range tracking on columns which are correlated to the partitioning column. In other words, enable tracking on secondary columns that are referenced in the WHERE clauses in your queries. Use this API to disable range tracking on columns when the query patterns don't use this secondary column anymore.

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable that the column belongs to
column_name	TEXT	Column to disable tracking range statistics for

Optional arguments

Name	Type	Description
if_not_exists	BOOLEAN	Set to true so that a notice is sent when ranges are not being tracked for a column. By default, an error is thrown

Returns

Column	Type	Description
hypertable_id	INTEGER	ID of the hypertable in TimescaleDB.
column_name	TEXT	Name of the column range tracking is disabled for
disabled	BOOLEAN	Returns true when tracking is disabled. false when if_not_exists is true and the entry was
not removed

To disable_chunk_skipping(), you must have first called enable_chunk_skipping and enabled range tracking on a column in the hypertable.

===== PAGE: https://docs.tigerdata.com/api/hypertable/remove_reorder_policy/ =====

remove_reorder_policy()

Remove a policy to reorder a particular hypertable.

Samples

SELECT remove_reorder_policy('conditions', if_exists => true);

removes the existing reorder policy for the conditions table if it exists.

Required arguments

Name	Type	Description
hypertable	REGCLASS	Name of the hypertable from which to remove the policy.

Optional arguments

Name	Type	Description
if_exists	BOOLEAN	Set to true to avoid throwing an error if the reorder_policy does not exist. A notice is issued instead. Defaults to false.

===== PAGE: https://docs.tigerdata.com/api/hypertable/reorder_chunk/ =====

reorder_chunk()

Reorder a single chunk's heap to follow the order of an index. This function acts similarly to the Postgres CLUSTER command , however it uses lower lock levels so that, unlike with the CLUSTER command, the chunk and hypertable are able to be read for most of the process. It does use a bit more disk space during the operation.

This command can be particularly useful when data is often queried in an order different from that in which it was originally inserted. For example, data is commonly inserted into a hypertable in loose time order (for example, many devices concurrently sending their current state), but one might typically query the hypertable about a specific device. In such cases, reordering a chunk using an index on (device_id, time) can lead to significant performance improvement for these types of queries.

One can call this function directly on individual chunks of a hypertable, but using add_reorder_policy is often much more convenient.

Samples

Reorder a chunk on an index:

SELECT reorder_chunk('_timescaledb_internal._hyper_1_10_chunk', '_timescaledb_internal.conditions_device_id_time_idx');

Required arguments

Name	Type	Description
chunk	REGCLASS	Name of the chunk to reorder.

Optional arguments

Name	Type	Description
index	REGCLASS	The name of the index (on either the hypertable or chunk) to order by.
verbose	BOOLEAN	Setting to true displays messages about the progress of the reorder command. Defaults to false.

Returns

This function returns void.

===== PAGE: https://docs.tigerdata.com/api/hypertable/add_reorder_policy/ =====

add_reorder_policy()

Create a policy to reorder the rows of a hypertable's chunks on a specific index. The policy reorders the rows for all chunks except the two most recent ones, because these are still getting writes. By default, the policy runs every 24 hours. To change the schedule, call alter_job and adjust schedule_interval.

You can have only one reorder policy on each hypertable.

For manual reordering of individual chunks, see reorder_chunk.

When a chunk's rows have been reordered by a policy, they are not reordered by subsequent runs of the same policy. If you write significant amounts of data into older chunks that have already been reordered, re-run reorder_chunk on them. If you have changed a lot of older chunks, it is better to drop and recreate the policy.

Samples

SELECT add_reorder_policy('conditions', 'conditions_device_id_time_idx');

Creates a policy to reorder chunks by the existing (device_id, time) index every 24 hours. This applies to all chunks except the two most recent ones.

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable to create the policy for
index_name	TEXT	Existing hypertable index by which to order the rows on disk

Optional arguments

Name	Type	Description
if_not_exists	BOOLEAN	Set to true to avoid an error if the reorder_policy already exists. A notice is issued instead. Defaults to false.
initial_start	TIMESTAMPTZ	Controls when the policy first runs and how its future run schedule is calculated. If omitted or set to NULL (default): The first run is scheduled at now() + schedule_interval (defaults to 24 hours). The next run is scheduled at one full schedule_interval after the end of the previous run. If set: The first run is at the specified time. The next run is scheduled as initial_start + schedule_interval regardless of when the previous run ends.
timezone	TEXT	A valid time zone. If initial_start is also specified, subsequent runs of the reorder policy are aligned on its initial start. However, daylight savings time (DST) changes might shift this alignment. Set to a valid time zone if this is an issue you want to mitigate. If omitted, UTC bucketing is performed. Defaults to NULL.

Returns

Column	Type	Description
job_id	INTEGER	TimescaleDB background job ID created to implement this policy

===== PAGE: https://docs.tigerdata.com/api/hypertable/hypertable_detailed_size/ =====

hypertable_detailed_size()

hypertable_detailed_size()

Get detailed information about disk space used by a hypertable or continuous aggregate, returning size information for the table itself, any indexes on the table, any toast tables, and the total size of all. All sizes are reported in bytes. If the function is executed on a distributed hypertable, it returns size information as a separate row per node, including the access node.

When a continuous aggregate name is provided, the function transparently looks up the backing hypertable and returns its statistics instead.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get the size information for a hypertable.

-- disttable is a distributed hypertable --
SELECT * FROM hypertable_detailed_size('disttable') ORDER BY node_name;

 table_bytes | index_bytes | toast_bytes | total_bytes |  node_name
-------------+-------------+-------------+-------------+-------------
       16384 |       40960 |           0 |       57344 | data_node_1
        8192 |       24576 |           0 |       32768 | data_node_2
           0 |        8192 |           0 |        8192 |

The access node is listed without a user-given node name. Normally, the access node holds no data, but still maintains, for example, index information that occupies a small amount of disk space.

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable or continuous aggregate to show detailed size of.

Returns

Column	Type	Description
table_bytes	BIGINT	Disk space used by main_table (like pg_relation_size(main_table))
index_bytes	BIGINT	Disk space used by indexes
toast_bytes	BIGINT	Disk space of toast tables
total_bytes	BIGINT	Total disk space used by the specified table, including all indexes and TOAST data
node_name	TEXT	For distributed hypertables, this is the user-given name of the node for which the size is reported. NULL is returned for the access node and non-distributed hypertables.

If executed on a relation that is not a hypertable, the function returns NULL.

===== PAGE: https://docs.tigerdata.com/api/hypertable/show_chunks/ =====

show_chunks()

Get list of chunks associated with a hypertable.

Function accepts the following required and optional arguments. These arguments have the same semantics as the drop_chunks function.

Samples

Get list of all chunks associated with a table:

SELECT show_chunks('conditions');

Get all chunks from hypertable conditions older than 3 months:

SELECT show_chunks('conditions', older_than => INTERVAL '3 months');

Get all chunks from hypertable conditions created before 3 months:

SELECT show_chunks('conditions', created_before => INTERVAL '3 months');

Get all chunks from hypertable conditions created in the last 1 month:

SELECT show_chunks('conditions', created_after => INTERVAL '1 month');

Get all chunks from hypertable conditions before 2017:

SELECT show_chunks('conditions', older_than => DATE '2017-01-01');

Required arguments

Name	Type	Description
relation	REGCLASS	Hypertable or continuous aggregate from which to select chunks.

Optional arguments

Name	Type	Description
older_than	ANY	Specification of cut-off point where any chunks older than this timestamp should be shown.
newer_than	ANY	Specification of cut-off point where any chunks newer than this timestamp should be shown.
created_before	ANY	Specification of cut-off point where any chunks created before this timestamp should be shown.
created_after	ANY	Specification of cut-off point where any chunks created after this timestamp should be shown.

The older_than and newer_than parameters can be specified in two ways:

• interval type: The cut-off point is computed as now() - older_than and similarly now() - newer_than. An error is returned if an INTERVAL is supplied and the time column is not one of a TIMESTAMP, TIMESTAMPTZ, or DATE.

• timestamp, date, or integer type: The cut-off point is explicitly given as a TIMESTAMP / TIMESTAMPTZ / DATE or as a SMALLINT / INT / BIGINT. The choice of timestamp or integer must follow the type of the hypertable's time column.

The created_before and created_after parameters can be specified in two ways:

• interval type: The cut-off point is computed as now() - created_before and similarly now() - created_after. This uses the chunk creation time for the filtering.

• timestamp, date, or integer type: The cut-off point is explicitly given as a TIMESTAMP / TIMESTAMPTZ / DATE or as a SMALLINT / INT / BIGINT. The choice of integer value must follow the type of the hypertable's partitioning column. Otherwise the chunk creation time is used for the filtering.

When both older_than and newer_than arguments are used, the function returns the intersection of the resulting two ranges. For example, specifying newer_than => 4 months and older_than => 3 months shows all chunks between 3 and 4 months old. Similarly, specifying newer_than => '2017-01-01' and older_than => '2017-02-01' shows all chunks between '2017-01-01' and '2017-02-01'. Specifying parameters that do not result in an overlapping intersection between two ranges results in an error.

When both created_before and created_after arguments are used, the function returns the intersection of the resulting two ranges. For example, specifying created_after=> 4 monthsandcreated_before=> 3 months shows all chunks created between 3 and 4 months from now. Similarly, specifying created_after=> '2017-01-01'andcreated_before => '2017-02-01' shows all chunks created between '2017-01-01' and '2017-02-01'. Specifying parameters that do not result in an overlapping intersection between two ranges results in an error.

The created_before/created_after parameters cannot be used together with older_than/newer_than.

===== PAGE: https://docs.tigerdata.com/api/hypertable/merge_chunks/ =====

merge_chunks()

Merge two or more chunks into one.

The partition boundaries for the new chunk is the union of all partitions of the merged chunks. The new chunk retains the name, constraints, and triggers of the first chunk in the partition order.

You can only merge chunks that have directly adjacent partitions. It is not possible to merge chunks that have another chunk, or an empty range between them in any of the partitioning dimensions.

Chunk merging has the following limitations. You cannot:

• Merge chunks with tiered data
• Read or write from the chunks while they are being merged

Since2180

Refer to the installation documentation for detailed setup instructions.

Samples

• Merge two chunks:

  CALL merge_chunks('_timescaledb_internal._hyper_1_1_chunk', '_timescaledb_internal._hyper_1_2_chunk');
  

• Merge more than two chunks:

  CALL merge_chunks('{_timescaledb_internal._hyper_1_1_chunk, _timescaledb_internal._hyper_1_2_chunk, _timescaledb_internal._hyper_1_3_chunk}');
  

Arguments

You can merge either two chunks, or an arbitrary number of chunks specified as an array of chunk identifiers. When you call merge_chunks, you must specify either chunk1 and chunk2, or chunks. You cannot use both arguments.

Name	Type	Default	Required	Description
chunk1, chunk2	REGCLASS	-	✖	The two chunk to merge in partition order
chunks	REGCLASS[]	-	✖	The array of chunks to merge in partition order

===== PAGE: https://docs.tigerdata.com/api/hypertable/add_dimension/ =====

add_dimension()

Add an additional partitioning dimension to a TimescaleDB hypertable. You can only execute this add_dimension command on an empty hypertable. To convert a normal table to a hypertable, call create hypertable.

The column you select as the dimension can use either:

• Interval partitions: for example, for a second range partition.
• hash partitions: to enable parallelization across multiple disks.

Best practice is to not use additional dimensions. However, Tiger Cloud transparently provides seamless storage scaling, both in terms of storage capacity and available storage IOPS/bandwidth.

This page describes the generalized hypertable API introduced in TimescaleDB v2.13.0. For information about the deprecated interface, see add_dimension(), deprecated interface.

Samples

First convert table conditions to hypertable with just range partitioning on column time, then add an additional partition key on location with four partitions:

SELECT create_hypertable('conditions', by_range('time'));
SELECT add_dimension('conditions', by_hash('location', 4));

The by_range and by_hash dimension builders are an addition to TimescaleDB 2.13.

Convert table conditions to hypertable with range partitioning on time then add three additional dimensions: one hash partitioning on location, one range partition on time_received, and one hash partitionining on device_id.

SELECT create_hypertable('conditions', by_range('time'));
SELECT add_dimension('conditions', by_hash('location', 2));
SELECT add_dimension('conditions', by_range('time_received', INTERVAL '1 day'));
SELECT add_dimension('conditions', by_hash('device_id', 2));
SELECT add_dimension('conditions', by_hash('device_id', 2), if_not_exists => true);

Arguments

Name	Type	Default	Required	Description
chunk_time_interval	INTERVAL	-	✖	Interval that each chunk covers. Must be > 0.
dimension	DIMENSION_INFO	-	✔	To create a _timescaledb_internal.dimension_info instance to partition a hypertable, you call by_range and by_hash.
hypertable	REGCLASS	-	✔	The hypertable to add the dimension to.
if_not_exists	BOOLEAN	false	✖	Set to true to print an error if a dimension for the column already exists. By default an exception is raised.
number_partitions	INTEGER	-	✖	Number of hash partitions to use on column_name. Must be > 0.
partitioning_func	REGCLASS	-	✖	The function to use for calculating a value's partition. See create_hypertable for more information.

Dimension info

To create a _timescaledb_internal.dimension_info instance, you call add_dimension to an existing hypertable.

Samples

Hypertables must always have a primary range dimension, followed by an arbitrary number of additional dimensions that can be either range or hash, Typically this is just one hash. For example:

SELECT add_dimension('conditions', by_range('time'));
SELECT add_dimension('conditions', by_hash('location', 2));

For incompatible data types such as jsonb, you can specify a function to the partition_func argument of the dimension build to extract a compatible data type. Look in the example section below.

Custom partitioning

By default, TimescaleDB calls Postgres's internal hash function for the given type. You use a custom partitioning function for value types that do not have a native Postgres hash function.

You can specify a custom partitioning function for both range and hash partitioning. A partitioning function should take a anyelement argument as the only parameter and return a positive integer hash value. This hash value is not a partition identifier, but rather the inserted value's position in the dimension's key space, which is then divided across the partitions.

by_range()

Create a by-range dimension builder. You can partition by_range on it's own.

Samples

• Partition on time using CREATE TABLE

  The simplest usage is to partition on a time column:

  CREATE TABLE conditions (
     time        TIMESTAMPTZ       NOT NULL,
     location    TEXT              NOT NULL,
     device      TEXT              NOT NULL,
     temperature DOUBLE PRECISION  NULL,
     humidity    DOUBLE PRECISION  NULL
  ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='time'
  );
  

  If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

  This is the default partition, you do not need to add it explicitly.

• Extract time from a non-time column using create_hypertable

  If you have a table with a non-time column containing the time, such as a JSON column, add a partition function to extract the time:

  CREATE TABLE my_table (
     metric_id serial not null,
     data jsonb,
  );
  
  CREATE FUNCTION get_time(jsonb) RETURNS timestamptz AS $$
    SELECT ($1->>'time')::timestamptz
  $$ LANGUAGE sql IMMUTABLE;
  
  SELECT create_hypertable('my_table', by_range('data', '1 day', 'get_time'));
  

Arguments

Name	Type	Default	Required	Description
column_name	NAME	-	✔	Name of column to partition on.
partition_func	REGPROC	-	✖	The function to use for calculating the partition of a value.
partition_interval	ANYELEMENT	-	✖	Interval to partition column on.

If the column to be partitioned is a:

• TIMESTAMP, TIMESTAMPTZ, or DATE: specify partition_interval either as an INTERVAL type or an integer value in microseconds.

• Another integer type: specify partition_interval as an integer that reflects the column's underlying semantics. For example, if this column is in UNIX time, specify partition_interval in milliseconds.

The partition type and default value depending on column type is:

Column Type	Partition Type	Default value
TIMESTAMP WITHOUT TIMEZONE	INTERVAL/INTEGER	1 week
TIMESTAMP WITH TIMEZONE	INTERVAL/INTEGER	1 week
DATE	INTERVAL/INTEGER	1 week
SMALLINT	SMALLINT	10000
INT	INT	100000
BIGINT	BIGINT	1000000

by_hash()

The main purpose of hash partitioning is to enable parallelization across multiple disks within the same time interval. Every distinct item in hash partitioning is hashed to one of N buckets. By default, TimescaleDB uses flexible range intervals to manage chunk sizes.

Parallelizing disk I/O

You use Parallel I/O in the following scenarios:

• Two or more concurrent queries should be able to read from different disks in parallel.
• A single query should be able to use query parallelization to read from multiple disks in parallel.

For the following options:

• RAID: use a RAID setup across multiple physical disks, and expose a single logical disk to the hypertable. That is, using a single tablespace.

  Best practice is to use RAID when possible, as you do not need to manually manage tablespaces in the database.

• Multiple tablespaces: for each physical disk, add a separate tablespace to the database. TimescaleDB allows you to add multiple tablespaces to a single hypertable. However, although under the hood, a hypertable's chunks are spread across the tablespaces associated with that hypertable.

  When using multiple tablespaces, a best practice is to also add a second hash-partitioned dimension to your hypertable and to have at least one hash partition per disk. While a single time dimension would also work, it would mean that the first chunk is written to one tablespace, the second to another, and so on, and thus would parallelize only if a query's time range exceeds a single chunk.

When adding a hash partitioned dimension, set the number of partitions to a multiple of number of disks. For example, the number of partitions P=N*Pd where N is the number of disks and Pd is the number of partitions per disk. This enables you to add more disks later and move partitions to the new disk from other disks.

TimescaleDB does not benefit from a very large number of hash partitions, such as the number of unique items you expect in partition field. A very large number of hash partitions leads both to poorer per-partition load balancing (the mapping of items to partitions using hashing), as well as much increased planning latency for some types of queries.

Samples

CREATE TABLE conditions (
   "time"      TIMESTAMPTZ       NOT NULL,
   location    TEXT              NOT NULL,
   device      TEXT              NOT NULL,
   temperature DOUBLE PRECISION  NULL,
   humidity    DOUBLE PRECISION  NULL
) WITH (
   tsdb.hypertable,
   tsdb.partition_column='time',
   tsdb.chunk_interval='1 day'
);

SELECT add_dimension('conditions', by_hash('location', 2));

Arguments

Name	Type	Default	Required	Description
column_name	NAME	-	✔	Name of column to partition on.
partition_func	REGPROC	-	✖	The function to use to calcule the partition of a value.
number_partitions	ANYELEMENT	-	✔	Number of hash partitions to use for partitioning_column. Must be greater than 0.

Returns

by_range and by-hash return an opaque _timescaledb_internal.dimension_info instance, holding the dimension information used by this function.

Returns

Column	Type	Description
dimension_id	INTEGER	ID of the dimension in the TimescaleDB internal catalog
created	BOOLEAN	true if the dimension was added, false when you set if_not_exists to true and no dimension was added.

===== PAGE: https://docs.tigerdata.com/api/hypertable/add_dimension_old/ =====

add_dimension()

This interface is deprecated since TimescaleDB v2.13.0.

For information about the supported hypertable interface, see add_dimension().

Add an additional partitioning dimension to a TimescaleDB hypertable. The column selected as the dimension can either use interval partitioning (for example, for a second time partition) or hash partitioning.

The add_dimension command can only be executed after a table has been converted to a hypertable (via create_hypertable), but must similarly be run only on an empty hypertable.

Space partitions: Using space partitions is highly recommended for distributed hypertables to achieve efficient scale-out performance. For regular hypertables that exist only on a single node, additional partitioning can be used for specialized use cases and not recommended for most users.

Space partitions use hashing: Every distinct item is hashed to one of N buckets. Remember that we are already using (flexible) time intervals to manage chunk sizes; the main purpose of space partitioning is to enable parallelization across multiple data nodes (in the case of distributed hypertables) or across multiple disks within the same time interval (in the case of single-node deployments).

Samples

First convert table conditions to hypertable with just time partitioning on column time, then add an additional partition key on location with four partitions:

SELECT create_hypertable('conditions', 'time');
SELECT add_dimension('conditions', 'location', number_partitions => 4);

Convert table conditions to hypertable with time partitioning on time and space partitioning (2 partitions) on location, then add two additional dimensions.

SELECT create_hypertable('conditions', 'time', 'location', 2);
SELECT add_dimension('conditions', 'time_received', chunk_time_interval => INTERVAL '1 day');
SELECT add_dimension('conditions', 'device_id', number_partitions => 2);
SELECT add_dimension('conditions', 'device_id', number_partitions => 2, if_not_exists => true);

Now in a multi-node example for distributed hypertables with a cluster of one access node and two data nodes, configure the access node for access to the two data nodes. Then, convert table conditions to a distributed hypertable with just time partitioning on column time, and finally add a space partitioning dimension on location with two partitions (as the number of the attached data nodes).

SELECT add_data_node('dn1', host => 'dn1.example.com');
SELECT add_data_node('dn2', host => 'dn2.example.com');
SELECT create_distributed_hypertable('conditions', 'time');
SELECT add_dimension('conditions', 'location', number_partitions => 2);

Parallelizing queries across multiple data nodes

In a distributed hypertable, space partitioning enables inserts to be parallelized across data nodes, even while the inserted rows share timestamps from the same time interval, and thus increases the ingest rate. Query performance also benefits by being able to parallelize queries across nodes, particularly when full or partial aggregations can be "pushed down" to data nodes (for example, as in the query avg(temperature) FROM conditions GROUP BY hour, location when using location as a space partition). Please see our best practices about partitioning in distributed hypertables for more information.

Parallelizing disk I/O on a single node

Parallel I/O can benefit in two scenarios: (a) two or more concurrent queries should be able to read from different disks in parallel, or (b) a single query should be able to use query parallelization to read from multiple disks in parallel.

Thus, users looking for parallel I/O have two options:

1. Use a RAID setup across multiple physical disks, and expose a single logical disk to the hypertable (that is, via a single tablespace).

2. For each physical disk, add a separate tablespace to the database. TimescaleDB allows you to actually add multiple tablespaces to a single hypertable (although under the covers, a hypertable's chunks are spread across the tablespaces associated with that hypertable).

We recommend a RAID setup when possible, as it supports both forms of parallelization described above (that is, separate queries to separate disks, single query to multiple disks in parallel). The multiple tablespace approach only supports the former. With a RAID setup, no spatial partitioning is required.

That said, when using space partitions, we recommend using 1 space partition per disk.

TimescaleDB does not benefit from a very large number of space partitions (such as the number of unique items you expect in partition field). A very large number of such partitions leads both to poorer per-partition load balancing (the mapping of items to partitions using hashing), as well as much increased planning latency for some types of queries.

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable to add the dimension to
column_name	TEXT	Column to partition by

Optional arguments

Name	Type	Description
number_partitions	INTEGER	Number of hash partitions to use on column_name. Must be > 0
chunk_time_interval	INTERVAL	Interval that each chunk covers. Must be > 0
partitioning_func	REGCLASS	The function to use for calculating a value's partition (see create_hypertable instructions)
if_not_exists	BOOLEAN	Set to true to avoid throwing an error if a dimension for the column already exists. A notice is issued instead. Defaults to false

Returns

Column	Type	Description
dimension_id	INTEGER	ID of the dimension in the TimescaleDB internal catalog
schema_name	TEXT	Schema name of the hypertable
table_name	TEXT	Table name of the hypertable
column_name	TEXT	Column name of the column to partition by
created	BOOLEAN	True if the dimension was added, false when if_not_exists is true and no dimension was added

When executing this function, either number_partitions or chunk_time_interval must be supplied, which dictates if the dimension uses hash or interval partitioning.

The chunk_time_interval should be specified as follows:

• If the column to be partitioned is a TIMESTAMP, TIMESTAMPTZ, or DATE, this length should be specified either as an INTERVAL type or an integer value in microseconds.

• If the column is some other integer type, this length should be an integer that reflects the column's underlying semantics (for example, the chunk_time_interval should be given in milliseconds if this column is the number of milliseconds since the UNIX epoch).

Supporting more than one additional dimension is currently experimental. For any production environments, users are recommended to use at most one "space" dimension.

===== PAGE: https://docs.tigerdata.com/api/hypertable/hypertable_approximate_detailed_size/ =====

hypertable_approximate_detailed_size()

Get detailed information about approximate disk space used by a hypertable or continuous aggregate, returning size information for the table itself, any indexes on the table, any toast tables, and the total size of all. All sizes are reported in bytes.

When a continuous aggregate name is provided, the function transparently looks up the backing hypertable and returns its approximate size statistics instead.

This function relies on the per backend caching using the in-built Postgres storage manager layer to compute the approximate size cheaply. The PG cache invalidation clears off the cached size for a chunk when DML happens into it. That size cache is thus able to get the latest size in a matter of minutes. Also, due to the backend caching, any long running session will only fetch latest data for new or modified chunks and can use the cached data (which is calculated afresh the first time around) effectively for older chunks. Thus it is recommended to use a single connected Postgres backend session to compute the approximate sizes of hypertables to get faster results.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get the approximate size information for a hypertable.

SELECT * FROM hypertable_approximate_detailed_size('hyper_table');
 table_bytes | index_bytes | toast_bytes | total_bytes
-------------+-------------+-------------+-------------
        8192 |       24576 |       32768 |       65536

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable or continuous aggregate to show detailed approximate size of.

Returns

Column	Type	Description
table_bytes	BIGINT	Approximate disk space used by main_table (like pg_relation_size(main_table))
index_bytes	BIGINT	Approximate disk space used by indexes
toast_bytes	BIGINT	Approximate disk space of toast tables
total_bytes	BIGINT	Approximate total disk space used by the specified table, including all indexes and TOAST data

If executed on a relation that is not a hypertable, the function returns NULL.

===== PAGE: https://docs.tigerdata.com/api/hypertable/set_integer_now_func/ =====

set_integer_now_fun()

Override the now() date/time function used to set the current time in the integer time column in a hypertable. Many policies only apply to chunks of a certain age. integer_now_func determines the age of each chunk.

The function you set as integer_now_func has no arguments. It must be either:

• IMMUTABLE: Use when you execute the query each time rather than prepare it prior to execution. The value for integer_now_func is computed before the plan is generated. This generates a significantly smaller plan, especially if you have a lot of chunks.

• STABLE: integer_now_func is evaluated just before query execution starts. chunk pruning is executed at runtime. This generates a correct result, but may increase planning time.

set_integer_now_func does not work on tables where the time column type is TIMESTAMP, TIMESTAMPTZ, or DATE.

Samples

Set the integer now function for a hypertable with a time column in unix time.

• IMMUTABLE: when you execute the query each time:

  CREATE OR REPLACE FUNCTION unix_now_immutable() returns BIGINT LANGUAGE SQL IMMUTABLE as $$  SELECT extract (epoch from now())::BIGINT $$;
  
  SELECT set_integer_now_func('hypertable_name', 'unix_now_immutable');
  

• STABLE: for prepared statements:

  CREATE OR REPLACE FUNCTION unix_now_stable() returns BIGINT LANGUAGE SQL STABLE AS $$ SELECT extract(epoch from now())::BIGINT $$;
  
  SELECT set_integer_now_func('hypertable_name', 'unix_now_stable');
  

Required arguments

Name	Type	Description
main_table	REGCLASS	The hypertable integer_now_func is used in.
integer_now_func	REGPROC	A function that returns the current time set in each row in the time column in main_table.

Optional arguments

Name	Type	Description
replace_if_exists	BOOLEAN	Set to true to override integer_now_func when you have previously set a custom function. Default is false.

===== PAGE: https://docs.tigerdata.com/api/hypertable/create_index/ =====

CREATE INDEX (Transaction Per Chunk)

CREATE INDEX ... WITH (timescaledb.transaction_per_chunk, ...);

This option extends CREATE INDEX with the ability to use a separate transaction for each chunk it creates an index on, instead of using a single transaction for the entire hypertable. This allows INSERTs, and other operations to be performed concurrently during most of the duration of the CREATE INDEX command. While the index is being created on an individual chunk, it functions as if a regular CREATE INDEX were called on that chunk, however other chunks are completely unblocked.

This version of CREATE INDEX can be used as an alternative to CREATE INDEX CONCURRENTLY, which is not currently supported on hypertables.

• Not supported for CREATE UNIQUE INDEX.

• If the operation fails partway through, indexes might not be created on all hypertable chunks. If this occurs, the index on the root table of the hypertable is marked as invalid. You can check this by running \d+ on the hypertable. The index still works, and is created on new chunks, but if you want to ensure all chunks have a copy of the index, drop and recreate it.

  You can also use the following query to find all invalid indexes:

  SELECT * FROM pg_index i WHERE i.indisvalid IS FALSE;
  

Samples

Create an anonymous index:

CREATE INDEX ON conditions(time, device_id)
    WITH (timescaledb.transaction_per_chunk);

Alternatively:

CREATE INDEX ON conditions USING brin(time, location)
    WITH (timescaledb.transaction_per_chunk);

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/refresh_continuous_aggregate/ =====

refresh_continuous_aggregate()

Refresh all buckets of a continuous aggregate in the refresh window given by window_start and window_end.

A continuous aggregate materializes aggregates in time buckets. For example, min, max, average over 1 day worth of data, and is determined by the time_bucket interval. Therefore, when refreshing the continuous aggregate, only buckets that completely fit within the refresh window are refreshed. In other words, it is not possible to compute the aggregate over, for an incomplete bucket. Therefore, any buckets that do not fit within the given refresh window are excluded.

The function expects the window parameter values to have a time type that is compatible with the continuous aggregate's time bucket expression—for example, if the time bucket is specified in TIMESTAMP WITH TIME ZONE, then the start and end time should be a date or timestamp type. Note that a continuous aggregate using the TIMESTAMP WITH TIME ZONE type aligns with the UTC time zone, so, if window_start and window_end is specified in the local time zone, any time zone shift relative UTC needs to be accounted for when refreshing to align with bucket boundaries.

To improve performance for continuous aggregate refresh, see CREATE MATERIALIZED VIEW .

Samples

Refresh the continuous aggregate conditions between 2020-01-01 and 2020-02-01 exclusive.

CALL refresh_continuous_aggregate('conditions', '2020-01-01', '2020-02-01');

Alternatively, incrementally refresh the continuous aggregate conditions between 2020-01-01 and 2020-02-01 exclusive, working in 12h intervals:

DO
$$
DECLARE
  refresh_interval INTERVAL = '12h'::INTERVAL;
  start_timestamp TIMESTAMPTZ = '2020-01-01T00:00:00Z';
  end_timestamp TIMESTAMPTZ = start_timestamp + refresh_interval;
BEGIN
  WHILE start_timestamp < '2020-02-01T00:00:00Z' LOOP
    CALL refresh_continuous_aggregate('conditions', start_timestamp, end_timestamp);
    COMMIT;
    RAISE NOTICE 'finished with timestamp %', end_timestamp;
    start_timestamp = end_timestamp;
    end_timestamp = end_timestamp + refresh_interval;
  END LOOP;
END
$$;

Force the conditions continuous aggregate to refresh between 2020-01-01 and 2020-02-01 exclusive, even if the data has already been refreshed.

CALL refresh_continuous_aggregate('conditions', '2020-01-01', '2020-02-01', force => TRUE);

Required arguments

Name	Type	Description
continuous_aggregate	REGCLASS	The continuous aggregate to refresh.
window_start	INTERVAL, TIMESTAMPTZ, INTEGER	Start of the window to refresh, has to be before window_end.
window_end	INTERVAL, TIMESTAMPTZ, INTEGER	End of the window to refresh, has to be after window_start.

You must specify the window_start and window_end parameters differently, depending on the type of the time column of the hypertable. For hypertables with TIMESTAMP, TIMESTAMPTZ, and DATE time columns, set the refresh window as an INTERVAL type. For hypertables with integer-based timestamps, set the refresh window as an INTEGER type.

A NULL value for window_start is equivalent to the lowest changed element in the raw hypertable of the CAgg. A NULL value for window_end is equivalent to the largest changed element in raw hypertable of the CAgg. As changed element tracking is performed after the initial CAgg refresh, running CAgg refresh without window_start and window_end covers the entire time range.

Note that it's not guaranteed that all buckets will be updated: refreshes will not take place when buckets are materialized with no data changes or with changes that only occurred in the secondary table used in the JOIN.

Optional arguments

Name	Type	Description
force	BOOLEAN	Force refresh every bucket in the time range between window_start and window_end, even when the bucket has already been refreshed. This can be very expensive when a lot of data is refreshed. Default is FALSE.
refresh_newest_first	BOOLEAN	Set to FALSE to refresh the oldest data first. Default is TRUE.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/remove_policies/ =====

remove_policies()

Remove refresh, columnstore, and data retention policies from a continuous aggregate. The removed columnstore and retention policies apply to the continuous aggregate, not to the original hypertable.

timescaledb_experimental.remove_policies(
     relation REGCLASS,
     if_exists BOOL = false,
     VARIADIC policy_names TEXT[] = NULL
) RETURNS BOOL

To remove all policies on a continuous aggregate, see remove_all_policies().

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

Samples

Given a continuous aggregate named example_continuous_aggregate with a refresh policy and a data retention policy, remove both policies.

Throw an error if either policy doesn't exist. If the continuous aggregate has a columnstore policy, leave it unchanged:

SELECT timescaledb_experimental.remove_policies(
    'example_continuous_aggregate',
    false,
    'policy_refresh_continuous_aggregate',
    'policy_retention'
);

Required arguments

Name	Type	Description
relation	REGCLASS	The continuous aggregate to remove policies from

Optional arguments

Name	Type	Description
if_exists	BOOL	When true, prints a warning instead of erroring if the policy doesn't exist. Defaults to false.
policy_names	TEXT	The policies to remove. You can list multiple policies, separated by a comma. Allowed policy names are policy_refresh_continuous_aggregate, policy_compression, and policy_retention.

Returns

Returns true if successful.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/add_continuous_aggregate_policy/ =====

add_continuous_aggregate_policy()

Create a policy that automatically refreshes a continuous aggregate. To view the policies that you set or the policies that already exist, see informational views.

Samples

Add a policy that refreshes the last month once an hour, excluding the latest hour from the aggregate. For performance reasons, we recommend that you exclude buckets that see lots of writes:

SELECT add_continuous_aggregate_policy('conditions_summary',
  start_offset => INTERVAL '1 month',
  end_offset => INTERVAL '1 hour',
  schedule_interval => INTERVAL '1 hour');

Required arguments

Name	Type	Description
continuous_aggregate	REGCLASS	The continuous aggregate to add the policy for
start_offset	INTERVAL or integer	Start of the refresh window as an interval relative to the time when the policy is executed. NULL is equivalent to MIN(timestamp) of the hypertable.
end_offset	INTERVAL or integer	End of the refresh window as an interval relative to the time when the policy is executed. NULL is equivalent to MAX(timestamp) of the hypertable.
schedule_interval	INTERVAL	Interval between refresh executions in wall-clock time. Defaults to 24 hours
initial_start	TIMESTAMPTZ	Time the policy is first run. Defaults to NULL. If omitted, then the schedule interval is the intervalbetween the finish time of the last execution and the next start. If provided, it serves as the origin with respect to which the next_start is calculated

The start_offset should be greater than end_offset.

You must specify the start_offset and end_offset parameters differently, depending on the type of the time column of the hypertable:

• For hypertables with TIMESTAMP, TIMESTAMPTZ, and DATE time columns, set the offset as an INTERVAL type.
• For hypertables with integer-based timestamps, set the offset as an INTEGER type.

While setting end_offset to NULL is possible, it is not recommended. To include the data between end_offset and the current time in queries, enable real-time aggregation.

You can add concurrent refresh policies on each continuous aggregate, as long as the start_offset and end_offset does not overlap with another policy on the same continuous aggregate.

Optional arguments

Name	Type	Description
if_not_exists	BOOLEAN	Set to true to issue a notice instead of an error if the job already exists. Defaults to false.
timezone	TEXT	A valid time zone. If you specify initial_start, subsequent executions of the refresh policy are aligned on initial_start. However, daylight savings time (DST) changes may shift this alignment. If this is an issue you want to mitigate, set timezone to a valid time zone. Default is NULL, UTC bucketing is performed.
include_tiered_data	BOOLEAN	Enable/disable reading tiered data. This setting helps override the current settings for thetimescaledb.enable_tiered_reads GUC. The default is NULL i.e we use the current setting for timescaledb.enable_tiered_reads GUC
buckets_per_batch	INTEGER	Number of buckets to be refreshed by a batch. This value is multiplied by the CAgg bucket width to determine the size of the batch range. Default value is 1, single batch execution. Values of less than 0 are not allowed.
max_batches_per_execution	INTEGER	Limit the maximum number of batches to run when a policy executes. If some batches remain, they are processed the next time the policy runs. Default value is 0, for an unlimted number of batches. Values of less than 0 are not allowed.
refresh_newest_first	BOOLEAN	Control the order of incremental refreshes. Set to TRUE to refresh from the newest data to the oldest. Set to FALSE for oldest to newest. The default is TRUE.

Setting buckets_per_batch greater than zero means that the refresh window is split in batches of bucket width * buckets per batch. For example, a given Continuous Aggregate with bucket width of 1 day and buckets_per_batch of 10 has a batch size of 10 days to process the refresh. Because each batch is an individual transaction, executing a policy in batches make the data visible for the users before the entire job is executed. Batches are processed from the most recent data to the oldest.

Returns

Column	Type	Description
job_id	INTEGER	TimescaleDB background job ID created to implement this policy

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/hypertable_size/ =====

hypertable_size()

hypertable_size()

Get the total disk space used by a hypertable or continuous aggregate, that is, the sum of the size for the table itself including chunks, any indexes on the table, and any toast tables. The size is reported in bytes. This is equivalent to computing the sum of total_bytes column from the output of hypertable_detailed_size function.

When a continuous aggregate name is provided, the function transparently looks up the backing hypertable and returns its statistics instead.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get the size information for a hypertable.

SELECT hypertable_size('devices');

 hypertable_size
-----------------
           73728

Get the size information for all hypertables.

SELECT hypertable_name, hypertable_size(format('%I.%I', hypertable_schema, hypertable_name)::regclass)
  FROM timescaledb_information.hypertables;

Get the size information for a continuous aggregate.

SELECT hypertable_size('device_stats_15m');

 hypertable_size
-----------------
           73728

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable or continuous aggregate to show size of.

Returns

Name	Type	Description
hypertable_size	BIGINT	Total disk space used by the specified hypertable, including all indexes and TOAST data

NULL is returned if the function is executed on a non-hypertable relation.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/alter_policies/ =====

alter_policies()

Alter refresh, columnstore, or data retention policies on a continuous aggregate. The altered columnstore and retention policies apply to the continuous aggregate, not to the original hypertable.

timescaledb_experimental.alter_policies(
     relation REGCLASS,
     if_exists BOOL = false,
     refresh_start_offset "any" = NULL,
     refresh_end_offset "any" = NULL,
     compress_after "any" = NULL,
     drop_after "any" = NULL
) RETURNS BOOL

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

Samples

Given a continuous aggregate named example_continuous_aggregate with an existing columnstore policy, alter the columnstore policy to compress data older than 16 days:

SELECT timescaledb_experimental.alter_policies(
    'continuous_agg_max_mat_date',
    compress_after => '16 days'::interval
);

Required arguments

Name	Type	Description
relation	REGCLASS	The continuous aggregate that you want to alter policies for

Optional arguments

Name	Type	Description
if_not_exists	BOOL	When true, prints a warning instead of erroring if the policy doesn't exist. Defaults to false.
refresh_start_offset	INTERVAL or INTEGER	The start of the continuous aggregate refresh window, expressed as an offset from the policy run time.
refresh_end_offset	INTERVAL or INTEGER	The end of the continuous aggregate refresh window, expressed as an offset from the policy run time. Must be greater than refresh_start_offset.
compress_after	INTERVAL or INTEGER	Continuous aggregate chunks are compressed into the columnstore if they exclusively contain data older than this interval.
drop_after	INTERVAL or INTEGER	Continuous aggregate chunks are dropped if they exclusively contain data older than this interval.

For arguments that could be either an INTERVAL or an INTEGER, use an INTERVAL if your time bucket is based on timestamps. Use an INTEGER if your time bucket is based on integers.

Returns

Returns true if successful.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/remove_continuous_aggregate_policy/ =====

remove_continuous_aggregate_policy()

Remove all refresh policies from a continuous aggregate.

remove_continuous_aggregate_policy(
    continuous_aggregate REGCLASS,
    if_exists BOOL = NULL
) RETURNS VOID

To view the existing continuous aggregate policies, see the policies informational view.

Samples

Remove all refresh policies from the cpu_view continuous aggregate:

SELECT remove_continuous_aggregate_policy('cpu_view');

Required arguments

Name	Type	Description
continuous_aggregate	REGCLASS	Name of the continuous aggregate the policies should be removed from

Optional arguments

Name	Type	Description
if_exists (formerly if_not_exists)	BOOL	When true, prints a warning instead of erroring if the policy doesn't exist. Defaults to false. Renamed in TimescaleDB 2.8.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/add_policies/ =====

add_policies()

Add refresh, compression, and data retention policies to a continuous aggregate in one step. The added compression and retention policies apply to the continuous aggregate, not to the original hypertable.

timescaledb_experimental.add_policies(
     relation REGCLASS,
     if_not_exists BOOL = false,
     refresh_start_offset "any" = NULL,
     refresh_end_offset "any" = NULL,
     compress_after "any" = NULL,
     drop_after "any" = NULL)
) RETURNS BOOL

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

add_policies() does not allow the schedule_interval for the continuous aggregate to be set, instead using a default value of 1 hour.

If you would like to set this add your policies manually (see add_continuous_aggregate_policy).

Samples

Given a continuous aggregate named example_continuous_aggregate, add three policies to it:

1. Regularly refresh the continuous aggregate to materialize data between 1 day and 2 days old.
2. Compress data in the continuous aggregate after 20 days.
3. Drop data in the continuous aggregate after 1 year.

SELECT timescaledb_experimental.add_policies(
    'example_continuous_aggregate',
    refresh_start_offset => '1 day'::interval,
    refresh_end_offset => '2 day'::interval,
    compress_after => '20 days'::interval,
    drop_after => '1 year'::interval
);

Required arguments

Name	Type	Description
relation	REGCLASS	The continuous aggregate that the policies should be applied to

Optional arguments

Name	Type	Description
if_not_exists	BOOL	When true, prints a warning instead of erroring if the continuous aggregate doesn't exist. Defaults to false.
refresh_start_offset	INTERVAL or INTEGER	The start of the continuous aggregate refresh window, expressed as an offset from the policy run time.
refresh_end_offset	INTERVAL or INTEGER	The end of the continuous aggregate refresh window, expressed as an offset from the policy run time. Must be greater than refresh_start_offset.
compress_after	INTERVAL or INTEGER	Continuous aggregate chunks are compressed if they exclusively contain data older than this interval.
drop_after	INTERVAL or INTEGER	Continuous aggregate chunks are dropped if they exclusively contain data older than this interval.

For arguments that could be either an INTERVAL or an INTEGER, use an INTERVAL if your time bucket is based on timestamps. Use an INTEGER if your time bucket is based on integers.

Returns

Returns true if successful.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/create_materialized_view/ =====

CREATE MATERIALIZED VIEW (Continuous Aggregate)

The CREATE MATERIALIZED VIEW statement is used to create continuous aggregates. To learn more, see the continuous aggregate how-to guides.

The syntax is:

CREATE MATERIALIZED VIEW <view_name> [ ( column_name [, ...] ) ]
  WITH ( timescaledb.continuous [, timescaledb.<option> = <value> ] )
  AS
    <select_query>
  [WITH [NO] DATA]

<select_query> is of the form:

SELECT <grouping_exprs>, <aggregate_functions>
    FROM <hypertable or another continuous aggregate>
[WHERE ... ]
GROUP BY time_bucket( <const_value>, <partition_col_of_hypertable> ),
         [ optional grouping exprs>]
[HAVING ...]

The continuous aggregate view defaults to WITH DATA. This means that when the view is created, it refreshes using all the current data in the underlying hypertable or continuous aggregate. This occurs once when the view is created. If you want the view to be refreshed regularly, you can use a refresh policy. If you do not want the view to update when it is first created, use the WITH NO DATA parameter. For more information, see refresh_continuous_aggregate.

Continuous aggregates have some limitations of what types of queries they can support. For more information, see the continuous aggregates section.

TimescaleDB v2.17.1 and greater dramatically decrease the amount of data written on a continuous aggregate in the presence of a small number of changes, reduce the i/o cost of refreshing a continuous aggregate, and generate fewer Write-Ahead Logs (WAL), set thetimescaledb.enable_merge_on_cagg_refresh configuration parameter to TRUE. This enables continuous aggregate refresh to use merge instead of deleting old materialized data and re-inserting.

For more settings for continuous aggregates, see timescaledb_information.continuous_aggregates.

Samples

Create a daily continuous aggregate view:

CREATE MATERIALIZED VIEW continuous_aggregate_daily( timec, minl, sumt, sumh )
WITH (timescaledb.continuous) AS
  SELECT time_bucket('1day', timec), min(location), sum(temperature), sum(humidity)
    FROM conditions
    GROUP BY time_bucket('1day', timec)

Add a thirty day continuous aggregate on top of the same raw hypertable:

CREATE MATERIALIZED VIEW continuous_aggregate_thirty_day( timec, minl, sumt, sumh )
WITH (timescaledb.continuous) AS
  SELECT time_bucket('30day', timec), min(location), sum(temperature), sum(humidity)
    FROM conditions
    GROUP BY time_bucket('30day', timec);

Add an hourly continuous aggregate on top of the same raw hypertable:

CREATE MATERIALIZED VIEW continuous_aggregate_hourly( timec, minl, sumt, sumh )
WITH (timescaledb.continuous) AS
  SELECT time_bucket('1h', timec), min(location), sum(temperature), sum(humidity)
    FROM conditions
    GROUP BY time_bucket('1h', timec);

Parameters

Name	Type	Description
<view_name>	TEXT	Name (optionally schema-qualified) of continuous aggregate view to create
<column_name>	TEXT	Optional list of names to be used for columns of the view. If not given, the column names are calculated from the query
WITH clause	TEXT	Specifies options for the continuous aggregate view
<select_query>	TEXT	A SELECT query that uses the specified syntax

Required WITH clause options:

Name	Type	Description
timescaledb.continuous	BOOLEAN	If timescaledb.continuous is not specified, this is a regular PostgresSQL materialized view

Optional WITH clause options:

Name	Type	Description	Default value
timescaledb.chunk_interval	INTERVAL	Set the chunk interval. The default value is 10x the original hypertable.
timescaledb.create_group_indexes	BOOLEAN	Create indexes on the continuous aggregate for columns in its GROUP BY clause. Indexes are in the form (<GROUP_BY_COLUMN>, time_bucket)	TRUE
timescaledb.finalized	BOOLEAN	In TimescaleDB 2.7 and above, use the new version of continuous aggregates, which stores finalized results for aggregate functions. Supports all aggregate functions, including ones that use FILTER, ORDER BY, and DISTINCT clauses.	TRUE
timescaledb.materialized_only	BOOLEAN	Return only materialized data when querying the continuous aggregate view	TRUE
timescaledb.invalidate_using	TEXT	Since TimescaleDB v2.22.0Set to wal to read changes from the WAL using logical decoding, then update the materialization invalidations for continuous aggregates using this information. This reduces the I/O and CPU needed to manage the hypertable invalidation log. Set to trigger to collect invalidations whenever there are inserts, updates, or deletes to a hypertable. This default behaviour uses more resources than wal.	trigger

For more information, see the real-time aggregates section.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/alter_materialized_view/ =====

ALTER MATERIALIZED VIEW (Continuous Aggregate)

You use the ALTER MATERIALIZED VIEW statement to modify some of the WITH clause options for a continuous aggregate view. You can only set the continuous and create_group_indexes options when you create a continuous aggregate. ALTER MATERIALIZED VIEW also supports the following Postgres clauses on the continuous aggregate view:

• RENAME TO: rename the continuous aggregate view
• RENAME [COLUMN]: rename the continuous aggregate column
• SET SCHEMA: set the new schema for the continuous aggregate view
• SET TABLESPACE: move the materialization of the continuous aggregate view to the new tablespace
• OWNER TO: set a new owner for the continuous aggregate view

Samples

• Enable real-time aggregates for a continuous aggregate:

  ALTER MATERIALIZED VIEW contagg_view SET (timescaledb.materialized_only = false);
  

• Enable hypercore for a continuous aggregate Since TimescaleDB v2.18.0:

   ALTER MATERIALIZED VIEW contagg_view SET (
    timescaledb.enable_columnstore = true,
    timescaledb.segmentby = 'symbol' );
  

• Rename a column for a continuous aggregate:

  ALTER MATERIALIZED VIEW contagg_view RENAME COLUMN old_name TO new_name;
  

Arguments

The syntax is:

ALTER MATERIALIZED VIEW <view_name> SET ( timescaledb.<argument> =  <value> [, ... ] )

Name	Type	Default	Required	Description
view_name	TEXT	-	✖	The name of the continuous aggregate view to be altered.
timescaledb.materialized_only	BOOLEAN	true	✖	Enable real-time aggregation.
timescaledb.enable_columnstore	BOOLEAN	true	✖	Since TimescaleDB v2.18.0 Enable columnstore. Effectively the same as timescaledb.compress.
timescaledb.compress	TEXT	Disabled.	✖	Enable compression.
timescaledb.orderby	TEXT	Descending order on the time column in table_name.	✖	Since TimescaleDB v2.18.0 Set the order in which items are used in the columnstore. Specified in the same way as an ORDER BY clause in a SELECT query.
timescaledb.compress_orderby	TEXT	Descending order on the time column in table_name.	✖	Set the order used by compression. Specified in the same way as the ORDER BY clause in a SELECT query.
timescaledb.segmentby	TEXT	No segementation by column.	✖	Since TimescaleDB v2.18.0 Set the list of columns used to segment data in the columnstore for table. An identifier representing the source of the data such as device_id or tags_id is usually a good candidate.
timescaledb.compress_segmentby	TEXT	No segementation by column.	✖	Set the list of columns used to segment the compressed data. An identifier representing the source of the data such as device_id or tags_id is usually a good candidate.
column_name	TEXT	-	✖	Set the name of the column to order by or segment by.
timescaledb.compress_chunk_time_interval	TEXT	-	✖	Reduce the total number of compressed/columnstore chunks for table. If you set compress_chunk_time_interval, compressed/columnstore chunks are merged with the previous adjacent chunk within chunk_time_interval whenever possible. These chunks are irreversibly merged. If you call to decompress/convert_to_rowstore, merged chunks are not split up. You can call compress_chunk_time_interval independently of other compression settings; timescaledb.compress/timescaledb.enable_columnstore is not required.
timescaledb.enable_cagg_window_functions	BOOLEAN	false	✖	EXPERIMENTAL: enable window functions on continuous aggregates. Support is experimental, as there is a risk of data inconsistency. For example, in backfill scenarios, buckets could be missed.
timescaledb.chunk_interval (formerly timescaledb.chunk_time_interval)	INTERVAL	10x the original hypertable.	✖	Set the chunk interval. Renamed in TimescaleDB V2.20.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/cagg_migrate/ =====

cagg_migrate()

Migrate a continuous aggregate from the old format to the new format introduced in TimescaleDB 2.7.

CALL cagg_migrate (
    cagg REGCLASS,
    override BOOLEAN DEFAULT FALSE,
    drop_old BOOLEAN DEFAULT FALSE
);

TimescaleDB 2.7 introduced a new format for continuous aggregates that improves performance. It also makes continuous aggregates compatible with more types of SQL queries.

The new format, also called the finalized format, stores the continuous aggregate data exactly as it appears in the final view. The old format, also called the partial format, stores the data in a partially aggregated state.

Use this procedure to migrate continuous aggregates from the old format to the new format.

For more information, see the migration how-to guide.

There are known issues with cagg_migrate() in version TimescaleDB 2.8.0. Upgrade to version 2.8.1 or above before using it.

Required arguments

Name	Type	Description
cagg	REGCLASS	The continuous aggregate to migrate

Optional arguments

Name	Type	Description
override	BOOLEAN	If false, the old continuous aggregate keeps its name. The new continuous aggregate is named <OLD_CONTINUOUS_AGGREGATE_NAME>_new. If true, the new continuous aggregate gets the old name. The old continuous aggregate is renamed <OLD_CONTINUOUS_AGGREGATE_NAME>_old. Defaults to false.
drop_old	BOOLEAN	If true, the old continuous aggregate is deleted. Must be used together with override. Defaults to false.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/drop_materialized_view/ =====

DROP MATERIALIZED VIEW (Continuous Aggregate)

Continuous aggregate views can be dropped using the DROP MATERIALIZED VIEW statement.

This statement deletes the continuous aggregate and all its internal objects. It also removes refresh policies for that aggregate. To delete other dependent objects, such as a view defined on the continuous aggregate, add the CASCADE option. Dropping a continuous aggregate does not affect the data in the underlying hypertable from which the continuous aggregate is derived.

DROP MATERIALIZED VIEW <view_name>;

Samples

Drop existing continuous aggregate.

DROP MATERIALIZED VIEW contagg_view;

Parameters

Name	Type	Description
<view_name>	TEXT	Name (optionally schema-qualified) of continuous aggregate view to be dropped.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/remove_all_policies/ =====

remove_all_policies()

Remove all policies from a continuous aggregate. The removed columnstore and retention policies apply to the continuous aggregate, not to the original hypertable.

timescaledb_experimental.remove_all_policies(
     relation REGCLASS,
     if_exists BOOL = false
) RETURNS BOOL

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

Samples

Remove all policies from a continuous aggregate named example_continuous_aggregate. This includes refresh policies, columnstore policies, and data retention policies. It doesn't include custom jobs:

SELECT timescaledb_experimental.remove_all_policies('example_continuous_aggregate');

Required arguments

Name	Type	Description
relation	REGCLASS	The continuous aggregate to remove all policies from

Optional arguments

Name	Type	Description
if_exists	BOOL	When true, prints a warning instead of erroring if any policies are missing. Defaults to false.

Returns

Returns true if successful.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/hypertable_detailed_size/ =====

hypertable_detailed_size()

hypertable_detailed_size()

Get detailed information about disk space used by a hypertable or continuous aggregate, returning size information for the table itself, any indexes on the table, any toast tables, and the total size of all. All sizes are reported in bytes. If the function is executed on a distributed hypertable, it returns size information as a separate row per node, including the access node.

When a continuous aggregate name is provided, the function transparently looks up the backing hypertable and returns its statistics instead.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

Samples

Get the size information for a hypertable.

-- disttable is a distributed hypertable --
SELECT * FROM hypertable_detailed_size('disttable') ORDER BY node_name;

 table_bytes | index_bytes | toast_bytes | total_bytes |  node_name
-------------+-------------+-------------+-------------+-------------
       16384 |       40960 |           0 |       57344 | data_node_1
        8192 |       24576 |           0 |       32768 | data_node_2
           0 |        8192 |           0 |        8192 |

The access node is listed without a user-given node name. Normally, the access node holds no data, but still maintains, for example, index information that occupies a small amount of disk space.

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable or continuous aggregate to show detailed size of.

Returns

Column	Type	Description
table_bytes	BIGINT	Disk space used by main_table (like pg_relation_size(main_table))
index_bytes	BIGINT	Disk space used by indexes
toast_bytes	BIGINT	Disk space of toast tables
total_bytes	BIGINT	Total disk space used by the specified table, including all indexes and TOAST data
node_name	TEXT	For distributed hypertables, this is the user-given name of the node for which the size is reported. NULL is returned for the access node and non-distributed hypertables.

If executed on a relation that is not a hypertable, the function returns NULL.

===== PAGE: https://docs.tigerdata.com/api/continuous-aggregates/show_policies/ =====

show_policies()

Show all policies that are currently set on a continuous aggregate.

timescaledb_experimental.show_policies(
     relation REGCLASS
) RETURNS SETOF JSONB

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

Samples

Given a continuous aggregate named example_continuous_aggregate, show all the policies set on it:

SELECT timescaledb_experimental.show_policies('example_continuous_aggregate');

Example of returned data:

show_policies
--------------------------------------------------------------------------------
{"policy_name": "policy_compression", "compress_after": 11, "compress_interval": "@ 1 day"}
{"policy_name": "policy_refresh_continuous_aggregate", "refresh_interval": "@ 1 hour", "refresh_end_offset": 1, "refresh_start_offset": 10}
{"drop_after": 20, "policy_name": "policy_retention", "retention_interval": "@ 1 day"}

Required arguments

Name	Type	Description
relation	REGCLASS	The continuous aggregate to display policies for

Returns

Column	Type	Description
show_policies	JSONB	Details for each policy set on the continuous aggregate

===== PAGE: https://docs.tigerdata.com/api/hypercore/alter_table/ =====

ALTER TABLE (hypercore)

Enable the columnstore or change the columnstore settings for a hypertable. The settings are applied on a per-chunk basis. You do not need to convert the entire hypertable back to the rowstore before changing the settings. The new settings apply only to the chunks that have not yet been converted to columnstore, the existing chunks in the columnstore do not change. This means that chunks with different columnstore settings can co-exist in the same hypertable.

TimescaleDB calculates default columnstore settings for each chunk when it is created. These settings apply to each chunk, and not the entire hypertable. To explicitly disable the defaults, set a setting to an empty string. To remove the current configuration and re-enable the defaults, call ALTER TABLE <your_table_name> RESET (<columnstore_setting>);.

After you have enabled the columnstore, either:

• add_columnstore_policy: create a job that automatically moves chunks in a hypertable to the columnstore at a specific time interval.
• convert_to_columnstore: manually add a specific chunk in a hypertable to the columnstore.

Since TimescaleDB v2.18.0

Samples

To enable the columnstore:

• Configure a hypertable that ingests device data to use the columnstore:

  In this example, the metrics hypertable is often queried about a specific device or set of devices. Segment the hypertable by device_id to improve query performance.

   ALTER TABLE metrics SET(
     timescaledb.enable_columnstore,
     timescaledb.orderby = 'time DESC',
     timescaledb.segmentby = 'device_id');
  

• Specify the chunk interval without changing other columnstore settings:

  • Set the time interval when chunks are added to the columnstore:

    ALTER TABLE metrics SET (timescaledb.compress_chunk_time_interval = '24 hours');
    

  • To disable the option you set previously, set the interval to 0:

    ALTER TABLE metrics SET (timescaledb.compress_chunk_time_interval = '0');
    

Arguments

The syntax is:

ALTER TABLE  SET (timescaledb.enable_columnstore,
   timescaledb.compress_orderby = '<column_name> [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]',
   timescaledb.compress_segmentby = '<column_name> [, ...]',
   timescaledb.sparse_index = '<index>(<column_name>), <index>(<column_name>)'
   timescaledb.compress_chunk_time_interval='interval',
   SET ACCESS METHOD { new_access_method | DEFAULT },
   ALTER <column name> SET NOT NULL,
   ADD CONSTRAINT <constraint_name> UNIQUE (<column name>, ... )
);

Name	Type	Default	Required	Description
table_name	TEXT	-	✖	The hypertable to enable columstore for.
timescaledb.enable_columnstore	BOOLEAN	true	✖	Set to false to disable columnstore.
timescaledb.compress_orderby	TEXT	Descending order on the time column in table_name.	✖	The order in which items are used in the columnstore. Specified in the same way as an ORDER BY clause in a SELECT query. Setting timescaledb.compress_orderby automatically creates an implicit min/max sparse index on the orderby column.
timescaledb.compress_segmentby	TEXT	TimescaleDB looks at pg_stats and determines an appropriate column based on the data cardinality and distribution. If pg_stats is not available, TimescaleDB looks for an appropriate column from the existing indexes.	✖	Set the list of columns used to segment data in the columnstore for table. An identifier representing the source of the data such as device_id or tags_id is usually a good candidate.
column_name	TEXT	-	✖	The name of the column to orderby or segmentby.
timescaledb.sparse_index	TEXT	TimescaleDB evaluates the columns you already have indexed, checks which data types are a good fit for sparse indexing, then creates a sparse index as an optimization.	✖	Configure the sparse indexes for compressed chunks. Requires setting timescaledb.compress_orderby. Supported index types include: bloom(<column_name>): a probabilistic index, effective for = filters. Cannot be applied to timescaledb.compress_orderby columns. minmax(<column_name>): stores min/max values for each compressed chunk. Setting timescaledb.compress_orderby automatically creates an implicit min/max sparse index on the orderby column. Define multiple indexes using a comma-separated list. You can set only one index per column. Set to an empty string to avoid using sparse indexes and explicitly disable the default behavior. To remove the current sparse index configuration and re-enable default sparse index selection, call ALTER TABLE your_table_name RESET (timescaledb.sparse_index);.
timescaledb.compress_chunk_time_interval	TEXT	-	✖	EXPERIMENTAL: reduce the total number of chunks in the columnstore for table. If you set compress_chunk_time_interval, chunks added to the columnstore are merged with the previous adjacent chunk within chunk_time_interval whenever possible. These chunks are irreversibly merged. If you call convert_to_rowstore, merged chunks are not split up. You can call compress_chunk_time_interval independently of other compression settings; timescaledb.enable_columnstore is not required.
interval	TEXT	-	✖	Set to a multiple of the chunk_time_interval for table.
ALTER	TEXT		✖	Set a specific column in the columnstore to be NOT NULL.
ADD CONSTRAINT	TEXT		✖	Add UNIQUE constraints to data in the columnstore.

===== PAGE: https://docs.tigerdata.com/api/hypercore/chunk_columnstore_stats/ =====

chunk_columnstore_stats()

Retrieve statistics about the chunks in the columnstore

chunk_columnstore_stats returns the size of chunks in the columnstore, these values are computed when you call either:

• add_columnstore_policy: create a job that automatically moves chunks in a hypertable to the columnstore at a specific time interval.
• convert_to_columnstore: manually add a specific chunk in a hypertable to the columnstore.

Inserting into a chunk in the columnstore does not change the chunk size. For more information about how to compute chunk sizes, see chunks_detailed_size.

Since TimescaleDB v2.18.0

Samples

To retrieve statistics about chunks:

• Show the status of the first two chunks in the conditions hypertable:

  SELECT * FROM chunk_columnstore_stats('conditions')
    ORDER BY chunk_name LIMIT 2;
  

  Returns:

  -[ RECORD 1 ]------------------+----------------------
  chunk_schema                   | _timescaledb_internal
  chunk_name                     | _hyper_1_1_chunk
  compression_status             | Uncompressed
  before_compression_table_bytes |
  before_compression_index_bytes |
  before_compression_toast_bytes |
  before_compression_total_bytes |
  after_compression_table_bytes  |
  after_compression_index_bytes  |
  after_compression_toast_bytes  |
  after_compression_total_bytes  |
  node_name                      |
  -[ RECORD 2 ]------------------+----------------------
  chunk_schema                   | _timescaledb_internal
  chunk_name                     | _hyper_1_2_chunk
  compression_status             | Compressed
  before_compression_table_bytes | 8192
  before_compression_index_bytes | 32768
  before_compression_toast_bytes | 0
  before_compression_total_bytes | 40960
  after_compression_table_bytes  | 8192
  after_compression_index_bytes  | 32768
  after_compression_toast_bytes  | 8192
  after_compression_total_bytes  | 49152
  node_name                      |
  

• Use pg_size_pretty to return a more human friendly format:

  SELECT pg_size_pretty(after_compression_total_bytes) AS total
    FROM chunk_columnstore_stats('conditions')
    WHERE compression_status = 'Compressed';
  

  Returns:

  -[ RECORD 1 ]--+------
  total | 48 kB
  

Arguments

Name	Type	Default	Required	Description
hypertable	REGCLASS	-	✖	The name of a hypertable

Returns

Column	Type	Description
chunk_schema	TEXT	Schema name of the chunk.
chunk_name	TEXT	Name of the chunk.
compression_status	TEXT	Current compression status of the chunk.
before_compression_table_bytes	BIGINT	Size of the heap before compression. Returns NULL if compression_status == Uncompressed.
before_compression_index_bytes	BIGINT	Size of all the indexes before compression. Returns NULL if compression_status == Uncompressed.
before_compression_toast_bytes	BIGINT	Size the TOAST table before compression. Returns NULL if compression_status == Uncompressed.
before_compression_total_bytes	BIGINT	Size of the entire chunk table (before_compression_table_bytes + before_compression_index_bytes + before_compression_toast_bytes) before compression. Returns NULL if compression_status == Uncompressed.
after_compression_table_bytes	BIGINT	Size of the heap after compression. Returns NULL if compression_status == Uncompressed.
after_compression_index_bytes	BIGINT	Size of all the indexes after compression. Returns NULL if compression_status == Uncompressed.
after_compression_toast_bytes	BIGINT	Size the TOAST table after compression. Returns NULL if compression_status == Uncompressed.
after_compression_total_bytes	BIGINT	Size of the entire chunk table (after_compression_table_bytes + after_compression_index_bytes + after_compression_toast_bytes) after compression. Returns NULL if compression_status == Uncompressed.
node_name	TEXT	DEPRECATED: nodes the chunk is located on, applicable only to distributed hypertables.

===== PAGE: https://docs.tigerdata.com/api/hypercore/convert_to_rowstore/ =====

convert_to_rowstore()

Manually convert a specific chunk in the hypertable columnstore to the rowstore.

If you need to modify or add a lot of data to a chunk in the columnstore, best practice is to stop any jobs moving chunks to the columnstore, convert the chunk back to the rowstore, then modify the data. After the update, convert the chunk to the columnstore and restart the jobs. This workflow is especially useful if you need to backfill old data.

Since TimescaleDB v2.18.0

Samples

To modify or add a lot of data to a chunk:

1. Stop the jobs that are automatically adding chunks to the columnstore

   Retrieve the list of jobs from the timescaledb_information.jobs view to find the job you need to alter_job.

   SELECT alter_job(JOB_ID, scheduled => false);
   

2. Convert a chunk to update back to the rowstore

   CALL convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk');
   

3. Update the data in the chunk you added to the rowstore

   Best practice is to structure your INSERT statement to include appropriate partition key values, such as the timestamp. TimescaleDB adds the data to the correct chunk:

   INSERT INTO metrics (time, value)
   VALUES ('2025-01-01T00:00:00', 42);
   

4. Convert the updated chunks back to the columnstore

   CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk');
   

5. Restart the jobs that are automatically converting chunks to the columnstore

   SELECT alter_job(JOB_ID, scheduled => true);
   

Arguments

Name	Type	Default	Required	Description
chunk	REGCLASS	-	✖	Name of the chunk to be moved to the rowstore.
if_compressed	BOOLEAN	true	✔	Set to false so this job fails with an error rather than an warning if chunk is not in the columnstore

===== PAGE: https://docs.tigerdata.com/api/hypercore/hypertable_columnstore_stats/ =====

hypertable_columnstore_stats()

Retrieve compression statistics for the columnstore.

For more information about using hypertables, including chunk size partitioning, see hypertables.

Since TimescaleDB v2.18.0

Samples

To retrieve compression statistics:

• Show the compression status of the conditions hypertable:

  SELECT * FROM hypertable_columnstore_stats('conditions');
  

  Returns:

  -[ RECORD 1 ]------------------+------
  total_chunks                   | 4
  number_compressed_chunks       | 1
  before_compression_table_bytes | 8192
  before_compression_index_bytes | 32768
  before_compression_toast_bytes | 0
  before_compression_total_bytes | 40960
  after_compression_table_bytes  | 8192
  after_compression_index_bytes  | 32768
  after_compression_toast_bytes  | 8192
  after_compression_total_bytes  | 49152
  node_name                      |
  

• Use pg_size_pretty get the output in a more human friendly format:

  SELECT pg_size_pretty(after_compression_total_bytes) as total
    FROM hypertable_columnstore_stats('conditions');
  

  Returns:

  -[ RECORD 1 ]--+------
  total | 48 kB
  

Arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable to show statistics for

Returns

Column	Type	Description
total_chunks	BIGINT	The number of chunks used by the hypertable. Returns NULL if compression_status == Uncompressed.
number_compressed_chunks	INTEGER	The number of chunks used by the hypertable that are currently compressed. Returns NULL if compression_status == Uncompressed.
before_compression_table_bytes	BIGINT	Size of the heap before compression. Returns NULL if compression_status == Uncompressed.
before_compression_index_bytes	BIGINT	Size of all the indexes before compression. Returns NULL if compression_status == Uncompressed.
before_compression_toast_bytes	BIGINT	Size the TOAST table before compression. Returns NULL if compression_status == Uncompressed.
before_compression_total_bytes	BIGINT	Size of the entire table (before_compression_table_bytes + before_compression_index_bytes + before_compression_toast_bytes) before compression. Returns NULL if compression_status == Uncompressed.
after_compression_table_bytes	BIGINT	Size of the heap after compression. Returns NULL if compression_status == Uncompressed.
after_compression_index_bytes	BIGINT	Size of all the indexes after compression. Returns NULL if compression_status == Uncompressed.
after_compression_toast_bytes	BIGINT	Size the TOAST table after compression. Returns NULL if compression_status == Uncompressed.
after_compression_total_bytes	BIGINT	Size of the entire table (after_compression_table_bytes + after_compression_index_bytes + after_compression_toast_bytes) after compression. Returns NULL if compression_status == Uncompressed.
node_name	TEXT	nodes on which the hypertable is located, applicable only to distributed hypertables. Returns NULL if compression_status == Uncompressed.

===== PAGE: https://docs.tigerdata.com/api/hypercore/remove_columnstore_policy/ =====

remove_columnstore_policy()

Remove a columnstore policy from a hypertable or continuous aggregate.

To restart automatic chunk migration to the columnstore, you need to call add_columnstore_policy again.

Since TimescaleDB v2.18.0

Samples

You see the columnstore policies in the informational views.

• Remove the columnstore policy from the cpu table:

  CALL remove_columnstore_policy('cpu');
  

• Remove the columnstore policy from the cpu_weekly continuous aggregate:

  CALL remove_columnstore_policy('cpu_weekly');
  

Arguments

Name	Type	Default	Required	Description
hypertable	REGCLASS	-	✔	Name of the hypertable or continuous aggregate to remove the policy from
if_exists	BOOLEAN	false	✖	Set to true so this job fails with a warning rather than an error if a columnstore policy does not exist on hypertable

===== PAGE: https://docs.tigerdata.com/api/hypercore/chunk_columnstore_settings/ =====

timescaledb_information.chunk_columnstore_settings

Retrieve the compression settings for each chunk in the columnstore.

Since TimescaleDB v2.18.0

Samples

To retrieve information about settings:

• Show settings for all chunks in the columnstore:

  SELECT * FROM timescaledb_information.chunk_columnstore_settings
  

  Returns:

  hypertable | chunk | segmentby | orderby
  ------------+-------+-----------+---------
  measurements | _timescaledb_internal._hyper_1_1_chunk| | "time" DESC
  

• Find all chunk columnstore settings for a specific hypertable:

  SELECT *
  FROM timescaledb_information.chunk_columnstore_settings
  WHERE hypertable::TEXT LIKE 'metrics';
  

  Returns:

  hypertable | chunk | segmentby | orderby
  ------------+-------+-----------+---------
  metrics | _timescaledb_internal._hyper_2_3_chunk | metric_id | "time"
  

Returns

Name	Type	Description
hypertable	REGCLASS	The name of the hypertable in the columnstore.
chunk	REGCLASS	The name of the chunk in the hypertable.
segmentby	TEXT	The list of columns used to segment the hypertable.
orderby	TEXT	The list of columns used to order the data in the hypertable, along with the ordering and NULL ordering information.
index	TEXT	The sparse index details.

===== PAGE: https://docs.tigerdata.com/api/hypercore/add_columnstore_policy/ =====

add_columnstore_policy()

Create a job that automatically moves chunks in a hypertable to the columnstore after a specific time interval.

You enable the columnstore a hypertable or continuous aggregate before you create a columnstore policy. You do this by calling CREATE TABLE for hypertables and ALTER MATERIALIZED VIEW for continuous aggregates. When columnstore is enabled, bloom filters are enabled by default, and every new chunk has a bloom index. If you converted chunks to columnstore using TimescaleDB v2.19.3 or below, to enable bloom filters on that data you have to convert those chunks to the rowstore, then convert them back to the columnstore.

Bloom indexes are not retrofitted, meaning that the existing chunks need to be fully recompressed to have the bloom indexes present. Please check out the PR description for more in-depth explanations of how bloom filters in TimescaleDB work.

To view the policies that you set or the policies that already exist, see informational views, to remove a policy, see remove_columnstore_policy.

A columnstore policy is applied on a per-chunk basis. If you remove an existing policy and then add a new one, the new policy applies only to the chunks that have not yet been converted to columnstore. The existing chunks in the columnstore remain unchanged. This means that chunks with different columnstore settings can co-exist in the same hypertable.

Since TimescaleDB v2.18.0

Samples

To create a columnstore job:

1. Enable columnstore

   Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data. For example:

   • Use CREATE TABLE for a hypertable

     CREATE TABLE crypto_ticks (
        "time" TIMESTAMPTZ,
        symbol TEXT,
        price DOUBLE PRECISION,
        day_volume NUMERIC
     ) WITH (
       tsdb.hypertable,
       tsdb.partition_column='time',
       tsdb.segmentby='symbol',
       tsdb.orderby='time DESC'
     );
     

     If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

   • Use ALTER MATERIALIZED VIEW for a continuous aggregate

     ALTER MATERIALIZED VIEW assets_candlestick_daily set (
        timescaledb.enable_columnstore = true,
        timescaledb.segmentby = 'symbol' );
     

2. Add a policy to move chunks to the columnstore at a specific time interval

   For example:

   • 60 days after the data was added to the table:

     CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '60d');
     

   • 3 months prior to the moment you run the query:

     CALL add_columnstore_policy('crypto_ticks', created_before => INTERVAL '3 months');
     

   • With an integer-based time column:

     CALL add_columnstore_policy('table_with_bigint_time', BIGINT '600000');
     

   • Older than eight weeks:

     CALL add_columnstore_policy('cpu_weekly', INTERVAL '8 weeks');
     

   • Control the time your policy runs:

     When you use a policy with a fixed schedule, TimescaleDB uses the initial_start time to compute the next start time. When TimescaleDB finishes executing a policy, it picks the next available time on the schedule, skipping any candidate start times that have already passed.

     When you set the next_start time, it only changes the start time of the next immediate execution. It does not change the computation of the next scheduled execution after that next execution. To change the schedule so a policy starts at a specific time, you need to set initial_start. To change the next immediate execution, you need to set next_start. For example, to modify a policy to execute on a fixed schedule 15 minutes past the hour, and every hour, you need to set both initial_start and next_start using alter_job:

     select * from alter_job(1000, fixed_schedule => true, initial_start => '2025-07-11 10:15:00', next_start =>
     '2025-07-11 11:15:00');
     

3. View the policies that you set or the policies that already exist

   SELECT * FROM timescaledb_information.jobs
   WHERE proc_name='policy_compression';
   

   See timescaledb_information.jobs.

Arguments

Calls to add_columnstore_policy require either after or created_before, but cannot have both.

Name	Type	Default	Required	Description
hypertable	REGCLASS	-	✔	Name of the hypertable or continuous aggregate to run this job on.
after	INTERVAL or INTEGER	-	✖	Add chunks containing data older than now - {after}::interval to the columnstore. Use an object type that matchs the time column type in hypertable: TIMESTAMP, TIMESTAMPTZ, or DATE: use an INTERVAL type. Integer-based timestamps : set an integer type using the integer_now_func. after is mutually exclusive with created_before.
created_before	INTERVAL	NULL	✖	Add chunks with a creation time of now() - created_before to the columnstore. created_before is Not supported for continuous aggregates. Mutually exclusive with after.
schedule_interval	INTERVAL	12 hours when chunk_time_interval >= 1 day for hypertable. Otherwise chunk_time_interval / 2.	✖	Set the interval between the finish time of the last execution of this policy and the next start.
initial_start	TIMESTAMPTZ	The interval from the finish time of the last execution to the next_start.	✖	Set the time this job is first run. This is also the time that next_start is calculated from.
next_start	TIMESTAMPTZ	-	✖	Set the start time of the next immediate execution. It does not change the computation of the next scheduled time after the next execution.
timezone	TEXT	UTC. However, daylight savings time(DST) changes may shift this alignment.	✖	Set to a valid time zone to mitigate DST shifting. If initial_start is set, subsequent executions of this policy are aligned on initial_start.
if_not_exists	BOOLEAN	false	✖	Set to true so this job fails with a warning rather than an error if a columnstore policy already exists on hypertable

===== PAGE: https://docs.tigerdata.com/api/hypercore/hypertable_columnstore_settings/ =====

timescaledb_information.hypertable_columnstore_settings

Retrieve information about the settings for all hypertables in the columnstore.

Since TimescaleDB v2.18.0

Samples

To retrieve information about settings:

• Show columnstore settings for all hypertables:

  SELECT * FROM timescaledb_information.hypertable_columnstore_settings;
  

  Returns:

  hypertable               | measurements
  segmentby                |
  orderby                  | "time" DESC
  compress_interval_length |
  

• Retrieve columnstore settings for a specific hypertable:

  SELECT * FROM timescaledb_information.hypertable_columnstore_settings WHERE hypertable::TEXT LIKE 'metrics';
  

  Returns:

  hypertable               | metrics
  segmentby                | metric_id
  orderby                  | "time"
  compress_interval_length |
  

Returns

Name	Type	Description
hypertable	REGCLASS	A hypertable which has the columnstore enabled.
segmentby	TEXT	The list of columns used to segment data.
orderby	TEXT	List of columns used to order the data, along with ordering and NULL ordering information.
compress_interval_length	TEXT	Interval used for rolling up chunks during compression.
index	TEXT	The sparse index details.

===== PAGE: https://docs.tigerdata.com/api/hypercore/convert_to_columnstore/ =====

convert_to_columnstore()

Manually convert a specific chunk in the hypertable rowstore to the columnstore.

Although convert_to_columnstore gives you more fine-grained control, best practice is to use add_columnstore_policy. You can also add chunks to the columnstore at a specific time running the job associated with your columnstore policy manually.

To move a chunk from the columnstore back to the rowstore, use convert_to_rowstore.

Since TimescaleDB v2.18.0

Samples

To convert a single chunk to columnstore:

CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk');

Arguments

Name	Type	Default	Required	Description
chunk	REGCLASS	-	✔	Name of the chunk to add to the columnstore.
if_not_columnstore	BOOLEAN	true	✖	Set to false so this job fails with an error rather than a warning if chunk is already in the columnstore.
recompress	BOOLEAN	false	✖	Set to true to add a chunk that had more data inserted after being added to the columnstore.

Returns

Calls to convert_to_columnstore return:

Column	Type	Description
chunk name or table	REGCLASS or String	The name of the chunk added to the columnstore, or a table-like result set with zero or more rows.

===== PAGE: https://docs.tigerdata.com/api/compression/decompress_chunk/ =====

decompress_chunk()

Old API since TimescaleDB v2.18.0 Replaced by convert_to_rowstore().

Before decompressing chunks, stop any compression policy on the hypertable you are decompressing. You can use SELECT alter_job(JOB_ID, scheduled => false); to prevent scheduled execution.

Samples

Decompress a single chunk:

SELECT decompress_chunk('_timescaledb_internal._hyper_2_2_chunk');

Decompress all compressed chunks in a hypertable named metrics:

SELECT decompress_chunk(c, true) FROM show_chunks('metrics') c;

Required arguments

Name	Type	Description
chunk_name	REGCLASS	Name of the chunk to be decompressed.

Optional arguments

Name	Type	Description
if_compressed	BOOLEAN	Disabling this will make the function error out on chunks that are not compressed. Defaults to true.

Returns

Column	Type	Description
decompress_chunk	REGCLASS	Name of the chunk that was decompressed.

===== PAGE: https://docs.tigerdata.com/api/compression/remove_compression_policy/ =====

remove_compression_policy()

Old API since TimescaleDB v2.18.0 Replaced by remove_columnstore_policy().

If you need to remove the compression policy. To restart policy-based compression you need to add the policy again. To view the policies that already exist, see informational views.

Samples

Remove the compression policy from the 'cpu' table:

SELECT remove_compression_policy('cpu');

Remove the compression policy from the 'cpu_weekly' continuous aggregate:

SELECT remove_compression_policy('cpu_weekly');

Required arguments

Name	Type	Description
hypertable	REGCLASS	Name of the hypertable or continuous aggregate the policy should be removed from

Optional arguments

Name	Type	Description
if_exists	BOOLEAN	Setting to true causes the command to fail with a notice instead of an error if a compression policy does not exist on the hypertable. Defaults to false.

===== PAGE: https://docs.tigerdata.com/api/compression/alter_table_compression/ =====

ALTER TABLE (Compression)

Old API since TimescaleDB v2.18.0 Replaced by ALTER TABLE (Hypercore).

'ALTER TABLE' statement is used to turn on compression and set compression options.

By itself, this ALTER statement alone does not compress a hypertable. To do so, either create a compression policy using the add_compression_policy function or manually compress a specific hypertable chunk using the compress_chunk function.

The syntax is:

ALTER TABLE  SET (timescaledb.compress,
   timescaledb.compress_orderby = '<column_name> [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]',
   timescaledb.compress_segmentby = '<column_name> [, ...]',
   timescaledb.compress_chunk_time_interval='interval'
);

Samples

Configure a hypertable that ingests device data to use compression. Here, if the hypertable is often queried about a specific device or set of devices, the compression should be segmented using the device_id for greater performance.

ALTER TABLE metrics SET (timescaledb.compress, timescaledb.compress_orderby = 'time DESC', timescaledb.compress_segmentby = 'device_id');

You can also specify compressed chunk interval without changing other compression settings:

ALTER TABLE metrics SET (timescaledb.compress_chunk_time_interval = '24 hours');

To disable the previously set option, set the interval to 0:

ALTER TABLE metrics SET (timescaledb.compress_chunk_time_interval = '0');

Required arguments

Name	Type	Description
timescaledb.compress	BOOLEAN	Enable or disable compression

Optional arguments

Name	Type	Description
timescaledb.compress_orderby	TEXT	Order used by compression, specified in the same way as the ORDER BY clause in a SELECT query. The default is the descending order of the hypertable's time column.
timescaledb.compress_segmentby	TEXT	Column list on which to key the compressed segments. An identifier representing the source of the data such as device_id or tags_id is usually a good candidate. The default is no segment by columns.
timescaledb.compress_chunk_time_interval	TEXT	EXPERIMENTAL: Set compressed chunk time interval used to roll chunks into. This parameter compresses every chunk, and then irreversibly merges it into a previous adjacent chunk if possible, to reduce the total number of chunks in the hypertable. Note that chunks will not be split up during decompression. It should be set to a multiple of the current chunk interval. This option can be changed independently of other compression settings and does not require the timescaledb.compress argument.

Parameters

Name	Type	Description
table_name	TEXT	Hypertable that supports compression
column_name	TEXT	Column used to order by or segment by
interval	TEXT	Time interval used to roll compressed chunks into

===== PAGE: https://docs.tigerdata.com/api/compression/hypertable_compression_stats/ =====

hypertable_compression_stats()

Old API since TimescaleDB v2.18.0 Replaced by hypertable_columnstore_stats().

Get statistics related to hypertable compression. All sizes are in bytes.

For more information about using hypertables, including chunk size partitioning, see the hypertable section.

For more information about compression, see the compression section.

Samples

SELECT * FROM hypertable_compression_stats('conditions');

-[ RECORD 1 ]------------------+------
total_chunks                   | 4
number_compressed_chunks       | 1
before_compression_table_bytes | 8192
before_compression_index_bytes | 32768
before_compression_toast_bytes | 0
before_compression_total_bytes | 40960
after_compression_table_bytes  | 8192
after_compression_index_bytes  | 32768
after_compression_toast_bytes  | 8192
after_compression_total_bytes  | 49152
node_name                      |

Use pg_size_pretty get the output in a more human friendly format.

SELECT pg_size_pretty(after_compression_total_bytes) as total
  FROM hypertable_compression_stats('conditions');

-[ RECORD 1 ]--+------
total | 48 kB

Required arguments

Name	Type	Description
hypertable	REGCLASS	Hypertable to show statistics for

Returns

Column	Type	Description
total_chunks	BIGINT	The number of chunks used by the hypertable
number_compressed_chunks	BIGINT	The number of chunks used by the hypertable that are currently compressed
before_compression_table_bytes	BIGINT	Size of the heap before compression
before_compression_index_bytes	BIGINT	Size of all the indexes before compression
before_compression_toast_bytes	BIGINT	Size the TOAST table before compression
before_compression_total_bytes	BIGINT	Size of the entire table (table+indexes+toast) before compression
after_compression_table_bytes	BIGINT	Size of the heap after compression
after_compression_index_bytes	BIGINT	Size of all the indexes after compression
after_compression_toast_bytes	BIGINT	Size the TOAST table after compression
after_compression_total_bytes	BIGINT	Size of the entire table (table+indexes+toast) after compression
node_name	TEXT	nodes on which the hypertable is located, applicable only to distributed hypertables

Returns show NULL if the data is currently uncompressed.

===== PAGE: https://docs.tigerdata.com/api/compression/compress_chunk/ =====

compress_chunk()

Old API since TimescaleDB v2.18.0 Replaced by convert_to_columnstore().

The compress_chunk function is used for synchronous compression (or recompression, if necessary) of a specific chunk. This is most often used instead of the add_compression_policy function, when a user wants more control over the scheduling of compression. For most users, we suggest using the policy framework instead.

You can also compress chunks by running the job associated with your compression policy. compress_chunk gives you more fine-grained control by allowing you to target a specific chunk that needs compressing.

You can get a list of chunks belonging to a hypertable using the show_chunks function.

Samples

Compress a single chunk.

SELECT compress_chunk('_timescaledb_internal._hyper_1_2_chunk');

Required arguments

Name	Type	Description
chunk_name	REGCLASS	Name of the chunk to be compressed

Optional arguments

Name	Type	Description
if_not_compressed	BOOLEAN	Disabling this will make the function error out on chunks that are already compressed. Defaults to true.

Returns

Column	Type	Description
compress_chunk	REGCLASS	Name of the chunk that was compressed

===== PAGE: https://docs.tigerdata.com/api/compression/chunk_compression_stats/ =====

chunk_compression_stats()

Old API since TimescaleDB v2.18.0 Replaced by chunk_columnstore_stats().

Get chunk-specific statistics related to hypertable compression. All sizes are in bytes.

This function shows the compressed size of chunks, computed when the compress_chunk is manually executed, or when a compression policy processes the chunk. An insert into a compressed chunk does not update the compressed sizes. For more information about how to compute chunk sizes, see the chunks_detailed_size section.

Samples

SELECT * FROM chunk_compression_stats('conditions')
  ORDER BY chunk_name LIMIT 2;

-[ RECORD 1 ]------------------+----------------------
chunk_schema                   | _timescaledb_internal
chunk_name                     | _hyper_1_1_chunk
compression_status             | Uncompressed
before_compression_table_bytes |
before_compression_index_bytes |
before_compression_toast_bytes |
before_compression_total_bytes |
after_compression_table_bytes  |
after_compression_index_bytes  |
after_compression_toast_bytes  |
after_compression_total_bytes  |
node_name                      |
-[ RECORD 2 ]------------------+----------------------
chunk_schema                   | _timescaledb_internal
chunk_name                     | _hyper_1_2_chunk
compression_status             | Compressed
before_compression_table_bytes | 8192
before_compression_index_bytes | 32768
before_compression_toast_bytes | 0
before_compression_total_bytes | 40960
after_compression_table_bytes  | 8192
after_compression_index_bytes  | 32768
after_compression_toast_bytes  | 8192
after_compression_total_bytes  | 49152
node_name                      |

Use pg_size_pretty get the output in a more human friendly format.

SELECT pg_size_pretty(after_compression_total_bytes) AS total
  FROM chunk_compression_stats('conditions')
  WHERE compression_status = 'Compressed';

-[ RECORD 1 ]--+------
total | 48 kB

Required arguments

Name	Type	Description
hypertable	REGCLASS	Name of the hypertable

Returns

Column	Type	Description
chunk_schema	TEXT	Schema name of the chunk
chunk_name	TEXT	Name of the chunk
compression_status	TEXT	the current compression status of the chunk
before_compression_table_bytes	BIGINT	Size of the heap before compression (NULL if currently uncompressed)
before_compression_index_bytes	BIGINT	Size of all the indexes before compression (NULL if currently uncompressed)
before_compression_toast_bytes	BIGINT	Size the TOAST table before compression (NULL if currently uncompressed)
before_compression_total_bytes	BIGINT	Size of the entire chunk table (table+indexes+toast) before compression (NULL if currently uncompressed)
after_compression_table_bytes	BIGINT	Size of the heap after compression (NULL if currently uncompressed)
after_compression_index_bytes	BIGINT	Size of all the indexes after compression (NULL if currently uncompressed)
after_compression_toast_bytes	BIGINT	Size the TOAST table after compression (NULL if currently uncompressed)
after_compression_total_bytes	BIGINT	Size of the entire chunk table (table+indexes+toast) after compression (NULL if currently uncompressed)
node_name	TEXT	nodes on which the chunk is located, applicable only to distributed hypertables

===== PAGE: https://docs.tigerdata.com/api/compression/add_compression_policy/ =====

add_compression_policy()

Old API since TimescaleDB v2.18.0 Replaced by add_columnstore_policy().

Allows you to set a policy by which the system compresses a chunk automatically in the background after it reaches a given age.

Compression policies can only be created on hypertables or continuous aggregates that already have compression enabled. To set timescaledb.compress and other configuration parameters for hypertables, use the ALTER TABLE command. To enable compression on continuous aggregates, use the ALTER MATERIALIZED VIEW command. To view the policies that you set or the policies that already exist, see informational views.

Samples

Add a policy to compress chunks older than 60 days on the cpu hypertable.

SELECT add_compression_policy('cpu', compress_after => INTERVAL '60d');

Add a policy to compress chunks created 3 months before on the 'cpu' hypertable.

SELECT add_compression_policy('cpu', compress_created_before => INTERVAL '3 months');

Note above that when compress_after is used then the time data range present in the partitioning time column is used to select the target chunks. Whereas, when compress_created_before is used then the chunks which were created 3 months ago are selected.

Add a compress chunks policy to a hypertable with an integer-based time column:

SELECT add_compression_policy('table_with_bigint_time', BIGINT '600000');

Add a policy to compress chunks of a continuous aggregate called cpu_weekly, that are older than eight weeks:

SELECT add_compression_policy('cpu_weekly', INTERVAL '8 weeks');

Required arguments

Name	Type	Description
hypertable	REGCLASS	Name of the hypertable or continuous aggregate
compress_after	INTERVAL or INTEGER	The age after which the policy job compresses chunks. compress_after is calculated relative to the current time, so chunks containing data older than now - {compress_after}::interval are compressed. This argument is mutually exclusive with compress_created_before.
compress_created_before	INTERVAL	Chunks with creation time older than this cut-off point are compressed. The cut-off point is computed as now() - compress_created_before. Defaults to NULL. Not supported for continuous aggregates yet. This argument is mutually exclusive with compress_after.

The compress_after parameter should be specified differently depending on the type of the time column of the hypertable or continuous aggregate:

• For hypertables with TIMESTAMP, TIMESTAMPTZ, and DATE time columns: the time interval should be an INTERVAL type.
• For hypertables with integer-based timestamps: the time interval should be an integer type (this requires the integer_now_func to be set).

Optional arguments

Name	Type	Description
schedule_interval	INTERVAL	The interval between the finish time of the last execution and the next start. Defaults to 12 hours for hyper tables with a chunk_interval >= 1 day and chunk_interval / 2 for all other hypertables.
initial_start	TIMESTAMPTZ	Time the policy is first run. Defaults to NULL. If omitted, then the schedule interval is the interval from the finish time of the last execution to the next start. If provided, it serves as the origin with respect to which the next_start is calculated
timezone	TEXT	A valid time zone. If initial_start is also specified, subsequent executions of the compression policy are aligned on its initial start. However, daylight savings time (DST) changes may shift this alignment. Set to a valid time zone if this is an issue you want to mitigate. If omitted, UTC bucketing is performed. Defaults to NULL.
if_not_exists	BOOLEAN	Setting to true causes the command to fail with a warning instead of an error if a compression policy already exists on the hypertable. Defaults to false.

===== PAGE: https://docs.tigerdata.com/api/compression/recompress_chunk/ =====

recompress_chunk()

Old API since TimescaleDB v2.18.0 Replaced by convert_to_columnstore().

Recompresses a compressed chunk that had more data inserted after compression.

recompress_chunk(
    chunk REGCLASS,
    if_not_compressed BOOLEAN = false
)

You can also recompress chunks by running the job associated with your compression policy. recompress_chunk gives you more fine-grained control by allowing you to target a specific chunk.

recompress_chunk is deprecated since TimescaleDB v2.14 and will be removed in the future. The procedure is now a wrapper which calls compress_chunk instead of it.

recompress_chunk is implemented as an SQL procedure and not a function. Call the procedure with CALL. Don't use a SELECT statement.

recompress_chunk only works on chunks that have previously been compressed. To compress a chunk for the first time, use compress_chunk.

Samples

Recompress the chunk timescaledb_internal._hyper_1_2_chunk:

CALL recompress_chunk('_timescaledb_internal._hyper_1_2_chunk');

Required arguments

Name	Type	Description
chunk	REGCLASS	The chunk to be recompressed. Must include the schema, for example _timescaledb_internal, if it is not in the search path.

Optional arguments

Name	Type	Description
if_not_compressed	BOOLEAN	If true, prints a notice instead of erroring if the chunk is already compressed. Defaults to false.

Troubleshooting

In TimescaleDB 2.6.0 and above, recompress_chunk is implemented as a procedure. Previously, it was implemented as a function. If you are upgrading to TimescaleDB 2.6.0 or above, therecompress_chunk function could cause an error. For example, trying to run SELECT recompress_chunk(i.show_chunks, true) FROM... gives the following error:

ERROR:  recompress_chunk(regclass, boolean) is a procedure

To fix the error, use CALL instead of SELECT. You might also need to write a procedure to replace the full functionality in your SELECT statement. For example:

DO $$
DECLARE chunk regclass;
BEGIN
  FOR chunk IN SELECT format('%I.%I', chunk_schema, chunk_name)::regclass
  FROM timescaledb_information.chunks
  WHERE is_compressed = true
  LOOP
    RAISE NOTICE 'Recompressing %', chunk::text;
    CALL recompress_chunk(chunk, true);
  END LOOP;
END
$$;

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/saturating_add_pos/ =====

saturating_add_pos()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/saturating_multiply/ =====

saturating_mul()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/downsampling-intro/ =====

Downsample your data to visualize trends while preserving fewer data points. Downsampling replaces a set of values with a much smaller set that is highly representative of the original data. This is particularly useful for graphing applications.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/saturating_sub/ =====

saturating_sub()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gp_lttb/ =====

gp_lttb()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/saturating-math-intro/ =====

The saturating math hyperfunctions help you perform saturating math on integers. In saturating math, the final result is bounded. If the result of a normal mathematical operation exceeds either the minimum or maximum bound, the result of the corresponding saturating math operation is capped at the bound. For example, 2 + (-3) = -1. But in a saturating math function with a lower bound of 0, such as saturating_add_pos, the result is 0.

You can use saturating math to make sure your results don't overflow the allowed range of integers, or to force a result to be greater than or equal to zero.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/lttb/ =====

lttb()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/saturating_add/ =====

saturating_add()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/asap_smooth/ =====

asap_smooth()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/saturating_sub_pos/ =====

saturating_sub_pos()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/timeline_agg/ =====

state_agg()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/state_timeline/ =====

state_timeline()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/interpolated_state_timeline/ =====

interpolated_state_timeline()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/interpolated_duration_in/ =====

interpolated_duration_in()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/duration_in/ =====

duration_in()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/intro/ =====

Given a system or value that switches between discrete states, track transitions between the states. For example, you can use state_agg to create a state of state transitions, or to calculate the durations of states. state_agg extends the capabilities of compact_state_agg.

state_agg is designed to work with a relatively small number of states. It might not perform well on datasets where states are mostly distinct between rows.

Because state_agg tracks more information, it uses more memory than compact_state_agg. If you want to minimize memory use and don't need to query the timestamps of state transitions, consider using compact_state_agg instead.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/into_values/ =====

into_values()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/rollup/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/state_at/ =====

state_at()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/interpolated_state_periods/ =====

interpolated_state_periods()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/state_agg/state_periods/ =====

state_periods()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_bucket_gapfill/interpolate/ =====

interpolate()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_bucket_gapfill/time_bucket_gapfill/ =====

time_bucket_gapfill()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_bucket_gapfill/intro/ =====

Aggregate data by time interval, while filling in gaps of missing data.

time_bucket_gapfill works similarly to time_bucket, but adds gapfilling capabilities. The other functions in this group must be used in the same query as time_bucket_gapfill. They control how missing values are treated.

time_bucket_gapfill must be used as a top-level expression in a query or subquery. You cannot, for example, nest time_bucket_gapfill in another function (such as round(time_bucket_gapfill(...))), or cast the result of the gapfilling call. If you need to cast, you can use time_bucket_gapfill in a subquery, and let the outer query do the type cast.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_bucket_gapfill/locf/ =====

locf()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/tdigest/tdigest/ =====

tdigest()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/tdigest/mean/ =====

mean()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/tdigest/approx_percentile/ =====

approx_percentile()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/tdigest/num_vals/ =====

num_vals()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/tdigest/intro/ =====

Estimate the value at a given percentile, or the percentile rank of a given value, using the t-digest algorithm. This estimation is more memory- and CPU-efficient than an exact calculation using Postgres's percentile_cont and percentile_disc functions.

tdigest is one of two advanced percentile approximation aggregates provided in TimescaleDB Toolkit. It is a space-efficient aggregation, and it provides more accurate estimates at extreme quantiles than traditional methods.

tdigest is somewhat dependent on input order. If tdigest is run on the same data arranged in different order, the results should be nearly equal, but they are unlikely to be exact.

The other advanced percentile approximation aggregate is uddsketch, which produces stable estimates within a guaranteed relative error. If you aren't sure which to use, try the default percentile estimation method, percentile_agg. It uses the uddsketch algorithm with some sensible defaults.

For more information about percentile approximation algorithms, see the algorithms overview.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/tdigest/approx_percentile_rank/ =====

approx_percentile_rank()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/tdigest/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/min_n_by/min_n_by/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/min_n_by/intro/ =====

Get the N smallest values from a column, with an associated piece of data per value. For example, you can return an accompanying column, or the full row.

The min_n_by() functions give the same results as the regular SQL query SELECT ... ORDER BY ... LIMIT n. But unlike the SQL query, they can be composed and combined like other aggregate hyperfunctions.

To get the N largest values with accompanying data, use max_n_by(). To get the N smallest values without accompanying data, use min_n().

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/min_n_by/into_values/ =====

into_values()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/min_n_by/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/live_ranges/ =====

live_ranges()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/interpolate/ =====

interpolate()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/downtime/ =====

downtime()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/interpolated_uptime/ =====

interpolated_uptime()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/uptime/ =====

uptime()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/num_gaps/ =====

num_gaps()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/trim_to/ =====

trim_to()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/intro/ =====

Given a series of timestamped heartbeats and a liveness interval, determine the overall liveness of a system. This aggregate can be used to report total uptime or downtime as well as report the time ranges where the system was live or dead.

It's also possible to combine multiple heartbeat aggregates to determine the overall health of a service. For example, the heartbeat aggregates from a primary and standby server could be combined to see if there was ever a window where both machines were down at the same time.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/dead_ranges/ =====

dead_ranges()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/live_at/ =====

live_at()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/heartbeat_agg/ =====

heartbeat_agg()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/rollup/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/num_live_ranges/ =====

num_live_ranges()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/heartbeat_agg/interpolated_downtime/ =====

interpolated_downtime()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/min_n/min_n/ =====

min_n()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/min_n/intro/ =====

Get the N smallest values from a column.

The min_n() functions give the same results as the regular SQL query SELECT ... ORDER BY ... LIMIT n. But unlike the SQL query, they can be composed and combined like other aggregate hyperfunctions.

To get the N largest values, use max_n(). To get the N smallest values with accompanying data, use min_n_by().

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/min_n/into_array/ =====

into_array()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/min_n/into_values/ =====

into_values()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/min_n/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/max_n_by/intro/ =====

Get the N largest values from a column, with an associated piece of data per value. For example, you can return an accompanying column, or the full row.

The max_n_by() functions give the same results as the regular SQL query SELECT ... ORDER BY ... LIMIT n. But unlike the SQL query, they can be composed and combined like other aggregate hyperfunctions.

To get the N smallest values with accompanying data, use min_n_by(). To get the N largest values without accompanying data, use max_n().

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/max_n_by/into_values/ =====

into_values()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/max_n_by/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/max_n_by/max_n_by/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/kurtosis/ =====

kurtosis()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/num_vals/ =====

num_vals()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/intro/ =====

Perform common statistical analyses, such as calculating averages and standard deviations, using this group of functions. These functions are similar to the Postgres statistical aggregates, but they include more features and are easier to use in continuous aggregates and window functions.

These functions work on one-dimensional data. To work with two-dimensional data, for example to perform linear regression, see the two-dimensional stats_agg functions.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/sum/ =====

sum()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/stats_agg/ =====

stats_agg() (one variable)

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/average/ =====

average()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/stddev/ =====

stddev()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/skewness/ =====

skewness()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/rolling/ =====

rolling()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-one-variable/variance/ =====

variance()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/delta/ =====

delta()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/idelta_left/ =====

idelta_left()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/intro/ =====

Analyze data coming from gauges. Unlike counters, gauges can decrease as well as increase.

If your value can only increase, use counter_agg instead to appropriately account for resets.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/irate_right/ =====

irate_right()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/extrapolated_delta/ =====

extrapolated_delta()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/interpolated_delta/ =====

interpolated_delta()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/irate_left/ =====

irate_left()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/num_changes/ =====

num_changes()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/interpolated_rate/ =====

interpolated_rate()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/intercept/ =====

intercept()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/extrapolated_rate/ =====

extrapolated_rate()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/rollup/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/gauge_zero_time/ =====

gauge_zero_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/corr/ =====

corr()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/idelta_right/ =====

idelta_right()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/gauge_agg/ =====

gauge_agg()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/rate/ =====

rate()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/with_bounds/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/time_delta/ =====

time_delta()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/slope/ =====

slope()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/gauge_agg/num_elements/ =====

num_elements()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/open/ =====

open()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/low/ =====

low()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/candlestick/ =====

candlestick()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/volume/ =====

volume()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/candlestick_agg/ =====

candlestick_agg()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/low_time/ =====

low_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/intro/ =====

Perform analysis of financial asset data. These specialized hyperfunctions make it easier to write financial analysis queries that involve candlestick data.

They help you answer questions such as:

• What are the opening and closing prices of these stocks?
• When did the highest price occur for this stock?

This function group uses the two-step aggregation pattern. In addition to the usual aggregate function, candlestick_agg, it also includes the pseudo-aggregate function candlestick. candlestick_agg produces a candlestick aggregate from raw tick data, which can then be used with the accessor and rollup functions in this group. candlestick takes pre-aggregated data and transforms it into the same format that candlestick_agg produces. This allows you to use the accessors and rollups with existing candlestick data.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/close_time/ =====

close_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/close/ =====

close()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/open_time/ =====

open_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/vwap/ =====

vwap()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/rollup/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/high/ =====

high()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/candlestick_agg/high_time/ =====

high_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/count_min_sketch/approx_count/ =====

approx_count()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/count_min_sketch/intro/ =====

Count the number of times a value appears in a column, using the probabilistic count-min sketch data structure and its associated algorithms. For applications where a small error rate is tolerable, this can result in huge savings in both CPU time and memory, especially for large datasets.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/count_min_sketch/count_min_sketch/ =====

count_min_sketch()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/freq_agg/topn/ =====

topn()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/freq_agg/intro/ =====

Get the most common elements of a set and their relative frequency. The estimation uses the SpaceSaving algorithm.

This group of functions contains two aggregate functions, which let you set the cutoff for keeping track of a value in different ways. freq_agg allows you to specify a minimum frequency, and mcv_agg allows you to specify the target number of values to keep.

To estimate the absolute number of times a value appears, use count_min_sketch.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/freq_agg/min_frequency/ =====

min_frequency()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/freq_agg/freq_agg/ =====

freq_agg()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/freq_agg/max_frequency/ =====

max_frequency()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/freq_agg/into_values/ =====

into_values()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/freq_agg/rollup/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/freq_agg/mcv_agg/ =====

mcv_agg()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/compact_state_agg/interpolated_duration_in/ =====

interpolated_duration_in()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/compact_state_agg/duration_in/ =====

duration_in()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/compact_state_agg/intro/ =====

Given a system or value that switches between discrete states, aggregate the amount of time spent in each state. For example, you can use the compact_state_agg functions to track how much time a system spends in error, running, or starting states.

compact_state_agg is designed to work with a relatively small number of states. It might not perform well on datasets where states are mostly distinct between rows.

If you need to track when each state is entered and exited, use the state_agg functions. If you need to track the liveness of a system based on a heartbeat signal, consider using the heartbeat_agg functions.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/compact_state_agg/compact_state_agg/ =====

compact_state_agg()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/compact_state_agg/into_values/ =====

into_values()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/compact_state_agg/rollup/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/hyperloglog/intro/ =====

Estimate the number of distinct values in a dataset. This is also known as cardinality estimation. For large datasets and datasets with high cardinality (many distinct values), this can be much more efficient in both CPU and memory than an exact count using count(DISTINCT).

The estimation uses the hyperloglog++ algorithm. If you aren't sure what parameters to set for the hyperloglog, try using the approx_count_distinct aggregate, which sets some reasonable default values.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/hyperloglog/distinct_count/ =====

distinct_count()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/hyperloglog/hyperloglog/ =====

hyperloglog()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/hyperloglog/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/hyperloglog/stderror/ =====

stderror()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/hyperloglog/approx_count_distinct/ =====

approx_count_distinct()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/max_n/max_n/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/max_n/intro/ =====

Get the N largest values from a column.

The max_n() functions give the same results as the regular SQL query SELECT ... ORDER BY ... LIMIT n. But unlike the SQL query, they can be composed and combined like other aggregate hyperfunctions.

To get the N smallest values, use min_n(). To get the N largest values with accompanying data, use max_n_by().

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/max_n/into_array/ =====

into_array()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/max_n/into_values/ =====

into_values()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/max_n/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/interpolated_integral/ =====

interpolated_integral()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/first_time/ =====

first_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/intro/ =====

Calculate time-weighted summary statistics, such as averages (means) and integrals. Time weighting is used when data is unevenly sampled over time. In that case, a straight average gives misleading results, as it biases towards more frequently sampled values.

For example, a sensor might silently spend long periods of time in a steady state, and send data only when a significant change occurs. The regular mean counts the steady-state reading as only a single point, whereas a time-weighted mean accounts for the long period of time spent in the steady state. In essence, the time-weighted mean takes an integral over time, then divides by the elapsed time.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/last_val/ =====

last_val()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/interpolated_average/ =====

interpolated_average()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/average/ =====

average()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/first_val/ =====

first_val()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/time_weight/ =====

time_weight()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/integral/ =====

integral()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/time_weight/last_time/ =====

last_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/sum_y_x/ =====

sum_y() | sum_x()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/kurtosis_y_x/ =====

kurtosis_y() | kurtosis_x()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/x_intercept/ =====

x_intercept()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/determination_coeff/ =====

determination_coeff()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/variance_y_x/ =====

variance_y() | variance_x()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/skewness_y_x/ =====

skewness_y() | skewness_x()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/num_vals/ =====

num_vals()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/intro/ =====

Perform linear regression analysis, for example to calculate correlation coefficient and covariance, on two-dimensional data. You can also calculate common statistics, such as average and standard deviation, on each dimension separately. These functions are similar to the Postgres statistical aggregates, but they include more features and are easier to use in continuous aggregates and window functions. The linear regressions are based on the standard least-squares fitting method.

These functions work on two-dimensional data. To work with one-dimensional data, for example to calculate the average and standard deviation of a single variable, see the one-dimensional stats_agg functions.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/stats_agg/ =====

stats_agg() (two variables)

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/average_y_x/ =====

average_y() | average_x()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/intercept/ =====

intercept()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/stddev_y_x/ =====

stddev_y() | stddev_x()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/corr/ =====

corr()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/covariance/ =====

covariance()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/rolling/ =====

rolling()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/stats_agg-two-variables/slope/ =====

slope()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/uddsketch/ =====

uddsketch()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/percentile_agg/ =====

percentile_agg()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/mean/ =====

mean()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/approx_percentile/ =====

approx_percentile()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/num_vals/ =====

num_vals()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/intro/ =====

Estimate the value at a given percentile, or the percentile rank of a given value, using the UddSketch algorithm. This estimation is more memory- and CPU-efficient than an exact calculation using Postgres's percentile_cont and percentile_disc functions.

uddsketch is one of two advanced percentile approximation aggregates provided in TimescaleDB Toolkit. It produces stable estimates within a guaranteed relative error.

The other advanced percentile approximation aggregate is tdigest, which is more accurate at extreme quantiles, but is somewhat dependent on input order.

If you aren't sure which aggregate to use, try the default percentile estimation method, percentile_agg. It uses the uddsketch algorithm with some sensible defaults.

For more information about percentile approximation algorithms, see the algorithms overview.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/approx_percentile_rank/ =====

approx_percentile_rank()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/error/ =====

error()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/rollup/ =====

rollup()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/uddsketch/approx_percentile_array/ =====

approx_percentile_array()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/delta/ =====

delta()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/idelta_left/ =====

idelta_left()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/first_time/ =====

first_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/intro/ =====

Analyze data whose values are designed to monotonically increase, and where any decreases are treated as resets. The counter_agg functions simplify this task, which can be difficult to do in pure SQL.

If it's possible for your readings to decrease as well as increase, use gauge_agg instead.

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/irate_right/ =====

irate_right()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/last_val/ =====

last_val()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/extrapolated_delta/ =====

extrapolated_delta()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/interpolated_delta/ =====

interpolated_delta()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/counter_zero_time/ =====

counter_zero_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/irate_left/ =====

irate_left()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/num_changes/ =====

num_changes()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/interpolated_rate/ =====

interpolated_rate()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/intercept/ =====

intercept()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/extrapolated_rate/ =====

extrapolated_rate()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/rollup/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/corr/ =====

corr()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/idelta_right/ =====

idelta_right()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/first_val/ =====

first_val()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/num_resets/ =====

num_resets()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/last_time/ =====

last_time()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/counter_agg/ =====

counter_agg()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/rate/ =====

rate()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/with_bounds/ =====

API Reference

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/time_delta/ =====

time_delta()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/slope/ =====

slope()

===== PAGE: https://docs.tigerdata.com/api/_hyperfunctions/counter_agg/num_elements/ =====

num_elements()

===== PAGE: https://docs.tigerdata.com/migrate/dual-write-and-backfill/dual-write-from-timescaledb/ =====

Migrate from TimescaleDB using dual-write and backfill

This document provides detailed step-by-step instructions to migrate data using the dual-write and backfill migration method from a source database which is using TimescaleDB to Tiger Cloud.

In the context of migrations, your existing production database is referred to as the SOURCE database, the Tiger Cloud service that you are migrating your data to is the TARGET.

In detail, the migration process consists of the following steps:

1. Set up a target Tiger Cloud service.
2. Modify the application to write to a secondary database.
3. Migrate schema and relational data from source to target.
4. Start the application in dual-write mode.
5. Determine the completion point T.
6. Backfill time-series data from source to target.
7. Enable background jobs (policies) in the target database.
8. Validate that all data is present in target database.
9. Validate that target database can handle production load.
10. Switch application to treat target database as primary (potentially continuing to write into source database, as a backup).

If you get stuck, you can get help by either opening a support request, or take your issue to the #migration channel in the community slack, where the developers of this migration method are there to help.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

1. Set up a target database instance in Tiger Cloud

Create a Tiger Cloud service.

If you intend on migrating more than 400 GB, open a support request to ensure that enough disk is pre-provisioned on your Tiger Cloud service.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

2. Modify the application to write to the target database

How exactly to do this is dependent on the language that your application is written in, and on how exactly your ingestion and application function. In the simplest case, you simply execute two inserts in parallel. In the general case, you must think about how to handle the failure to write to either the source or target database, and what mechanism you want to or can build to recover from such a failure.

Should your time-series data have foreign-key references into a plain table, you must ensure that your application correctly maintains the foreign key relations. If the referenced column is a *SERIAL type, the same row inserted into the source and target may not obtain the same autogenerated id. If this happens, the data backfilled from the source to the target is internally inconsistent. In the best case it causes a foreign key violation, in the worst case, the foreign key constraint is maintained, but the data references the wrong foreign key. To avoid these issues, best practice is to follow live migration.

You may also want to execute the same read queries on the source and target database to evaluate the correctness and performance of the results which the queries deliver. Bear in mind that the target database spends a certain amount of time without all data being present, so you should expect that the results are not the same for some period (potentially a number of days).

3. Set up schema and migrate relational data to target database

This section leverages pg_dumpall and pg_dump to migrate the roles and relational schema that you are using in the source database to the target database.

The PostgresSQL versions of the source and target databases can be of different versions, as long as the target version is greater than that of the source.

The version of TimescaleDB used in both databases must be exactly the same.

For the sake of convenience, connection strings to the source and target databases are referred to as source and target throughout this guide.

This can be set in your shell, for example:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://<user>:<password>@<target host>:<target port>/<db_name>"

3a. Dump the database roles from the source database

pg_dumpall -d "source" \
  -l database name \
  --quote-all-identifiers \
  --roles-only \
  --file=roles.sql

Tiger Cloud services do not support roles with superuser access. If your SQL dump includes roles that have such permissions, you'll need to modify the file to be compliant with the security model.

You can use the following sed command to remove unsupported statements and permissions from your roles.sql file:

sed -i -E \
-e '/CREATE ROLE "postgres";/d' \
-e '/ALTER ROLE "postgres"/d' \
-e '/CREATE ROLE "tsdbadmin";/d' \
-e '/ALTER ROLE "tsdbadmin"/d' \
-e 's/(NO)*SUPERUSER//g' \
-e 's/(NO)*REPLICATION//g' \
-e 's/(NO)*BYPASSRLS//g' \
-e 's/GRANTED BY "[^"]*"//g' \
roles.sql

This command works only with the GNU implementation of sed (sometimes referred to as gsed). For the BSD implementation (the default on macOS), you need to add an extra argument to change the -i flag to -i ''.

To check the sed version, you can use the command sed --version. While the GNU version explicitly identifies itself as GNU, the BSD version of sed generally doesn't provide a straightforward --version flag and simply outputs an "illegal option" error.

A brief explanation of this script is:

• CREATE ROLE "postgres"; and ALTER ROLE "postgres": These statements are removed because they require superuser access, which is not supported by Timescale.

• (NO)SUPERUSER | (NO)REPLICATION | (NO)BYPASSRLS: These are permissions that require superuser access.

• GRANTED BY role_specification: The GRANTED BY clause can also have permissions that require superuser access and should therefore be removed. Note: according to the TimescaleDB documentation, the GRANTOR in the GRANTED BY clause must be the current user, and this clause mainly serves the purpose of SQL compatibility. Therefore, it's safe to remove it.

3b. Dump all plain tables and the TimescaleDB catalog from the source database

pg_dump -d "source" \
  --format=plain \
  --quote-all-identifiers \
  --no-tablespaces \
  --no-owner \
  --no-privileges \
  --exclude-table-data='_timescaledb_internal.*' \
  --file=dump.sql

• --exclude-table-data='_timescaledb_internal.*' dumps the structure of the hypertable chunks, but not the data. This creates empty chunks on the target, ready for the backfill process.

• --no-tablespaces is required because Tiger Cloud does not support tablespaces other than the default. This is a known limitation.

• --no-owner is required because Tiger Cloud's tsdbadmin user is not a superuser and cannot assign ownership in all cases. This flag means that everything is owned by the user used to connect to the target, regardless of ownership in the source. This is a known limitation.

• --no-privileges is required because the tsdbadmin user for your Tiger Cloud service is not a superuser and cannot assign privileges in all cases. This flag means that privileges assigned to other users must be reassigned in the target database as a manual clean-up task. This is a known limitation.

If the source database has the TimescaleDB extension installed in a schema other than "public" it causes issues on Tiger Cloud. Edit the dump file to remove any references to the non-public schema. The extension must be in the "public" schema on Tiger Cloud. This is a known limitation.

3c. Ensure that the correct TimescaleDB version is installed

It is very important that the version of the TimescaleDB extension is the same in the source and target databases. This requires upgrading the TimescaleDB extension in the source database before migrating.

You can determine the version of TimescaleDB in the target database with the following command:

psql target -c "SELECT extversion FROM pg_extension WHERE extname = 'timescaledb';"

To update the TimescaleDB extension in your source database, first ensure that the desired version is installed from your package repository. Then you can upgrade the extension with the following query:

psql source -c "ALTER EXTENSION timescaledb UPDATE TO '<version here>';"

For more information and guidance, consult the Upgrade TimescaleDB page.

3d. Load the roles and schema into the target database, and turn off all background jobs

psql -X -d "target" \
  -v ON_ERROR_STOP=1 \
  --echo-errors \
  -f roles.sql \
  -c 'select public.timescaledb_pre_restore();' \
  -f dump.sql \
  -f - <<'EOF'
begin;
select public.timescaledb_post_restore();

-- disable all background jobs
select public.alter_job(id::integer, scheduled=>false)
from _timescaledb_config.bgw_job
where id >= 1000
;
commit;
EOF

Background jobs are turned off to prevent continuous aggregate refresh jobs from updating the continuous aggregate with incomplete/missing data. The continuous aggregates must be manually updated in the required range once the migration is complete.

4. Start application in dual-write mode

With the target database set up, your application can now be started in dual-write mode.

5. Determine the completion point T

After dual-writes have been executing for a while, the target hypertable contains data in three time ranges: missing writes, late-arriving data, and the "consistency" range

Missing writes

If the application is made up of multiple writers, and these writers did not all simultaneously start writing into the target hypertable, there is a period of time in which not all writes have made it into the target hypertable. This period starts when the first writer begins dual-writing, and ends when the last writer begins dual-writing.

Late-arriving data

Some applications have late-arriving data: measurements which have a timestamp in the past, but which weren't written yet (for example from devices which had intermittent connectivity issues). The window of late-arriving data is between the present moment, and the maximum lateness.

Consistency range

The consistency range is the range in which there are no missing writes, and in which all data has arrived, that is between the end of the missing writes range and the beginning of the late-arriving data range.

The length of these ranges is defined by the properties of the application, there is no one-size-fits-all way to determine what they are.

Completion point

The completion point T is an arbitrarily chosen time in the consistency range. It is the point in time to which data can safely be backfilled, ensuring that there is no data loss.

The completion point should be expressed as the type of the time column of the hypertables to be backfilled. For instance, if you're using a TIMESTAMPTZ time column, then the completion point may be 2023-08-10T12:00:00.00Z. If you're using a BIGINT column it may be 1695036737000.

If you are using a mix of types for the time columns of your hypertables, you must determine the completion point for each type individually, and backfill each set of hypertables with the same type independently from those of other types.

6. Backfill data from source to target

The simplest way to backfill from TimescaleDB, is to use the timescaledb-backfill backfill tool. It efficiently copies hypertables with the columnstore or compression enabled, and data stored in continuous aggregates from one database to another.

timescaledb-backfill performs best when executed from a machine located close to the target database. The ideal scenario is an EC2 instance located in the same region as the Tiger Cloud service. Use a Linux-based distribution on x86_64.

With the instance that will run the timescaledb-backfill ready, log in and download timescaledb-backfill:

wget https://assets.timescale.com/releases/timescaledb-backfill-x86_64-linux.tar.gz
tar xf timescaledb-backfill-x86_64-linux.tar.gz
sudo mv timescaledb-backfill /usr/local/bin/

Running timescaledb-backfill is a four-phase process:

1. Stage: This step prepares metadata about the data to be copied in the target database. On completion, it outputs the number of chunks to be copied.

   timescaledb-backfill stage --source source --target target --until <completion point>
   

2. Copy: This step copies data on a chunk-by-chunk basis from the source to the target. If it fails or is interrupted, it can safely be resumed. You should be aware of the --parallelism parameter, which dictates how many connections are used to copy data. The default is 8, which, depending on the size of your source and target databases, may be too high or too low. You should closely observe the performance of your source database and tune this parameter accordingly.

   timescaledb-backfill copy --source source --target target
   

3. Verify (optional): This step verifies that the data in the source and target is the same. It reads all the data on a chunk-by-chunk basis from both the source and target databases, so may also impact the performance of your source database.

   timescaledb-backfill verify --source source --target target
   

4. Clean: This step removes the metadata which was created in the target database by the stage command.

   timescaledb-backfill clean --target target
   

7. Enable background jobs in target database

Before enabling the jobs, verify if any continuous aggregate refresh policies exist.

psql -d target \
  -c "select count(*)
  from _timescaledb_config.bgw_job
  where proc_name = 'policy_refresh_continuous_aggregate'"

If they do exist, refresh the continuous aggregates before re-enabling the jobs. The timescaledb-backfill tool provides a utility to do this:

timescaledb-backfill refresh-caggs --source source --target target

Once the continuous aggregates are updated, you can re-enable all background jobs:

psql -d target -f - <<EOF
  select public.alter_job(id::integer, scheduled=>true)
  from _timescaledb_config.bgw_job
  where id >= 1000;
EOF

If the backfill process took long enough for there to be significant retention/compression work to be done, it may be preferable to run the jobs manually to have control over the pacing of the work until it is caught up before re-enabling.

8. Validate that all data is present in target database

Now that all data has been backfilled, and the application is writing data to both databases, the contents of both databases should be the same. How exactly this should best be validated is dependent on your application.

If you are reading from both databases in parallel for every production query, you could consider adding an application-level validation that both databases are returning the same data.

Another option is to compare the number of rows in the source and target tables, although this reads all data in the table which may have an impact on your production workload. timescaledb-backfill's verify subcommand performs this check.

Another option is to run ANALYZE on both the source and target tables and then look at the reltuples column of the pg_class table on a chunk-by-chunk basis. The result is not exact, but doesn't require reading all rows from the table.

9. Validate that target database can handle production load

Now that dual-writes have been in place for a while, the target database should be holding up to production write traffic. Now would be the right time to determine if the target database can serve all production traffic (both reads and writes). How exactly this is done is application-specific and up to you to determine.

10. Switch production workload to target database

Once you've validated that all the data is present, and that the target database can handle the production workload, the final step is to switch to the target database as your primary. You may want to continue writing to the source database for a period, until you are certain that the target database is holding up to all production traffic.

===== PAGE: https://docs.tigerdata.com/migrate/dual-write-and-backfill/dual-write-from-other/ =====

Migrate from non-Postgres using dual-write and backfill

This document provides detailed step-by-step instructions to migrate data using the dual-write and backfill migration method from a source database which is not using Postgres to Tiger Cloud.

In the context of migrations, your existing production database is referred to as the SOURCE database, the Tiger Cloud service that you are migrating your data to is the TARGET.

In detail, the migration process consists of the following steps:

1. Set up a target Tiger Cloud service.
2. Modify the application to write to a secondary database.
3. Set up schema and migrate relational data to target database.
4. Start the application in dual-write mode.
5. Determine the completion point T.
6. Backfill time-series data from source to target.
7. Enable background jobs (policies) in the target database.
8. Validate that all data is present in target database.
9. Validate that target database can handle production load.
10. Switch application to treat target database as primary (potentially continuing to write into source database, as a backup).

If you get stuck, you can get help by either opening a support request, or take your issue to the #migration channel in the community slack, where the developers of this migration method are there to help.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

1. Set up a target database instance in Tiger Cloud

Create a Tiger Cloud service.

If you intend on migrating more than 400 GB, open a support request to ensure that enough disk is pre-provisioned on your Tiger Cloud service.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

2. Modify the application to write to the target database

How exactly to do this is dependent on the language that your application is written in, and on how exactly your ingestion and application function. In the simplest case, you simply execute two inserts in parallel. In the general case, you must think about how to handle the failure to write to either the source or target database, and what mechanism you want to or can build to recover from such a failure.

Should your time-series data have foreign-key references into a plain table, you must ensure that your application correctly maintains the foreign key relations. If the referenced column is a *SERIAL type, the same row inserted into the source and target may not obtain the same autogenerated id. If this happens, the data backfilled from the source to the target is internally inconsistent. In the best case it causes a foreign key violation, in the worst case, the foreign key constraint is maintained, but the data references the wrong foreign key. To avoid these issues, best practice is to follow live migration.

You may also want to execute the same read queries on the source and target database to evaluate the correctness and performance of the results which the queries deliver. Bear in mind that the target database spends a certain amount of time without all data being present, so you should expect that the results are not the same for some period (potentially a number of days).

3. Set up schema and migrate relational data to target database

Describing exactly how to migrate your data from every possible source is not feasible, instead we tell you what needs to be done, and hope that you find resources to support you.

In this step, you need to prepare the database to receive time-series data which is dual-written from your application. If you're migrating from another time-series database then you only need to worry about setting up the schema for the hypertables which will contain time-series data. For some background on what hypertables are, consult the tables and hypertables section of the getting started guide.

If you're migrating from a relational database containing both relational and time-series data, you also need to set up the schema for the relational data, and copy it over in this step, excluding any of the time-series data. The time-series data is backfilled in a subsequent step.

Our assumption in the dual-write and backfill scenario is that the volume of relational data is either very small in relation to the time-series data, so that it is not problematic to briefly stop your production application while you copy the relational data, or that it changes infrequently, so you can get a snapshot of the relational metadata without stopping your application. If this is not the case for your application, you should reconsider using the dual-write and backfill method.

If you're planning on experimenting with continuous aggregates, we recommend that you first complete the dual-write and backfill migration, and only then create continuous aggregates on the data. If you create continuous aggregates on a hypertable before backfilling data into it, you must refresh the continuous aggregate over the whole time range to ensure that there are no holes in the aggregated data.

4. Start application in dual-write mode

With the target database set up, your application can now be started in dual-write mode.

5. Determine the completion point T

After dual-writes have been executing for a while, the target hypertable contains data in three time ranges: missing writes, late-arriving data, and the "consistency" range

Missing writes

If the application is made up of multiple writers, and these writers did not all simultaneously start writing into the target hypertable, there is a period of time in which not all writes have made it into the target hypertable. This period starts when the first writer begins dual-writing, and ends when the last writer begins dual-writing.

Late-arriving data

Some applications have late-arriving data: measurements which have a timestamp in the past, but which weren't written yet (for example from devices which had intermittent connectivity issues). The window of late-arriving data is between the present moment, and the maximum lateness.

Consistency range

The consistency range is the range in which there are no missing writes, and in which all data has arrived, that is between the end of the missing writes range and the beginning of the late-arriving data range.

The length of these ranges is defined by the properties of the application, there is no one-size-fits-all way to determine what they are.

Completion point

The completion point T is an arbitrarily chosen time in the consistency range. It is the point in time to which data can safely be backfilled, ensuring that there is no data loss.

The completion point should be expressed as the type of the time column of the hypertables to be backfilled. For instance, if you're using a TIMESTAMPTZ time column, then the completion point may be 2023-08-10T12:00:00.00Z. If you're using a BIGINT column it may be 1695036737000.

If you are using a mix of types for the time columns of your hypertables, you must determine the completion point for each type individually, and backfill each set of hypertables with the same type independently from those of other types.

6. Backfill data from source to target

Dump the data from your source database on a per-table basis into CSV format, and restore those CSVs into the target database using the timescaledb-parallel-copy tool.

6a. Determine the time range of data to be copied

Determine the window of data that to be copied from the source database to the target. Depending on the volume of data in the source table, it may be sensible to split the source table into multiple chunks of data to move independently. In the following steps, this time range is called <start> and <end>.

Usually the time column is of type timestamp with time zone, so the values of <start> and <end> must be something like 2023-08-01T00:00:00Z. If the time column is not a timestamp with time zone then the values of <start> and <end> must be the correct type for the column.

If you intend to copy all historic data from the source table, then the value of <start> can be '-infinity', and the <end> value is the value of the completion point T that you determined.

6b. Remove overlapping data in the target

The dual-write process may have already written data into the target database in the time range that you want to move. In this case, the dual-written data must be removed. This can be achieved with a DELETE statement, as follows:

psql target -c "DELETE FROM <hypertable> WHERE time >= <start> AND time < <end>);"

The BETWEEN operator is inclusive of both the start and end ranges, so it is not recommended to use it.

6d. Copy the data

Refer to the documentation for your source database in order to determine how to dump a table into a CSV. You must ensure the CSV contains only data before the completion point. You should apply this filter when dumping the data from the source database.

You can load a CSV file into a hypertable using timescaledb-parallel-copy as follows. Set the number of workers equal to the number of CPU cores in your target database:

timescaledb-parallel-copy \
  --connection target \
  --table <target_hypertable> \
  --workers 8 \
  --file

The above command is not transactional. If there is a connection issue, or some other issue which causes it to stop copying, the partially copied rows must be removed from the target (using the instructions in step 6b above), and then the copy can be restarted.

6e. Enable policies that compress data in the target hypertable

In the following command, replace <hypertable> with the fully qualified table name of the target hypertable, for example public.metrics:

psql -d target -f -v hypertable=<hypertable> - <<'EOF'
SELECT public.alter_job(j.id, scheduled=>true)
FROM _timescaledb_config.bgw_job j
JOIN _timescaledb_catalog.hypertable h ON h.id = j.hypertable_id
WHERE j.proc_schema IN ('_timescaledb_internal', '_timescaledb_functions')
  AND j.proc_name = 'policy_compression'
  AND j.id >= 1000
  AND format('%I.%I', h.schema_name, h.table_name)::text::regclass = :'hypertable'::text::regclass;
EOF

7. Validate that all data is present in target database

Now that all data has been backfilled, and the application is writing data to both databases, the contents of both databases should be the same. How exactly this should best be validated is dependent on your application.

If you are reading from both databases in parallel for every production query, you could consider adding an application-level validation that both databases are returning the same data.

Another option is to compare the number of rows in the source and target tables, although this reads all data in the table which may have an impact on your production workload.

8. Validate that target database can handle production load

Now that dual-writes have been in place for a while, the target database should be holding up to production write traffic. Now would be the right time to determine if the target database can serve all production traffic (both reads and writes). How exactly this is done is application-specific and up to you to determine.

9. Switch production workload to target database

Once you've validated that all the data is present, and that the target database can handle the production workload, the final step is to switch to the target database as your primary. You may want to continue writing to the source database for a period, until you are certain that the target database is holding up to all production traffic.

===== PAGE: https://docs.tigerdata.com/migrate/dual-write-and-backfill/dual-write-from-postgres/ =====

Migrate from Postgres using dual-write and backfill

This document provides detailed step-by-step instructions to migrate data using the dual-write and backfill migration method from a source database which is using Postgres to Tiger Cloud.

In the context of migrations, your existing production database is referred to as the SOURCE database, the Tiger Cloud service that you are migrating your data to is the TARGET.

In detail, the migration process consists of the following steps:

1. Set up a target Tiger Cloud service.
2. Modify the application to write to the target database.
3. Migrate schema and relational data from source to target.
4. Start the application in dual-write mode.
5. Determine the completion point T.
6. Backfill time-series data from source to target.
7. Validate that all data is present in target database.
8. Validate that target database can handle production load.
9. Switch application to treat target database as primary (potentially continuing to write into source database, as a backup).

If you get stuck, you can get help by either opening a support request, or take your issue to the #migration channel in the community slack, where the developers of this migration method are there to help.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

1. Set up a target database instance in Tiger Cloud

Create a Tiger Cloud service.

If you intend on migrating more than 400 GB, open a support request to ensure that enough disk is pre-provisioned on your Tiger Cloud service.

You can open a support request directly from Tiger Cloud Console, or by email to support@tigerdata.com.

2. Modify the application to write to the target database

How exactly to do this is dependent on the language that your application is written in, and on how exactly your ingestion and application function. In the simplest case, you simply execute two inserts in parallel. In the general case, you must think about how to handle the failure to write to either the source or target database, and what mechanism you want to or can build to recover from such a failure.

Should your time-series data have foreign-key references into a plain table, you must ensure that your application correctly maintains the foreign key relations. If the referenced column is a *SERIAL type, the same row inserted into the source and target may not obtain the same autogenerated id. If this happens, the data backfilled from the source to the target is internally inconsistent. In the best case it causes a foreign key violation, in the worst case, the foreign key constraint is maintained, but the data references the wrong foreign key. To avoid these issues, best practice is to follow live migration.

You may also want to execute the same read queries on the source and target database to evaluate the correctness and performance of the results which the queries deliver. Bear in mind that the target database spends a certain amount of time without all data being present, so you should expect that the results are not the same for some period (potentially a number of days).

3. Set up schema and migrate relational data to target database

You would probably like to convert some of your large tables which contain time-series data into hypertables. This step consists of identifying those tables, excluding their data from the database dump, copying the database schema and tables, and setting up the time-series tables as hypertables. The data is backfilled into these hypertables in a subsequent step.

For the sake of convenience, connection strings to the source and target databases are referred to as source and target throughout this guide.

This can be set in your shell, for example:

export SOURCE="postgres://<user>:<password>@<source host>:<source port>/<db_name>"
export TARGET="postgres://<user>:<password>@<target host>:<target port>/<db_name>"

3a. Dump the database roles from the source database

pg_dumpall -d "source" \
  -l database name \
  --quote-all-identifiers \
  --roles-only \
  --file=roles.sql

Tiger Cloud services do not support roles with superuser access. If your SQL dump includes roles that have such permissions, you'll need to modify the file to be compliant with the security model.

You can use the following sed command to remove unsupported statements and permissions from your roles.sql file:

sed -i -E \
-e '/CREATE ROLE "postgres";/d' \
-e '/ALTER ROLE "postgres"/d' \
-e '/CREATE ROLE "tsdbadmin";/d' \
-e '/ALTER ROLE "tsdbadmin"/d' \
-e 's/(NO)*SUPERUSER//g' \
-e 's/(NO)*REPLICATION//g' \
-e 's/(NO)*BYPASSRLS//g' \
-e 's/GRANTED BY "[^"]*"//g' \
roles.sql

This command works only with the GNU implementation of sed (sometimes referred to as gsed). For the BSD implementation (the default on macOS), you need to add an extra argument to change the -i flag to -i ''.

To check the sed version, you can use the command sed --version. While the GNU version explicitly identifies itself as GNU, the BSD version of sed generally doesn't provide a straightforward --version flag and simply outputs an "illegal option" error.

A brief explanation of this script is:

• CREATE ROLE "postgres"; and ALTER ROLE "postgres": These statements are removed because they require superuser access, which is not supported by Timescale.

• (NO)SUPERUSER | (NO)REPLICATION | (NO)BYPASSRLS: These are permissions that require superuser access.

• GRANTED BY role_specification: The GRANTED BY clause can also have permissions that require superuser access and should therefore be removed. Note: according to the TimescaleDB documentation, the GRANTOR in the GRANTED BY clause must be the current user, and this clause mainly serves the purpose of SQL compatibility. Therefore, it's safe to remove it.

3b. Determine which tables to convert to hypertables

Ideal candidates for hypertables are large tables containing time-series data. This is usually data with some form of timestamp value (TIMESTAMPTZ, TIMESTAMP, BIGINT, INT etc.) as the primary dimension, and some other measurement values.

3c. Dump all tables from the source database, excluding data from hypertable candidates

pg_dump -d "source" \
  --format=plain \
  --quote-all-identifiers \
  --no-tablespaces \
  --no-owner \
  --no-privileges \
  --exclude-table-data= \
  --file=dump.sql

• --exclude-table-data is used to exclude all data from hypertable candidates. You can either specify a table pattern, or specify --exclude-table-data multiple times, once for each table to be converted.

• --no-tablespaces is required because Tiger Cloud does not support tablespaces other than the default. This is a known limitation.

• --no-owner is required because Tiger Cloud's tsdbadmin user is not a superuser and cannot assign ownership in all cases. This flag means that everything is owned by the user used to connect to the target, regardless of ownership in the source. This is a known limitation.

• --no-privileges is required because the tsdbadmin user for your Tiger Cloud service is not a superuser and cannot assign privileges in all cases. This flag means that privileges assigned to other users must be reassigned in the target database as a manual clean-up task. This is a known limitation.

3d. Load the roles and schema into the target database

psql -X -d "target" \
  -v ON_ERROR_STOP=1 \
  --echo-errors \
  -f roles.sql \
  -f dump.sql

3e. Convert the plain tables to hypertables, optionally compress data in the columnstore

For each table which should be converted to a hypertable in the target database, execute:

SELECT create_hypertable('', by_range('<time column name>'));

The by_range dimension builder is an addition to TimescaleDB 2.13. For simpler cases, like this one, you can also create the hypertable using the old syntax:

SELECT create_hypertable('', '<time column name>');

For more information about the options which you can pass to create_hypertable, consult the create_table API reference. For more information about hypertables in general, consult the hypertable documentation.

You may also wish to consider taking advantage of some of Tiger Cloud's killer features, such as:

• retention policies to automatically drop unneeded data
• tiered storage to automatically move data to Tiger Cloud's low-cost bottomless object storage tier
• hypercore to reduce the size of your hypertables by compressing data in the columnstore
• continuous aggregates to write blisteringly fast aggregate queries on your data

4. Start application in dual-write mode

With the target database set up, your application can now be started in dual-write mode.

5. Determine the completion point T

After dual-writes have been executing for a while, the target hypertable contains data in three time ranges: missing writes, late-arriving data, and the "consistency" range

Missing writes

If the application is made up of multiple writers, and these writers did not all simultaneously start writing into the target hypertable, there is a period of time in which not all writes have made it into the target hypertable. This period starts when the first writer begins dual-writing, and ends when the last writer begins dual-writing.

Late-arriving data

Some applications have late-arriving data: measurements which have a timestamp in the past, but which weren't written yet (for example from devices which had intermittent connectivity issues). The window of late-arriving data is between the present moment, and the maximum lateness.

Consistency range

The consistency range is the range in which there are no missing writes, and in which all data has arrived, that is between the end of the missing writes range and the beginning of the late-arriving data range.

The length of these ranges is defined by the properties of the application, there is no one-size-fits-all way to determine what they are.

Completion point

The completion point T is an arbitrarily chosen time in the consistency range. It is the point in time to which data can safely be backfilled, ensuring that there is no data loss.

The completion point should be expressed as the type of the time column of the hypertables to be backfilled. For instance, if you're using a TIMESTAMPTZ time column, then the completion point may be 2023-08-10T12:00:00.00Z. If you're using a BIGINT column it may be 1695036737000.

If you are using a mix of types for the time columns of your hypertables, you must determine the completion point for each type individually, and backfill each set of hypertables with the same type independently from those of other types.

6. Backfill data from source to target

Dump the data from your source database on a per-table basis into CSV format, and restore those CSVs into the target database using the timescaledb-parallel-copy tool.

6a. Determine the time range of data to be copied

Determine the window of data that to be copied from the source database to the target. Depending on the volume of data in the source table, it may be sensible to split the source table into multiple chunks of data to move independently. In the following steps, this time range is called <start> and <end>.

Usually the time column is of type timestamp with time zone, so the values of <start> and <end> must be something like 2023-08-01T00:00:00Z. If the time column is not a timestamp with time zone then the values of <start> and <end> must be the correct type for the column.

If you intend to copy all historic data from the source table, then the value of <start> can be '-infinity', and the <end> value is the value of the completion point T that you determined.

6b. Remove overlapping data in the target

The dual-write process may have already written data into the target database in the time range that you want to move. In this case, the dual-written data must be removed. This can be achieved with a DELETE statement, as follows:

psql target -c "DELETE FROM <hypertable> WHERE time >= <start> AND time < <end>);"

The BETWEEN operator is inclusive of both the start and end ranges, so it is not recommended to use it.

6d. Copy the data with a streaming copy

Execute the following command, replacing <source table> and <hypertable> with the fully qualified names of the source table and target hypertable respectively:

psql source -f - <<EOF
  \copy ( \
      SELECT * FROM <source table> WHERE time >= <start> AND time < <end> \
    ) TO stdout WITH (format CSV);" | timescaledb-parallel-copy \
  --connection target \
  --table <hypertable> \
  --log-batches \
  --batch-size=1000 \
  --workers=4
EOF

The above command is not transactional. If there is a connection issue, or some other issue which causes it to stop copying, the partially copied rows must be removed from the target (using the instructions in step 6b above), and then the copy can be restarted.

6e. Enable policies that compress data in the target hypertable

In the following command, replace <hypertable> with the fully qualified table name of the target hypertable, for example public.metrics:

psql -d target -f -v hypertable=<hypertable> - <<'EOF'
SELECT public.alter_job(j.id, scheduled=>true)
FROM _timescaledb_config.bgw_job j
JOIN _timescaledb_catalog.hypertable h ON h.id = j.hypertable_id
WHERE j.proc_schema IN ('_timescaledb_internal', '_timescaledb_functions')
  AND j.proc_name = 'policy_compression'
  AND j.id >= 1000
  AND format('%I.%I', h.schema_name, h.table_name)::text::regclass = :'hypertable'::text::regclass;
EOF

7. Validate that all data is present in target database

Now that all data has been backfilled, and the application is writing data to both databases, the contents of both databases should be the same. How exactly this should best be validated is dependent on your application.

If you are reading from both databases in parallel for every production query, you could consider adding an application-level validation that both databases are returning the same data.

Another option is to compare the number of rows in the source and target tables, although this reads all data in the table which may have an impact on your production workload.

Another option is to run ANALYZE on both the source and target tables and then look at the reltuples column of the pg_class table. This is not exact, but doesn't require reading all rows from the table. Note: for hypertables, the reltuples value belongs to the chunk table, so you must take the sum of reltuples for all chunks belonging to the hypertable. If the chunk is compressed in one database, but not the other, then this check cannot be used.

8. Validate that target database can handle production load

Now that dual-writes have been in place for a while, the target database should be holding up to production write traffic. Now would be the right time to determine if the target database can serve all production traffic (both reads and writes). How exactly this is done is application-specific and up to you to determine.

9. Switch production workload to target database

Once you've validated that all the data is present, and that the target database can handle the production workload, the final step is to switch to the target database as your primary. You may want to continue writing to the source database for a period, until you are certain that the target database is holding up to all production traffic.

===== PAGE: https://docs.tigerdata.com/migrate/dual-write-and-backfill/timescaledb-backfill/ =====

Migrate with timescaledb-backfill

Dual-write and backfill is a method to write from your application to two databases at once, and gives tooling and guidance to move your existing data from the one database to the other. It is specifically catered for, and relies on, your data being predominantly append-only time-series data. As such, it comes with some caveats and prerequisites which live migration does not (dual-write and backfill does not support executing UPDATE or DELETE statements on your data). Additionally, it requires you to make changes to the ingest pipeline of your application.

The timescaledb-backfill tool is a command-line utility designed to support migrations from Tiger Cloud services by copying historic data from one database to another ("backfilling"). timescaledb-backfill efficiently copies hypertable and continuous aggregates chunks directly, without the need for intermediate storage, or converting chunks from the columnstore to the rowstore. It operates transactionally, ensuring data integrity throughout the migration process. It is designed to be used in the dual-write and backfill migration procedure.

Limitations

• The tool only supports backfilling of hypertables. Schema migrations and non-hypertable migrations should be handled separately before using this tool.
• The tool is optimized for append-only workloads. Other scenarios may not be fully supported.
• To prevent continuous aggregates from refreshing with incomplete data, any refresh and retention policies targeting the tables that are going to be backfilled should be turned off.

Installation

The tool performs best when executed in an instance located close to the target database. The ideal scenario is an EC2 instance located in the same region as the Tiger Cloud service. Use a Linux-based distribution on x86_64.

With the instance that will run the timescaledb-backfill ready, log in and download the tool's binary:

wget https://assets.timescale.com/releases/timescaledb-backfill-x86_64-linux.tar.gz
tar xf timescaledb-backfill-x86_64-linux.tar.gz
sudo mv timescaledb-backfill /usr/local/bin/

How to use

The timescaledb-backfill tool offers four main commands: stage, copy, verify and clean. The workflow involves creating tasks, copying chunks, verifying data integrity and cleaning up the administrative schema after the migration.

In the context of migrations, your existing production database is referred to as the SOURCE database, the Tiger Cloud service that you are migrating your data to is the TARGET.

• Stage Command: is used to create copy tasks for hypertable chunks based on the specified completion point (--until). If a starting point (--from) is not specified, data will be copied from the beginning of time up to the completion point (--until). An optional filter (--filter) can be used to refine the hypertables and continuous aggregates targeted for staging.

  timescaledb-backfill stage --source source --target target --until '2016-01-02T00:00:00'
  

  The tables to be included in the stage can be controlled by providing filtering options:

  --filter: this option accepts a POSIX regular expression to match schema-qualified hypertable names or continuous aggregate view names. Only hypertables and/or continuous aggregates matching the filter are staged.

  By default, the filter includes only the matching objects, and does not concern itself with dependencies between objects. Depending on what is intended, this could be problematic for continuous aggregates, as they form a dependency hierarchy. This behaviour can be modified through cascade options.

  For example, assuming a hierarchy of continuous aggregates for hourly, daily, and weekly rollups of data in an underlying hypertable called raw_data (all in the public schema). This could look as follows:

  raw_data -> hourly_agg -> daily_agg -> monthly_agg
  

  If the filter --filter='^public\.raw_data$' is applied, then no data from the continuous aggregates is staged. If the filter --filter='^public\.daily_agg$' is applied, then only materialized data in the continuous aggregate daily_agg is staged.

  --cascade-up: when activated, this option ensures that any continuous aggregates which depend on the filtered object are included in the staging process. It is called "cascade up" because it cascades up the hierarchy. Using the example from before, if the filter --filter='^public\.raw_data$' --cascade up is applied, the data in raw_data, hourly_agg, daily_agg, and monthly_agg is staged.

  --cascade-down: when activated, this option ensures that any objects which the filtered object depends on are included in the staging process. It is called "cascade down" because it cascades down the hierarchy. Using the example from before, if the filter --filter='^public\.daily_agg$' --cascade-down is applied, the data in daily_agg, hourly_agg, and raw_data is staged.

  The --cascade-up and --cascade-down options can be combined. Using the example from before, if the filter --filter='^public\.daily_agg$' --cascade-up --cascade-down is applied, data in all objects in the example scenario is staged.

  timescaledb-backfill stage --source source --target target \
    --until '2016-01-02T00:00:00' \
    --filter '^public\.daily_agg$' \
    --cascade-up \
    --cascade-down
  

• Copy Command: processes the tasks created during the staging phase and copies the corresponding hypertable chunks to the target Tiger Cloud service.

  timescaledb-backfill copy --source source --target target
  

  In addition to the --source and --target parameters, the copy command takes one optional parameter:

  --parallelism specifies the number of COPY jobs which will be run in parallel, the default is 8. It should ideally be set to the number of cores that the source and target database have, and is the most important parameter in dictating both how much load the source database experiences, and how quickly data is transferred from the source to the target database.

• Verify Command: checks for discrepancies between the source and target chunks' data. It compares the results of the count for each chunk's table, as well as per-column count, max, min, and sum values (when applicable, depending on the column data type).

  timescaledb-backfill verify --source source --target target
  

  In addition to the --source and --target parameters, the verify command takes one optional parameter:

  --parallelism specifies the number of verification jobs which will be run in parallel, the default is 8. It should ideally be set to the number of cores that the source and target database have, and is the most important parameter in dictating both how much load the source and target databases experience during verification, and how long it takes for verification to complete.

• Refresh Continuous Aggregates Command: refreshes the continuous aggregates of the target system. It covers the period from the last refresh in the target to the last refresh in the source, solving the problem of continuous aggregates being outdated beyond the coverage of the refresh policies.

  timescaledb-backfill refresh-caggs --source source --target target
  

  To refresh the continuous aggregates, the command executes the following SQL statement for all the matched continuous aggregates:

  CALL refresh_continuous_aggregate({CAGG NAME}, {TARGET_WATERMARK}, {SOURCE_WATERMARK})
  

  The continuous aggregates to be refreshed can be controlled by providing filtering options:

  --filter: this option accepts a POSIX regular expression to match schema-qualified hypertable continuous aggregate view names.

  By default, the filter includes only the matching objects, and does not concern itself with dependencies between objects. Depending on what is intended, this could be problematic as continuous aggregates form a dependency hierarchy. This behaviour can be modified through cascade options.

  For example, assuming a hierarchy of continuous aggregates for hourly, daily, and weekly rollups of data in an underlying hypertable called raw_data (all in the public schema). This could look as follows:

  raw_data -> hourly_agg -> daily_agg -> monthly_agg
  

  If the filter --filter='^public\.daily_agg$' is applied, only materialized data in the continuous aggregate daily_agg will be updated. However, this approach can lead to potential issues. For example, if hourly_agg is not up to date, then daily_agg won't be either, as it requires the missing data from hourly_agg. Additionally, it's important to remember to refresh monthly_agg at some point to ensure its data remains current. In both cases, relying solely on refresh policies may result in data gaps if the policy doesn't cover the entire required period.

  --cascade-up: when activated, this option ensures that any continuous aggregates which depend on the filtered object are refreshed. It is called "cascade up" because it cascades up the hierarchy. Using the example from before, if the filter --filter='^public\.daily_agg$' --cascade up is applied, the hourly_agg, daily_agg, and monthly_agg will be refreshed.

  --cascade-down: when activated, this option ensures that any continuous aggregates which the filtered object depends on are refreshed. It is called "cascade down" because it cascades down the hierarchy. Using the example from before, if the filter --filter='^public\.daily_agg$' --cascade-down is applied, the data in daily_agg and hourly_agg will be refreshed.

  The --cascade-up and --cascade-down options can be combined. Using the example from before, if the filter --filter='^public\.daily_agg$' --cascade-up --cascade-down is applied, then all the continuous aggregates will be refreshed.

• Clean Command: removes the administrative schema (__backfill) that was used to store the tasks once the migration is completed successfully.

  timescaledb-backfill clean --target target
  

Usage examples

• Backfilling with a filter and until date:

  timescaledb-backfill stage --source $SOURCE_DB --target $TARGET_DB \
    --filter '.*\.my_table.*' \
    --until '2016-01-02T00:00:00'
  
  timescaledb-backfill copy --source source --target target
  
  timescaledb-backfill refresh-caggs --source source --target target
  
  timescaledb-backfill verify --source source --target target
  
  timescaledb-backfill clean --target target
  

• Running multiple stages with different filters and until dates:

  timescaledb-backfill stage --source source --target target \
    --filter '^schema1\.table_with_time_as_timestampz$' \
    --until '2015-01-01T00:00:00'
  
  timescaledb-backfill stage --source source --target target \
    --filter '^schema1\.table_with_time_as_bigint$' \
    --until '91827364'
  
  timescaledb-backfill stage --source source --target target \
    --filter '^schema2\..*' \
    --until '2017-01-01T00:00:00'
  
  timescaledb-backfill copy --source source --target target
  
  timescaledb-backfill refresh-caggs --source source --target target
  
  timescaledb-backfill verify --source source --target target
  
  timescaledb-backfill clean --target target
  

• Backfilling a specific period of time with from and until:

  timescaledb-backfill stage --source $SOURCE_DB --target $TARGET_DB \
    --from '2015-01-02T00:00:00' \
    --until '2016-01-02T00:00:00'

  timescaledb-backfill copy --source source --target target

  timescaledb-backfill clean --target target

• Refreshing a continuous aggregates hierarchy

  timescaledb-backfill refresh-caggs --source source --target target \
    --filter='^public\.daily_agg$' --cascade-up --cascade-down

Stop and resume

The copy command can be safely stopped by sending an interrupt signal (SIGINT) to the process. This can be achieved by using the Ctrl-C keyboard shortcut from the terminal where the tool is currently running.

When the tool receives the first signal, it interprets it as a request for a graceful shutdown. It then notifies the copy workers that they should exit once they finish copying the chunk they are currently processing. Depending on the chunk size, this could take many minutes to complete.

When a second signal is received, it forces the tool to shut down immediately, interrupting all ongoing work. Due to the tool's usage of transactions, there is no risk of data inconsistency when using forced shutdown.

While a graceful shutdown waits for in-progress chunks to finish copying, a force shutdown rolls back the in-progress copy transactions. Any data copied into those chunks is lost, but the database is left in a transactional consistent state, and the backfill process can be safely resumed.

Inspect tasks progress

Each hypertable chunk that's going to be backfilled has a corresponding task stored in the target's database __backfill.task table. You can use this information to inspect the backfill's progress:

select
    hypertable_schema,
    hypertable_name,
    count(*) as total_chunks,
    count(worked) as finished_chunks,
    count(worked is null) pending_chunks
from __backfill.task
group by
    1,
    2

===== PAGE: https://docs.tigerdata.com/use-timescale/query-data/about-query-data/ =====

About querying data

Querying data in TimescaleDB works just like querying data in Postgres. You can reuse your existing queries if you're moving from another Postgres database.

TimescaleDB also provides some additional features to help with data analysis:

• Use PopSQL to work on data with centralized SQL queries, interactive visuals and real-time collaboration
• The SkipScan feature speeds up DISTINCT queries
• Hyperfunctions improve the experience of writing many data analysis queries
• Function pipelines bring functional programming to SQL queries, making it easier to perform consecutive transformations of data

===== PAGE: https://docs.tigerdata.com/use-timescale/query-data/select/ =====

SELECT data

You can query data from a hypertable using a standard SELECT command. All SQL clauses and features are supported.

Basic query examples

Here are some examples of basic SELECT queries.

Return the 100 most-recent entries in the table conditions. Order the rows from newest to oldest:

SELECT * FROM conditions ORDER BY time DESC LIMIT 100;

Return the number of entries written to the table conditions in the last 12 hours:

SELECT COUNT(*) FROM conditions
  WHERE time > NOW() - INTERVAL '12 hours';

Advanced query examples

Here are some examples of more advanced SELECT queries.

Get information about the weather conditions at each location, for each 15-minute period within the last 3 hours. Calculate the number of measurements taken, the maximum temperature, and the maximum humidity. Order the results by maximum temperature.

This examples uses the time_bucket function to aggregate data into 15-minute buckets:

SELECT time_bucket('15 minutes', time) AS fifteen_min,
    location,
    COUNT(*),
    MAX(temperature) AS max_temp,
    MAX(humidity) AS max_hum
  FROM conditions
  WHERE time > NOW() - INTERVAL '3 hours'
  GROUP BY fifteen_min, location
  ORDER BY fifteen_min DESC, max_temp DESC;

Count the number of distinct locations with air conditioning that have reported data in the last day:

SELECT COUNT(DISTINCT location) FROM conditions
  JOIN locations
    ON conditions.location = locations.location
  WHERE locations.air_conditioning = True
    AND time > NOW() - INTERVAL '1 day';

===== PAGE: https://docs.tigerdata.com/use-timescale/query-data/advanced-analytic-queries/ =====

Perform advanced analytic queries

You can use TimescaleDB for a variety of analytical queries. Some of these queries are native Postgres, and some are additional functions provided by TimescaleDB and TimescaleDB Toolkit. This section contains the most common and useful analytic queries.

Calculate the median and percentile

Use percentile_cont to calculate percentiles. You can also use this function to look for the fiftieth percentile, or median. For example, to find the median temperature:

SELECT percentile_cont(0.5)
  WITHIN GROUP (ORDER BY temperature)
  FROM conditions;

You can also use TimescaleDB Toolkit to find the approximate percentile.

Calculate the cumulative sum

Use sum(sum(column)) OVER(ORDER BY group) to find the cumulative sum. For example:

SELECT location, sum(sum(temperature)) OVER(ORDER BY location)
  FROM conditions
  GROUP BY location;

Calculate the moving average

For a simple moving average, use the OVER windowing function over a number of rows, then compute an aggregation function over those rows. For example, to find the smoothed temperature of a device by averaging the ten most recent readings:

SELECT time, AVG(temperature) OVER(ORDER BY time
      ROWS BETWEEN 9 PRECEDING AND CURRENT ROW)
    AS smooth_temp
  FROM conditions
  WHERE location = 'garage' and time > NOW() - INTERVAL '1 day'
  ORDER BY time DESC;

Calculate the increase in a value

To calculate the increase in a value, you need to account for counter resets. Counter resets can occur if a host reboots or container restarts. This example finds the number of bytes sent, and takes counter resets into account:

SELECT
  time,
  (
    CASE
      WHEN bytes_sent >= lag(bytes_sent) OVER w
        THEN bytes_sent - lag(bytes_sent) OVER w
      WHEN lag(bytes_sent) OVER w IS NULL THEN NULL
      ELSE bytes_sent
    END
  ) AS "bytes"
  FROM net
  WHERE interface = 'eth0' AND time > NOW() - INTERVAL '1 day'
  WINDOW w AS (ORDER BY time)
  ORDER BY time

Calculate the rate of change

Like increase, rate applies to a situation with monotonically increasing counters. If your sample interval is variable or you use different sampling intervals between different series, it is helpful to normalize the values to a common time interval to make the calculated values comparable. This example finds bytes per second sent, and takes counter resets into account:

SELECT
  time,
  (
    CASE
      WHEN bytes_sent >= lag(bytes_sent) OVER w
        THEN bytes_sent - lag(bytes_sent) OVER w
      WHEN lag(bytes_sent) OVER w IS NULL THEN NULL
      ELSE bytes_sent
    END
  ) / extract(epoch from time - lag(time) OVER w) AS "bytes_per_second"
  FROM net
  WHERE interface = 'eth0' AND time > NOW() - INTERVAL '1 day'
  WINDOW w AS (ORDER BY time)
  ORDER BY time

Calculate the delta

In many monitoring and IoT use cases, devices or sensors report metrics that do not change frequently, and any changes are considered anomalies. When you query for these changes in values over time, you usually do not want to transmit all the values, but only the values where changes were observed. This helps to minimize the amount of data sent. You can use a combination of window functions and subselects to achieve this. This example uses diffs to filter rows where values have not changed and only transmits rows where values have changed:

SELECT time, value FROM (
  SELECT time,
    value,
    value - LAG(value) OVER (ORDER BY time) AS diff
  FROM hypertable) ht
WHERE diff IS NULL OR diff != 0;

Calculate the change in a metric within a group

To group your data by some field, and calculate the change in a metric within each group, use LAG ... OVER (PARTITION BY ...). For example, given some weather data, calculate the change in temperature for each city:

SELECT ts, city_name, temp_delta
FROM (
  SELECT
    ts,
    city_name,
    avg_temp - LAG(avg_temp) OVER (PARTITION BY city_name ORDER BY ts) as temp_delta
  FROM weather_metrics_daily
) AS temp_change
WHERE temp_delta IS NOT NULL
ORDER BY bucket;

Group data into time buckets

The time_bucket function in TimescaleDB extends the Postgres date_bin function. Time bucket accepts arbitrary time intervals, as well as optional offsets, and returns the bucket start time. For example:

SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
  FROM metrics
  GROUP BY five_min
  ORDER BY five_min DESC LIMIT 12;

Get the first or last value in a column

The first and last functions allow you to get the value of one column as ordered by another. This is commonly used in an aggregation. These examples find the last element of a group:

SELECT location, last(temperature, time)
  FROM conditions
  GROUP BY location;

SELECT time_bucket('5 minutes', time) five_min, location, last(temperature, time)
  FROM conditions
  GROUP BY five_min, location
  ORDER BY five_min DESC LIMIT 12;

Generate a histogram

The histogram function allows you to generate a histogram of your data. This example defines a histogram with five buckets defined over the range 60 to 85. The generated histogram has seven bins; the first is for values below the minimum threshold of 60, the middle five bins are for values in the stated range and the last is for values above 85:

SELECT location, COUNT(*),
    histogram(temperature, 60.0, 85.0, 5)
   FROM conditions
   WHERE time > NOW() - INTERVAL '7 days'
   GROUP BY location;

This query outputs data like this:

 location   | count |        histogram
------------+-------+-------------------------
 office     | 10080 | {0,0,3860,6220,0,0,0}
 basement   | 10080 | {0,6056,4024,0,0,0,0}
 garage     | 10080 | {0,2679,957,2420,2150,1874,0}

Fill gaps in time-series data

You can display records for a selected time range, even if no data exists for part of the range. This is often called gap filling, and usually involves an operation to record a null value for any missing data.

In this example, the trading data that includes a time timestamp, the asset_code being traded, the price of the asset, and the volume of the asset being traded is used.

Create a query for the volume of the asset 'TIMS' being traded every day for the month of September:

SELECT
    time_bucket('1 day', time) AS date,
    sum(volume) AS volume
  FROM trades
  WHERE asset_code = 'TIMS'
    AND time >= '2021-09-01' AND time < '2021-10-01'
  GROUP BY date
  ORDER BY date DESC;

This query outputs data like this:

          date          | volume
------------------------+--------
 2021-09-29 00:00:00+00 |  11315
 2021-09-28 00:00:00+00 |   8216
 2021-09-27 00:00:00+00 |   5591
 2021-09-26 00:00:00+00 |   9182
 2021-09-25 00:00:00+00 |  14359
 2021-09-22 00:00:00+00 |   9855

You can see from the output that no records are included for 09-23, 09-24, or 09-30, because no trade data was recorded for those days. To include time records for each missing day, you can use the time_bucket_gapfill function, which generates a series of time buckets according to a given interval across a time range. In this example, the interval is one day, across the month of September:

SELECT
  time_bucket_gapfill('1 day', time) AS date,
  sum(volume) AS volume
FROM trades
WHERE asset_code = 'TIMS'
  AND time >= '2021-09-01' AND time < '2021-10-01'
GROUP BY date
ORDER BY date DESC;

This query outputs data like this:

          date          | volume
------------------------+--------
 2021-09-30 00:00:00+00 |
 2021-09-29 00:00:00+00 |  11315
 2021-09-28 00:00:00+00 |   8216
 2021-09-27 00:00:00+00 |   5591
 2021-09-26 00:00:00+00 |   9182
 2021-09-25 00:00:00+00 |  14359
 2021-09-24 00:00:00+00 |
 2021-09-23 00:00:00+00 |
 2021-09-22 00:00:00+00 |   9855

You can also use the time_bucket_gapfill function to generate data points that also include timestamps. This can be useful for graphic libraries that require even null values to have a timestamp so that they can accurately draw gaps in a graph. In this example, you generate 1080 data points across the last two weeks, fill in the gaps with null values, and give each null value a timestamp:

SELECT
  time_bucket_gapfill(INTERVAL '2 weeks' / 1080, time, now() - INTERVAL '2 weeks', now()) AS btime,
  sum(volume) AS volume
FROM trades
WHERE asset_code = 'TIMS'
  AND time >= now() - INTERVAL '2 weeks' AND time < now()
GROUP BY btime
ORDER BY btime;

This query outputs data like this:

         btime          | volume
------------------------+----------
 2021-03-09 17:28:00+00 |  1085.25
 2021-03-09 17:46:40+00 |  1020.42
 2021-03-09 18:05:20+00 |
 2021-03-09 18:24:00+00 |  1031.25
 2021-03-09 18:42:40+00 |  1049.09
 2021-03-09 19:01:20+00 |  1083.80
 2021-03-09 19:20:00+00 |  1092.66
 2021-03-09 19:38:40+00 |
 2021-03-09 19:57:20+00 |  1048.42
 2021-03-09 20:16:00+00 |  1063.17
 2021-03-09 20:34:40+00 |  1054.10
 2021-03-09 20:53:20+00 |  1037.78

Fill gaps by carrying the last observation forward

If your data collections only record rows when the actual value changes, your visualizations might still need all data points to properly display your results. In this situation, you can carry forward the last observed value to fill the gap. For example:

SELECT
  time_bucket_gapfill(INTERVAL '5 min', time, now() - INTERVAL '2 weeks', now()) as 5min,
  meter_id,
  locf(avg(data_value)) AS data_value
FROM my_hypertable
WHERE
  time > now() - INTERVAL '2 weeks'
  AND meter_id IN (1,2,3,4)
GROUP BY 5min, meter_id

Find the last point for each unique item

You can find the last point for each unique item in your database. For example, the last recorded measurement from each IoT device, the last location of each item in asset tracking, or the last price of a security. The standard approach to minimize the amount of data to be searched for the last point is to use a time predicate to tightly bound the amount of time, or the number of chunks, to traverse. This method does not work unless all items have at least one record within the time range. A more robust method is to use a last point query to determine the last record for each unique item.

In this example, useful for asset tracking or fleet management, you create a metadata table for each vehicle being tracked, and a second time-series table containing the vehicle's location at a given time:

CREATE TABLE vehicles (
  vehicle_id INTEGER PRIMARY KEY,
  vin_number CHAR(17),
  last_checkup TIMESTAMP
);

CREATE TABLE location (
  time TIMESTAMP NOT NULL,
  vehicle_id INTEGER REFERENCES vehicles (vehicle_id),
  latitude FLOAT,
  longitude FLOAT
) WITH (
  tsdb.hypertable,
  tsdb.partition_column='time'
);

If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

You can use the first table, which gives a distinct set of vehicles, to perform a LATERAL JOIN against the location table:

SELECT data.* FROM vehicles v
  INNER JOIN LATERAL (
    SELECT * FROM location l
      WHERE l.vehicle_id = v.vehicle_id
      ORDER BY time DESC LIMIT 1
  ) AS data
ON true
ORDER BY v.vehicle_id, data.time DESC;

            time            | vehicle_id | latitude  |  longitude
----------------------------+------------+-----------+-------------
 2017-12-19 20:58:20.071784 |         72 | 40.753690 |  -73.980340
 2017-12-20 11:19:30.837041 |        156 | 40.729265 |  -73.993611
 2017-12-15 18:54:01.185027 |        231 | 40.350437 |  -74.651954

This approach requires keeping a separate table of distinct item identifiers or names. You can do this by using a foreign key from the hypertable to the metadata table, as shown in the REFERENCES definition in the example.

The metadata table can be populated through business logic, for example when a vehicle is first registered with the system. Alternatively, you can dynamically populate it using a trigger when inserts or updates are performed against the hypertable. For example:

CREATE OR REPLACE FUNCTION create_vehicle_trigger_fn()
  RETURNS TRIGGER LANGUAGE PLPGSQL AS
body$
BEGIN
  INSERT INTO vehicles VALUES(NEW.vehicle_id, NULL, NULL) ON CONFLICT DO NOTHING;
  RETURN NEW;
END
body$;

CREATE TRIGGER create_vehicle_trigger
  BEFORE INSERT OR UPDATE ON location
  FOR EACH ROW EXECUTE PROCEDURE create_vehicle_trigger_fn();

You could also implement this functionality without a separate metadata table by performing a loose index scan over the location hypertable, although this requires more compute resources. Alternatively, you speed up your SELECT DISTINCT queries by structuring them so that TimescaleDB can use its SkipScan feature.

===== PAGE: https://docs.tigerdata.com/use-timescale/query-data/skipscan/ =====

Get faster DISTINCT queries with SkipScan

Tiger Data SkipScan dramatically speeds up DISTINCT queries. It jumps directly to the first row of each distinct value in an index instead of scanning all rows. First introduced for the rowstore hypertables and relational tables, SkipScan now extends to columnstore hypertables, distinct aggregates like COUNT(DISTINCT), and even multiple columns.

Since TimescaleDB v2.2.0

Speed up DISTINCT queries

You use DISTINCT queries to get only the unique values in your data. For example, the IDs of customers who placed orders, the countries where your users are located, or the devices reporting into an IoT system. You might also have graphs and alarms that repeatedly query the most recent values for every device or service.

As your tables get larger, DISTINCT queries tend to get slower. Even when your index matches the exact order and columns for these kinds of queries, Postgres (without SkipScan) has to scan the entire index and then run deduplication. As the table grows, this operation keeps getting slower.

SkipScan is an optimization for DISTINCT and DISTINCT ON queries, including multi-column DISTINCT. SkipScan allows queries to incrementally jump from one ordered value to the next, without reading the rows in between. Conceptually, SkipScan is a regular IndexScan that skips across an index looking for the next value that is greater than the current value.

When you issue a query that uses SkipScan, the EXPLAIN output includes a new Custom Scan (SkipScan) operator, or node, that can quickly return distinct items from a properly ordered index. As it locates one item, the SkipScan node quickly restarts the search for the next item. This is a much more efficient way of finding distinct items in an ordered index.

SkipScan cost is based on the ratio of distinct tuples to total tuples. If the number of distinct tuples is close to the total number of tuples, SkipScan is unlikely to be used due to its higher estimated cost.

Multi-column SkipScan is supported for queries that do not produce NULL distinct values. For example:

CREATE INDEX ON metrics(region, device, metric_type);
-- All distinct columns have filters which don't allow NULLs: can use SkipScan
SELECT DISTINCT ON (region, device, metric_type) *
FROM   metrics
WHERE region IN ('UK','EU','JP') AND device > 1 AND metric_type IS NOT NULL
ORDER  BY region, device, metric_type, time DESC;
-- Distinct columns are declared NOT NULL: can use SkipScan with index on (region, device)
CREATE TABLE metrics(region TEXT NOT NULL, device INT NOT NULL, ...);
SELECT DISTINCT ON (region, device) *
FROM   metrics
ORDER  BY region, device, time DESC;

For benchmarking information on how SkipScan compares to regular DISTINCT queries, see the SkipScan blog post.

Use SkipScan queries

Design your layout:

• Rowstore: create an index starting with the DISTINCT columns, followed by your time sort. If the DISTINCT columns are not the first in your index, ensure any leading columns are used as constraints in your query. This means that if you are asking a question such as "retrieve a list of unique IDs in order" and "retrieve the last reading of each ID," you need at least one index like this:

  CREATE INDEX "cpu_customer_tags_id_time_idx" \
  ON readings (customer_id, tags_id, time DESC)
  

• Columnstore: set timescaledb.compress_segmentby to the distinct columns and compress_orderby to match your query’s sort. Compress your historical chunks.

With your index set up correctly, you should start to see immediate benefit for DISTINCT queries. When SkipScan is chosen for your query, the EXPLAIN ANALYZE output shows one or more Custom Scan (SkipScan) nodes, like this:

->  Unique
  ->  Merge Append
    Sort Key: _hyper_8_79_chunk.tags_id, _hyper_8_79_chunk."time" DESC
     ->  Custom Scan (SkipScan) on _hyper_8_79_chunk
      ->  Index Only Scan using _hyper_8_79_chunk_cpu_tags_id_time_idx on _hyper_8_79_chunk
          Index Cond: (tags_id > NULL::integer)
     ->  Custom Scan (SkipScan) on _hyper_8_80_chunk
      ->  Index Only Scan using _hyper_8_80_chunk_cpu_tags_id_time_idx on _hyper_8_80_chunk
         Index Cond: (tags_id > NULL::integer)

===== PAGE: https://docs.tigerdata.com/use-timescale/configuration/about-configuration/ =====

About configuration in Tiger Cloud

By default, Tiger Cloud uses the default Postgres server configuration settings. Most configuration values for a Tiger Cloud service are initially set in accordance with best practices given the compute and storage settings of the service. Any time you increase or decrease the compute for a service, the most essential values are set to reflect the size of the new service.

There are times, however, when your specific workload could require tuning some of the many available Tiger Cloud-specific and Postgres parameters. By providing the ability to tune various runtime settings, Tiger Cloud provides the balance and flexibility you need when running your workloads in a hosted environment. You can use service settings and service operations to customize Tiger Cloud configurations.

===== PAGE: https://docs.tigerdata.com/use-timescale/configuration/customize-configuration/ =====

Configure database parameters

Tiger Cloud allows you to customize many Tiger Cloud-specific and Postgres configuration options for each service individually. Most configuration values for a service are initially set in accordance with best practices given the compute and storage settings of the service. Any time you increase or decrease the compute for a service, the most essential values are set to reflect the size of the new service.

You can modify most parameters without restarting the service. However, some changes do require a restart, resulting in some brief downtime that is usually about 30 seconds. An example of a change that needs a restart is modifying the compute resources of a running service.

View service operation details

To modify configuration parameters, first select the service that you want to modify. This displays the service details, with these tabs across the top: Overview, Actions, Explorer, Monitoring, Connections, SQL Editor, Operations, and AI. Select Operations, then Database parameters.

Modify basic parameters

Under the Common parameters tab, you can modify a limited set of the parameters that are most often modified in a Tiger Cloud or Postgres instance. To modify a configured value, hover over the value and click the revealed pencil icon. This reveals an editable field to apply your change. Clicking anywhere outside of that field saves the value to be applied.

Apply configuration changes

When you have modified the configuration parameters that you would like to change, click Apply changes. For some changes, such as timescaledb.max_background_workers, the service needs to be restarted. In this case, the button reads Apply changes and restart.

A confirmation dialog is displayed which indicates whether a restart is required. Click Confirm to apply the changes, and restart if necessary.

===== PAGE: https://docs.tigerdata.com/use-timescale/configuration/advanced-parameters/ =====

Advanced parameters

It is possible to configure a wide variety of Tiger Cloud service database parameters by navigating to the Advanced parameters tab under the Database configuration heading. The advanced parameters are displayed in a scrollable and searchable list.

As with the basic database configuration parameters, any changes are highlighted and the Apply changes, or Apply changes and restart, button is available, prompting you to confirm changes before the service is modified.

Multiple databases

To create more than one database, you need to create a new service for each database. Tiger Cloud does not support multiple databases within the same service. Having a separate service for each database affords each database its own isolated resources.

You can also use schemas to organize tables into logical groups. A single database can contain multiple schemas, which in turn contain tables. The main difference between isolating with databases versus schemas is that a user can access objects in any of the schemas in the database they are connected to, so long as they have the corresponding privileges. Schemas can help isolate smaller use cases that do not warrant their own service.

Please refer to the Grand Unified Configuration (GUC) parameters for a complete list.

Policies

timescaledb.max_background_workers (int)

Max background worker processes allocated to TimescaleDB. Set to at least 1 + the number of databases loaded with the TimescaleDB extension in a Postgres instance. Default value is 16.

Tiger Cloud service tuning

timescaledb.disable_load (bool)

Disable the loading of the actual extension

===== PAGE: https://docs.tigerdata.com/use-timescale/ha-replicas/read-scaling/ =====

Read scaling

When read-intensive workloads compete with high ingest rates, your primary data instance can become a bottleneck. Spiky query traffic, analytical dashboards, and business intelligence tools risk slowing down ingest performance and disrupting critical write operations.

With read replica sets in Tiger Cloud, you can scale reads horizontally and keep your applications responsive. By offloading queries to replicas, your service maintains high ingest throughput while serving large or unpredictable read traffic with ease. This approach not only protects write performance but also gives you confidence that your read-heavy apps and BI workloads will run smoothly—even under pressure.

This page shows you how to create and manage read replica sets in Tiger Cloud Console.

What is read replication?

A read replica is a read-only copy of your primary database instance. Queries on read replicas have minimal impact on the performance of the primary instance. This enables you to interact with up-to-date production data for analysis, or to scale out reads beyond the limits of your primary instance. Read replicas can be short-lived and deleted when a session of data analysis is complete, or long-running to power an application or a business intelligence tool.

A read replica set in Tiger Cloud is a group of one or more read replica nodes that are accessed through the same endpoint. You query each set as a single replica. Tiger Cloud balances the load between the nodes in the set for you.

You can create as many read replica sets as you need. For security and resource isolation, each read replica set has unique connection details.

You use read replica sets for horizontal read scaling. To limit data loss for your Tiger Cloud services, use high-availability replicas.

Prerequisites

To follow this procedure:

• Create a target Tiger Cloud service.

• Create a read-only user on the primary data instance.

  A user with read-only permissions cannot make changes in the primary database. This user is propagated to the read replica set when you create it.

Create a read replica set

To create a secure read replica set for your read-intensive apps:

1. In Tiger Cloud Console, select your target service

2. Click Operations > Read scaling > Add a read replica set

3. Configure your replica set

   Configure the number of nodes, compute size, connection pooling, and the name for your replica, then click Create read replica set.

   

4. Save the connection information

   The username and password of a read replica set are the same as the primary service. They cannot be changed independently.

   The connection information for each read replica set is unique. You can add or remove nodes from an existing set and the connection information of that set will remain the same. To find the connection information for an existing read replica set:

   1. Select the primary service in Tiger Cloud Console.

   2. Click Operations > Read scaling.

   3. Click the 🔗 icon next to the replica set in the list.

Edit a read replica set

You can edit an existing read replica set to better handle your reads. This includes changing the number of nodes, compute size, storage, and IOPS, as well as configuring VPC and other features.

To change the compute and storage configuration of your read replica set:

1. In Tiger Cloud Console, expand and click the read replica set under your primary service

   

2. Click Operations > Compute and storage

   

3. Change the replica configuration and click Apply

Manage data lag for your read replica sets

Read replica sets use asynchronous replication. This can cause a slight lag in data to the primary database instance. The lag is measured in bytes, against the current state of the primary instance. To check the status and lag for your read replica set:

1. In Tiger Cloud Console, select your primary service

2. Click Operations > Read scaling

   You see a list of configured read replica sets for this service, including their status and lag:

   

3. Configure the allowable lag

   1. Select the replica set in the list.

   2. Click Operations > Database parameters.

   3. Adjust max_standby_streaming_delay and max_standby_archive_delay.

      This is not recommended for cases where changes must be immediately represented, for example, for user credentials.

Delete a read replica set

To delete a replica set:

1. In Tiger Cloud Console, select your primary service

2. Click Operations > Read scaling

3. Click the trash icon next to a replica set

   Confirm the deletion when prompted.

===== PAGE: https://docs.tigerdata.com/use-timescale/ha-replicas/high-availability/ =====

Manage high availability

For Tiger Cloud services where every second of uptime matters, Tiger Cloud delivers High Availability (HA) replicas. These replicas safeguard your data and keep your service running smoothly, even in the face of unexpected failures. By minimizing downtime and protecting against data loss, HA replicas ensure business continuity and give you the confidence to operate without interruption, including during routine maintenance.

This page shows you how to choose the best high availability option for your service.

What is HA replication?

HA replicas are exact, up-to-date copies of your database hosted in multiple AWS availability zones (AZ) within the same region as your primary node. They automatically take over operations if the original primary data node becomes unavailable. The primary node streams its write-ahead log (WAL) to the replicas to minimize the chances of data loss during failover.

HA replicas can be synchronous and asynchronous.

• Synchronous: the primary commits its next write once the replica confirms that the previous write is complete. There is no lag between the primary and the replica. They are in the same state at all times. This is preferable if you need the highest level of data integrity. However, this affects the primary ingestion time.

• Asynchronous: the primary commits its next write without the confirmation of the previous write completion. The asynchronous HA replicas often have a lag, in both time and data, compared to the primary. This is preferable if you need the shortest primary ingest time.

HA replicas have separate unique addresses that you can use to serve read-only requests in parallel to your primary data node. When your primary data node fails, Tiger Cloud automatically fails over to an HA replica within 30 seconds. During failover, the read-only address is unavailable while Tiger Cloud automatically creates a new HA replica. The time to make this replica depends on several factors, including the size of your data.

Operations such as upgrading your service to a new major or minor version may necessitate a service restart. Restarts are run during the maintenance window. To avoid any downtime, each data node is updated in turn. That is, while the primary data node is updated, a replica is promoted to primary. After the primary is updated and online, the same maintenance is performed on the HA replicas.

To ensure that all services have minimum downtime and data loss in the most common failure scenarios and during maintenance, rapid recovery is enabled by default for all services.

Choose an HA strategy

The following HA configurations are available in Tiger Cloud:

• Non-production: no replica, best for developer environments.

• High availability: a single async replica in a different AWS availability zone from your primary. Provides high availability with cost efficiency. Best for production apps.

• Highest availability: two replicas in different AWS availability zones from your primary. Available replication modes are:

  • High performance - two async replicas. Provides the highest level of availability with two AZs and the ability to query the HA system. Best for apps where service availability is most critical.
  • High data integrity - one sync replica and one async replica. The sync replica is identical to the primary at all times. Best for apps that can tolerate no data loss.

The following table summarizes the differences between these HA configurations:

	High availability (1 async)	High performance (2 async)	High data integrity (1 sync + 1 async)
Write flow	The primary streams its WAL to the async replica, which may have a slight lag compared to the primary, providing 99.9% uptime SLA.	The primary streams its writes to both async replicas, providing 99.9+% uptime SLA.	The primary streams its writes to the sync and async replicas. The async replica is never ahead of the sync one.
Additional read replica	Recommended. Reads from the HA replica may cause availability and lag issues.	Not needed. You can still read from the HA replica even if one of them is down. Configure an additional read replica only if your read use case is significantly different from your write use case.	Highly recommended. If you run heavy queries on a sync replica, it may fall behind the primary. Specifically, if it takes too long for the replica to confirm a transaction, the next transaction is canceled.
Choosing the replica to read from manually	Not applicable.	Not available. Queries are load-balanced against all available HA replicas.	Not available. Queries are load-balanced against all available HA replicas.
Sync replication	Only async replicas are supported in this configuration.	Only async replicas are supported in this configuration.	Supported.
Failover flow	If the primary fails, the replica becomes the primary while a new node is created, with only seconds of downtime. If the replica fails, a new async replica is created without impacting the primary. If you read from the async HA replica, those reads fail until the new replica is available.	If the primary fails, one of the replicas becomes the primary while a new node is created, with the other one still available for reads. If the replica fails, a new async replica is created in another AZ, without impacting the primary. The newly created replica is behind the primary and the original replica while it catches up.	If the primary fails, the sync replica becomes the primary while a new node is created, with the async one still available for reads. If the async replica fails, a new async replica is created. Heavy reads on the sync replica may delay the ingest time of the primary while a new async replica is created. Data integrity remains high but primary ingest performance may degrade. If the sync replica fails, the async replica becomes the sync one, and a new async replica is created. The primary may experience some ingest performance degradation during this time.
Cost composition	Primary + async (2x)	Primary + 2 async (3x)	Primary + 1 async + 1 sync (3x)
Tier	Performance, Scale, and Enterprise	Scale and Enterprise	Scale and Enterprise

The High and Highest HA strategies are available with the Scale and the Enterprise pricing plans.

To enable HA for a service:

1. In Tiger Cloud Console, select the service to enable replication for.

2. Click Operations, then select High availability.

3. Choose your replication strategy, then click Change configuration.

   

4. In Change high availability configuration, click Change config.

To change your HA replica strategy, click Change configuration, choose a strategy and click Change configuration. To download the connection information for the HA replica, either click the link next to the replica Active configuration, or find the information in the Overview tab for this service.

Test failover for your HA replicas

To test the failover mechanism, you can trigger a switchover. A switchover is a safe operation that attempts a failover, and throws an error if the replica or primary is not in a state to safely switch.

1. Connect to your primary node as tsdbadmin or another user that is part of the tsdbowner group.

   You can also connect to the HA replica and check its node using this procedure.

2. At the psql prompt, connect to the postgres database:

   \c postgres
   

   You should see postgres=> prompt.

3. Check if your node is currently in recovery:

   select pg_is_in_recovery();
   

4. Check which node is currently your primary:

   select * from pg_stat_replication;
   

   Note the application_name. This is your service ID followed by the node. The important part is the -an-0 or -an-1.

5. Schedule a switchover:

   CALL tscloud.cluster_switchover();
   

   By default, the switchover occurs in 30 secs. You can change the time by passing an interval, like this:

   CALL tscloud.cluster_switchover('15 seconds'::INTERVAL);
   

6. Wait for the switchover to occur, then check which node is your primary:

   SELECT * FROM pg_stat_replication;
   

   You should see a notice that your connection has been reset, like this:

   FATAL:  terminating connection due to administrator command
   SSL connection has been closed unexpectedly
   The connection to the server was lost. Attempting reset: Succeeded.
   

7. Check the application_name. If your primary was -an-1 before, it should now be -an-0. If it was -an-0, it should now be -an-1.

===== PAGE: https://docs.tigerdata.com/use-timescale/data-tiering/tiered-data-replicas-forks/ =====

Replicas and forks with tiered data

There is one more thing that makes Tiered Storage even more amazing: when you keep data in the low-cost object storage tier, you pay for this data only once, regardless of whether you have a high-availability replica or read replicas running in your service. We call this the savings multiplication effect of Tiered Storage.

The same applies to forks, which you can use, for example, for running tests or creating dev environments. When creating one (or more) forks, you won't be billed for data shared with the primary in the low-cost storage.

If you decide to tier more data that's not in the primary, you will pay to store it in the low-cost tier, but you will still see substantial savings by moving that data from the high-performance tier of the fork to the cheaper object storage tier.

How this works behind the scenes

Once you tier data to the low-cost object storage tier, we keep a reference to that data on your Database's catalog.

Creating a replica or forking a primary server only copies the references and the metadata we keep on the catalog for all tiered data.

On the billing side, we only count and bill once for the data tiered, not for each reference there may exist towards that data.

What happens when a chunk is dropped or untiered on a fork

Dropping or untiering a chunk from a fork does not delete it from any other servers that reference the same chunk.

You can have one, multiple or 0 servers referencing the same chunk of data:

• That means that deleting data from a fork does not affect the other servers (including the primary); it just removes the reference to that data, which is for all intends and purposes equal to deleting that data from the point of view of that fork
• The primary and other servers are unaffected, as they still have their references and the metadata on their catalogs intact
• We never delete anything on the object storage tier if at least one server references it: The data is only permanently deleted (or hard deleted as we internally call this operation) once the references drop to 0

As described above, tiered chunks are only counted once for billing purposes, so dropping or untiering a chunk that is shared with other servers from a fork will not affect billing as it was never counted for billing purposes.

Droping or untiering a chunk that was only tiered on that fork works as expected and is covered in more detail in the following section.

What happens when a chunk is modified on a fork

As a reminder, tiered data is immutable - there is no such thing as updating the data.

You can untier or drop a chunk, in which case what is described in the previous section covers what happens.

And you can tier new data, at which point a fork deviates from the primary in a similar way as all forks do.

New data tiered are not shared with parent or sibling servers, this is new data tiered for that server and we count them as a new object for the purposes of billing.

If you decide to tier more data that's not in the primary, you will pay to store it in the low-cost tier, but you will still see substantial savings by moving that data from the high-performance tier of the fork to the cheaper object storage tier.

Similar to other types of storage tiers, this type of deviation can not happen for replicas as they have to be identical with the primary server, that's why we don't mention replicas when discussing about droping chunks or tiering additional data.

What happens with backups and PITR

As discussed above, we never delete anything on the object storage tier if at least one server references it. The data is only permanently deleted (or hard deleted as we internally call this operation) once the references drop to 0.

In addition to that, we delay hard deleting the data by 14 days, so that in case of a restore or PITR, all tiered data will be available. In the case of such a restore, new references are added to the deleted tiered chunks, so they are not any more candidates for a hard deletion.

Once 14 days pass after soft deleting the data,that is the number of references to the tiered data drop to 0, we hard delete the tiered data.

===== PAGE: https://docs.tigerdata.com/use-timescale/data-tiering/enabling-data-tiering/ =====

Manage storage and tiering

The tiered storage architecture in Tiger Cloud includes a high-performance storage tier and a low-cost object storage tier:

• You use high-performance storage to store and query frequently accessed data.

• You use low-cost object storage to cut costs by migrating rarely used data from the high-performance storage. After you enable tiered storage, you then either create automated tiering policies or manually tier and untier data.

You can query the data on the object storage tier, but you cannot modify it. Make sure that you are not tiering data that needs to be actively modified.

For low-cost storage, Tiger Data charges only for the size of your data in S3 in the Apache Parquet format, regardless of whether it was compressed in Tiger Cloud before tiering. There are no additional expenses, such as data transfer or compute.

High-performance storage tier

By default, Tiger Cloud stores your service data in the standard high-performance storage. This storage tier comes in the standard and enhanced types. Enhanced storage is available under the Enterprise pricing plan only.

Standard high-performance storage

This storage type gives you up to 16 TB of storage and is available under all pricing plans. You change the IOPS value to better suit your needs in Tiger Cloud Console:

1. In Tiger Cloud Console, select your service, then click Operations > Compute and storage

   By default, the type of high-performance storage is set to Standard.

2. Select the IOPS value in the I/O boost dropdown

   • Under the Performance pricing plan, IOPS is set to 3,000 - 5,000 autoscale and cannot be changed.
   • Under the Scale and Enterprise pricing plans, IOPS is set to 5,000 - 8,000 autoscale and can be upgraded to 16,000 IOPS.

   

3. Click Apply

Enhanced high-performance storage

<Availability products={['cloud']} price_plans={['enterprise']} />

This storage type gives you up to 64 TB and 32,000 IOPS, and is available under the Enterprise pricing plan. To get enhanced storage:

1. In Tiger Cloud Console, select your service, then click Operations > Compute and storage

2. Select Enhanced in the Storage type dropdown

   

   The enhanced storage is currently not available in sa-east-1.

3. Select the IOPS value in the I/O boost dropdown

   Select between 8,000, 16,000, 24,000, and 32,0000 IOPS. The value that you can apply depends on the number of CPUs in your service. Tiger Cloud Console notifies you if your selected IOPS requires increasing the number of CPUs. To increase IOPS to 64,000, click Contact us and we will be in touch to confirm the details.

   

4. Click Apply

You change from enhanced storage to standard in the same way. If you are using over 16 TB of enhanced storage, changing back to standard is not available until you shrink your data to be under 16 TB. You can make changes to the storage type and I/O boost settings without any downtime. Wait at least 6 hours to attempt another change.

Low-cost object storage tier

<Availability products={['cloud']} price_plans={['enterprise', 'scale']} />

You enable the low-cost object storage tier in Tiger Cloud Console and then tier the data with policies or manually.

Enable tiered storage

You enable tiered storage from the Overview tab in Tiger Cloud Console.

1. In Tiger Cloud Console, select the service to modify

2. In Explorer, click Storage configuration > Tiering storage, then click Enable tiered storage

   

   Once enabled, you can proceed to tier data manually or set up tiering policies. When tiered storage is enabled, you see the amount of data in the tiered object storage.

Automate tiering with policies

A tiering policy automatically moves any chunks that only contain data older than the move_after threshold to the object storage tier. This works similarly to a data retention policy, but chunks are moved rather than deleted.

A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. Chunks are considered tiered once they appear in the timescaledb_osm.tiered_chunks view.

You can add tiering policies to hypertables, including continuous aggregates. To manage tiering policies, connect to your service and run the queries below in the data mode, the SQL editor, or using psql.

Add a tiering policy

To add a tiering policy, call add_tiering_policy:

SELECT add_tiering_policy(hypertable REGCLASS, move_after INTERVAL, if_not_exists BOOL = false);

For example, to tier chunks that are more than three days old in the example hypertable:

SELECT add_tiering_policy('example', INTERVAL '3 days');

By default, a tiering policy runs hourly on your database. To change this interval, call alter_job.

Remove a tiering policy

To remove an existing tiering policy, call remove_tiering_policy:

SELECT remove_tiering_policy(hypertable REGCLASS, if_exists BOOL = false);

For example, to remove the tiering policy from the example hypertable:

SELECT remove_tiering_policy('example');

If you remove a tiering policy, the remaining scheduled chunks are not tiered. However, chunks in tiered storage are not untiered. You untier chunks manually to local storage.

Manually tier and untier chunks

If tiering policies do not meet your current needs, you can tier and untier chunks manually. To do so, connect to your service and run the queries below in the data mode, the SQL editor, or using psql.

Tier chunks

Tiering a chunk is an asynchronous process that schedules the chunk to be tiered. In the following example, you tier chunks older than three days in the example hypertable. You then list the tiered chunks.

1. Select all chunks in example that are older than three days:

   SELECT show_chunks('example', older_than => INTERVAL '3 days');
   

   This returns a list of chunks. Take a note of the chunk names:

   _timescaledb_internal._hyper_1_1_chunk
   _timescaledb_internal._hyper_1_2_chunk
   

2. Call tier_chunk to manually tier each chunk:

   SELECT tier_chunk('_timescaledb_internal._hyper_1_1_chunk');
   

3. Repeat for all chunks you want to tier.

   Tiering a chunk schedules it for migration to the object storage tier, but the migration won't happen immediately. Chunks are tiered one at a time in order to minimize database resource consumption. A chunk is marked as migrated and deleted from the standard storage only after it has been durably stored in the object storage tier. You can continue to query a chunk during migration.

4. To see which chunks are tiered into the object storage tier, use the tiered_chunks informational view:

   SELECT * FROM timescaledb_osm.tiered_chunks;
   

To see which chunks are scheduled for tiering either by policy or by a manual call, but have not yet been tiered, use this view:

SELECT * FROM timescaledb_osm.chunks_queued_for_tiering ;

Untier chunks

To update data in a tiered chunk, move it back to the standard high-performance storage tier in Tiger Cloud. Untiering chunks is a synchronous process. Chunks are renamed when the data is untiered.

To untier a chunk, call the untier_chunk stored procedure.

1. Check which chunks are currently tiered:

   SELECT * FROM timescaledb_osm.tiered_chunks ;
   

   Sample output:

    hypertable_schema | hypertable_name |    chunk_name    |      range_start       |       range_end
   -------------------+-----------------+------------------+------------------------+------------------------
   public            | sample          | _hyper_1_1_chunk | 2023-02-16 00:00:00+00 | 2023-02-23 00:00:00+00
   (1 row)
   

2. Call untier_chunk:

   CALL untier_chunk('_hyper_1_1_chunk');
   

3. See the details of the chunk with timescaledb_information.chunks:

   SELECT * FROM timescaledb_information.chunks;
   

   Sample output:

   -[ RECORD 1 ]----------+-------------------------
   hypertable_schema      | public
   hypertable_name        | sample
   chunk_schema           | _timescaledb_internal
   chunk_name             | _hyper_1_4_chunk
   primary_dimension      | ts
   primary_dimension_type | timestamp with time zone
   range_start            | 2023-02-16 00:00:00+00
   range_end              | 2020-03-23 00:00:00+00
   range_start_integer    |
   range_end_integer      |
   is_compressed          | f
   chunk_tablespace       |
   data_nodes             |
   

Disable tiering

If you no longer want to use tiered storage for a particular hypertable, drop the associated metadata by calling disable_tiering.

1. To drop all tiering policies associated with a table, call remove_tiering_policy.

2. Make sure that there is no tiered data associated with this hypertable:

   1. List the tiered chunks associated with this hypertable:

      select * from timescaledb_osm.tiered_chunks
      

   2. If you have any tiered chunks, either untier this data, or drop these chunks from tiered storage.

3. Use disable_tiering to drop all tiering-related metadata for the hypertable:

   select disable_tiering('my_hypertable_name');
   

4. Verify that tiering has been disabled by listing the hypertables that have tiering enabled:

   select * from timescaledb_osm.tiered_hypertables;
   

===== PAGE: https://docs.tigerdata.com/use-timescale/data-tiering/querying-tiered-data/ =====

Querying Tiered Data

Once rarely used data is tiered and migrated to the object storage tier, it can still be queried with standard SQL by enabling the timescaledb.enable_tiered_reads GUC. By default, the GUC is set to false, so that queries do not touch tiered data.

The timescaledb.enable_tiered_reads GUC, or Grand Unified Configuration variable, is a setting that controls if tiered data is queried. The configuration variable can be set at different levels, including globally for the entire database server, for individual databases, and for individual sessions.

With tiered reads enabled, you can query your data normally even when it's distributed across different storage tiers. Your hypertable is spread across the tiers, so queries and JOINs work and fetch the same data as usual.

By default, tiered data is not accessed by queries. Querying tiered data may slow down query performance as the data is not stored locally on the high-performance storage tier. See Performance considerations.

Enable querying tiered data for a single query

1. Enable timescaledb.enable_tiered_reads before querying the hypertable with tiered data and reset it after it is complete:

set timescaledb.enable_tiered_reads = true; SELECT count(*) FROM example; set timescaledb.enable_tiered_reads = false;

This queries data from all chunks including tiered chunks and non tiered chunks:

 ```sql
 ||count|
 |---|
 |1000|
 ```

Enable querying tiered data for a single session

All future queries within a session can be enabled to use the object storage tier by enabling timescaledb.enable_tiered_reads within a session.

1. Enable timescaledb.enable_tiered_reads for an entire session:

   set timescaledb.enable_tiered_reads = true;
   

   All future queries in that session are configured to read from tiered data and locally stored data.

Enable querying tiered data in all future sessions

You can also enable queries to read from tiered data always by following these steps:

1. Enable timescaledb.enable_tiered_reads for all future sessions:

   alter database tsdb set timescaledb.enable_tiered_reads = true;
   

   In all future created sessions, timescaledb.enable_tiered_reads initializes with enabled.

Query data in the object storage tier

This section illustrates how querying tiered storage works.

Consider a simple database with a standard devices table and a metrics hypertable. After enabling tiered storage, you can see which chunks are tiered to the object storage tier:

    chunk_name    |      range_start       |       range_end
------------------+------------------------+------------------------
 _hyper_2_4_chunk | 2015-12-31 00:00:00+00 | 2016-01-07 00:00:00+00
 _hyper_2_3_chunk | 2017-08-17 00:00:00+00 | 2017-08-24 00:00:00+00
(2 rows)

The following query fetches data only from the object storage tier. This makes sense based on the WHERE clause specified by the query and the chunk ranges listed above for this hypertable.

 EXPLAIN SELECT * FROM metrics where ts < '2017-01-01 00:00+00';
                             QUERY PLAN
---------------------------------------------------------------------
 Foreign Scan on osm_chunk_2  (cost=0.00..0.00 rows=2 width=20)
   Filter: (ts < '2017-01-01 00:00:00'::timestamp without time zone)
   Match tiered objects: 1
   Row Groups:
     _timescaledb_internal._hyper_2_4_chunk: 0
(5 rows)

If your query does not need to touch the object storage tier, it will only process the chunks in the standard storage. The following query refers to newer data that is not yet tiered to the object storage tier. Match tiered objects :0 in the plan indicates that no tiered data matches the query constraint. So data in the object storage is not touched at all.

 EXPLAIN SELECT * FROM metrics where ts > '2022-01-01 00:00+00';
                                                    QUERY PLAN

--------------------------------------------------------------------------------
----------------------------------
 Append  (cost=0.15..25.02 rows=568 width=20)
   ->  Index Scan using _hyper_2_5_chunk_metrics_ts_idx on _hyper_2_5_chunk  (co
st=0.15..22.18 rows=567 width=20)
         Index Cond: (ts > '2022-01-01 00:00:00'::timestamp without time zone)
   ->  Foreign Scan on osm_chunk_2  (cost=0.00..0.00 rows=1 width=20)
         Filter: (ts > '2022-01-01 00:00:00'::timestamp without time zone)
         Match tiered objects: 0
         Row Groups:
(7 rows)

Here is another example with a JOIN that does not touch tiered data:

 EXPLAIN SELECT ts, device_id, description FROM metrics
   JOIN devices ON metrics.device_id = devices.id
   WHERE metrics.ts > '2023-08-01';
                            QUERY PLAN

--------------------------------------------------------------------------------
 Hash Join  (cost=32.12..184.55 rows=3607 width=44)
   Hash Cond: (devices.id = _hyper_4_9_chunk.device_id)
   ->  Seq Scan on devices  (cost=0.00..22.70 rows=1270 width=36)
   ->  Hash  (cost=25.02..25.02 rows=568 width=12)
         ->  Append  (cost=0.15..25.02 rows=568 width=12)
               ->  Index Scan using _hyper_4_9_chunk_metrics_ts_idx on _hyper_4_
9_chunk  (cost=0.15..22.18 rows=567 width=12)
                     Index Cond: (ts > '2023-08-01 00:00:00+00'::timestamp with
time zone)
               ->  Foreign Scan on osm_chunk_3  (cost=0.00..0.00 rows=1 width=12
)
                     Filter: (ts > '2023-08-01 00:00:00+00'::timestamp with time
 zone)
                     Match tiered objects: 0
                     Row Groups:
(11 rows)

Performance considerations

Queries over tiered data are expected to be slower than over local data. However, in a limited number of scenarios tiered reads can impact query planning time over local data as well. In order to prevent any unexpected performance degradation for application queries, we keep the GUC timescaledb.enable_tiered_reads set to false.

• Queries without time boundaries specified are expected to perform slower when querying tiered data, both during query planning and during query execution. TimescaleDBs chunk exclusion algorithms cannot be applied for this case.

  SELECT * FROM device_readings WHERE id = 10;
  

• Queries with predicates computed at runtime (such as NOW()) are not always optimized at planning time and as a result might perform slower than statically assigned values when querying against the object storage tier.

  For example, this query is optimized at planning time:

  SELECT * FROM metrics WHERE ts > '2023-01-01' AND ts < '2023-02-01'
  

  The following query does not do chunk pruning at query planning time:

  SELECT * FROM metrics WHERE ts < now() - '10 days':: interval
  

  At the moment, queries against tiered data work best when the query optimizer can apply planning time optimizations.

• Text and non-native types (JSON, JSONB, GIS) filtering is slower when querying tiered data.

===== PAGE: https://docs.tigerdata.com/use-timescale/data-tiering/about-data-tiering/ =====

About Tiger Cloud storage tiers

The tiered storage architecture in Tiger Cloud includes a high-performance storage tier and a low-cost object storage tier. You use the high-performance tier for data that requires quick access, and the object tier for rarely used historical data. Tiering policies move older data asynchronously and periodically from high-performance to low-cost storage, sparing you the need to do it manually. Chunks from a single hypertable, including compressed chunks, can stretch across these two storage tiers.

High-performance storage

High-performance storage is where your data is stored by default, until you enable tiered storage and move older data to the low-cost tier. In the high-performance storage, your data is stored in the block format and optimized for frequent querying. The hypercore row-columnar storage engine available in this tier is designed specifically for real-time analytics. It enables you to compress the data in the high-performance storage by up to 90%, while improving performance. Coupled with other optimizations, Tiger Cloud high-performance storage makes sure your data is always accessible and your queries run at lightning speed.

Tiger Cloud high-performance storage comes in the following types:

• Standard (default): based on AWS EBS gp3 and designed for general workloads. Provides up to 16 TB of storage and 16,000 IOPS.
• Enhanced: based on EBS io2 and designed for high-scale, high-throughput workloads. Provides up to 64 TB of storage and 32,000 IOPS.

See the differences in the underlying AWS storage. You enable enhanced storage as needed in Tiger Cloud Console.

Low-cost storage

<Availability products={['cloud']} price_plans={['enterprise', 'scale']} />

Once you enable tiered storage, you can start moving rarely used data to the object tier. The object tier is based on AWS S3 and stores your data in the Apache Parquet format. Within a Parquet file, a set of rows is grouped together to form a row group. Within a row group, values for a single column across multiple rows are stored together. The original size of the data in your service, compressed or uncompressed, does not correspond directly to its size in S3. A compressed hypertable may even take more space in S3 than it does in Tiger Cloud.

Apache Parquet allows for more efficient scans across longer time periods, and Tiger Cloud uses other metadata and query optimizations to reduce the amount of data that needs to be fetched to satisfy a query, such as:

• Chunk skipping: exclude the chunks that fall outside the query time window.
• Row group skipping: identify the row groups within the Parquet object that satisfy the query.
• Column skipping: fetch only columns that are requested by the query.

The following query is against a tiered dataset and illustrates the optimizations:

EXPLAIN ANALYZE
SELECT count(*) FROM
( SELECT device_uuid,  sensor_id FROM public.device_readings
  WHERE observed_at > '2023-08-28 00:00+00' and observed_at < '2023-08-29 00:00+00'
  GROUP BY device_uuid,  sensor_id ) q;
            QUERY PLAN

-------------------------------------------------------------------------------------------------
 Aggregate  (cost=7277226.78..7277226.79 rows=1 width=8) (actual time=234993.749..234993.750 rows=1 loops=1)
   ->  HashAggregate  (cost=4929031.23..7177226.78 rows=8000000 width=68) (actual time=184256.546..234913.067 rows=1651523 loops=1)
         Group Key: osm_chunk_1.device_uuid, osm_chunk_1.sensor_id
         Planned Partitions: 128  Batches: 129  Memory Usage: 20497kB  Disk Usage: 4429832kB
         ->  Foreign Scan on osm_chunk_1  (cost=0.00..0.00 rows=92509677 width=68) (actual time=345.890..128688.459 rows=92505457 loops=1)
               Filter: ((observed_at > '2023-08-28 00:00:00+00'::timestamp with time zone) AND (observed_at < '2023-08-29 00:00:00+00'::timestamp with t
ime zone))
               Rows Removed by Filter: 4220
               Match tiered objects: 3
               Row Groups:
                 _timescaledb_internal._hyper_1_42_chunk: 0-74
                 _timescaledb_internal._hyper_1_43_chunk: 0-29
                 _timescaledb_internal._hyper_1_44_chunk: 0-71
               S3 requests: 177
               S3 data: 224423195 bytes
 Planning Time: 6.216 ms
 Execution Time: 235372.223 ms
(16 rows)

EXPLAIN illustrates which chunks are being pulled in from the object storage tier:

1. Fetch data from chunks 42, 43, and 44 from the object storage tier.
2. Skip row groups and limit the fetch to a subset of the offsets in the Parquet object that potentially match the query filter. Only fetch the data for device_uuid, sensor_id, and observed_at as the query needs only these 3 columns.

The object storage tier is more than an archiving solution. It is also:

• Cost-effective: store high volumes of data at a lower cost. You pay only for what you store, with no extra cost for queries.
• Scalable: scale past the restrictions of even the enhanced high-performance storage tier.
• Online: your data is always there and can be queried when needed.

By default, tiered data is not included when you query from a Tiger Cloud service. To access tiered data, you enable tiered reads for a query, a session, or even for all sessions. After you enable tiered reads, when you run regular SQL queries, a behind-the-scenes process transparently pulls data from wherever it's located: the standard high-performance storage tier, the object storage tier, or both. You can JOIN against tiered data, build views, and even define continuous aggregates on it. In fact, because the implementation of continuous aggregates also uses hypertables, they can be tiered to low-cost storage as well.

For low-cost storage, Tiger Data charges only for the size of your data in S3 in the Apache Parquet format, regardless of whether it was compressed in Tiger Cloud before tiering. There are no additional expenses, such as data transfer or compute.

The low-cost storage tier comes with the following limitations:

• Limited schema modifications: some schema modifications are not allowed on hypertables with tiered chunks.

  Allowed modifications include: renaming the hypertable, adding columns with NULL defaults, adding indexes, changing or renaming the hypertable schema, and adding CHECK constraints. For CHECK constraints, only untiered data is verified. Columns can also be deleted, but you cannot subsequently add a new column to a tiered hypertable with the same name as the now-deleted column.

  Disallowed modifications include: adding a column with non-NULL defaults, renaming a column, changing the data type of a column, and adding a NOT NULL constraint to the column.

• Limited data changes: you cannot insert data into, update, or delete a tiered chunk. These limitations take effect as soon as the chunk is scheduled for tiering.

• Inefficient query planner filtering for non-native data types: the query planner speeds up reads from our object storage tier by using metadata to filter out columns and row groups that don't satisfy the query. This works for all native data types, but not for non-native types, such as JSON, JSONB, and GIS.

• Latency: S3 has higher access latency than local storage. This can affect the execution time of queries in latency-sensitive environments, especially lighter queries.

• Number of dimensions: you cannot use tiered storage with hypertables partitioned on more than one dimension. Make sure your hypertables are partitioned on time only, before you enable tiered storage.

===== PAGE: https://docs.tigerdata.com/use-timescale/security/overview/ =====

About security in Tiger Cloud

Protecting data starts with secure software engineering. At Tiger Data, we embed security into every stage of development, from static code analysis and automated dependency scanning to rigorous code security reviews. To go even further, we developed pgspot, an open-source extension to identify security issues with Postgres extensions, which strengthens the broader ecosystem as well as our own platform. Tiger Data products do not have any identified weaknesses.

This page lists the additional things we do to ensure operational security and to lock down Tiger Cloud services. To see our security features at a glance, see Tiger Data Security.

Role-based access

Tiger Cloud provides role-based access for you to:

• Administer your Tiger Cloud project In Tiger Cloud Console, users with the Owner, Admin, and Viewer roles have different permissions to manage users and services in the project.
• Manage data in each service To restrict access to your data on the database level, you can create other roles on top of the default tsdbadmin role.

Data encryption

Your data on Tiger Cloud is encrypted both in transit and at rest. Both active databases and backups are encrypted.

Tiger Cloud uses AWS as its cloud provider, with all the security that AWS provides. Data encryption uses the industry-standard AES-256 algorithm. Cryptographic keys are managed by AWS Key Management Service (AWS KMS). Keys are never stored in plaintext.

For more information about AWS security, see the AWS documentation on security in Amazon Elastic Compute Cloud and Elastic Block Storage.

Networking security

Customer access to Tiger Cloud services is only provided over TLS-encrypted connections. There is no option to use unencrypted plaintext connections.

Networking with Virtual Private Cloud (VPC) peering

When using VPC peering, no public Internet-based access is provided to the service. Service addresses are published in public DNS, but they can only be connected to from the customer's peered VPC using private network addresses.

VPC peering only enables communication to be initiated from your Customer VPC to Tiger Cloud services running in the Tiger Cloud VPC. Tiger Cloud cannot initiate communication with your VPC. To learn how to set up VPC Peering, see Secure your Tiger Cloud services with VPC Peering and AWS PrivateLink.

IP address allow lists

You can allow only trusted IP addresses to access your Tiger Cloud services. You do this by creating IP address allow lists and attaching them to your services.

Operator access

Normally all the resources required for providing Tiger Cloud services are automatically created, maintained and terminated by the Tiger Cloud infrastructure. No manual operator intervention is required.

However, the Tiger Data operations team has the capability to securely log in to the service virtual machines for troubleshooting purposes. These accesses are audit logged.

No customer access to the virtual machine level is provided.

GDPR compliance

Tiger Data complies with the European Union's General Data Protection Regulation (GDPR), and all practices are covered by our Privacy Policy and the Terms of Service. All customer data is processed in accordance with Tiger Data's GDPR-compliant Data Processor Addendum, which applies to all Tiger Data customers.

Tiger Data operators never access customer data, unless explicitly requested by the customer to troubleshoot a technical issue. The Tiger Data operations team has mandatory recurring training regarding the applicable policies.

HIPAA compliance

The Tiger Cloud Enterprise plan is Health Insurance Portability and Accountability Act (HIPAA) compliant. This allows organizations to securely manage and analyze sensitive healthcare data, ensuring they meet regulatory requirements while building compliant applications.

SOC 2 compliance

Tiger Cloud is SOC 2 Type 2 compliant. This ensures that organizations can securely manage customer data in alignment with industry standards for security, availability, processing integrity, confidentiality, and privacy. It helps businesses meet trust requirements while confidently building applications that handle sensitive information. The annual SOC 2 report is available to customers on the Scale or Enterprise pricing plans. Open a support ticket to get access to it.

===== PAGE: https://docs.tigerdata.com/use-timescale/security/strict-ssl/ =====

Connect with a stricter SSL mode

The default connection string for Tiger Cloud uses the Secure Sockets Layer (SSL) mode require. Users can choose not to use Transport Layer Security (TLS) while connecting to their databases, but connecting to production databases without encryption is strongly discouraged. To achieve even stronger security, clients may select to verify the identity of the server. If you want your connection client to verify the server's identity, you can connect with an SSL mode of verify-ca or verify-full. To do so, you need to store a copy of the certificate chain where your connection tool can find it.

This section provides instructions for setting up a stricter SSL connection.

SSL certificates

As part of the secure connection protocol, the server proves its identity by providing clients with a certificate. This certificate should be issued and signed by a well-known and trusted Certificate Authority.

Because requesting a certificate from a Certificate Authority takes some time, Tiger Cloud services are initialized with a self-signed certificate. This lets you start up a service immediately. After your service is started, a signed certificate is requested behind the scenes. The new certificate is usually received within 30 minutes. Your certificate is then replaced with almost no interruption. Connections are reset, and most clients reconnect automatically.

With the signed certificate, you can switch your connections to a stricter SSL mode, such as verify-ca or verify-full.

For more information on the different SSL modes, see the Postgres SSL mode descriptions.

Connect to your database with a stricter SSL mode

To set up a stricter SSL connection:

1. Generate a copy of your certificate chain and store it in the right location
2. Change your Tiger Cloud connection string

Connecting to your database with a stricter SSL mode

1. Use the openssl tool to connect to your Tiger Cloud service and get the certificate bundle. Store the bundle in a file called bundle.crt.

   Replace service URL with port with your Tiger Cloud connection URL:

   openssl s_client -showcerts -partial_chain -starttls postgres \
                -connect service URL with port < /dev/null 2>/dev/null | \
                awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ print }' > bundle.crt
   

2. Copy the bundle to your clipboard:

   pbcopy < bundle.crt
   

   xclip -sel clip < bundle.crt
   

   clip.exe < bundle.crt
   

3. Navigate to https://whatsmychaincert.com/. This online tool generates a full certificate chain, including the root Certificate Authority certificate, which is not included in the certificate bundle returned by the database.

4. Paste your certificate bundle in the provided box. Check Include Root Certificate. Click Generate Chain.

5. Save the downloaded certificate chain to ~/.postgresql/root.crt.

6. Change your Tiger Cloud connection string from sslmode=require to either sslmode=verify-full or sslmode=verify-ca. For example, to connect to your database with psql, run:

   psql "postgres://tsdbadmin@service URL with port/tsdb?sslmode=verify-full"
   

Verify the certificate type used by your database

To check whether the certificate has been replaced yet, connect to your database instance and inspect the returned certificate. We are using two certificate providers - Google and ZeroSSL, that's why chances are you can have a certificate issued by either of those CAs:

openssl s_client -showcerts -partial_chain -starttls postgres -connect <HOST>:<PORT> < /dev/null 2>/dev/null  | grep "Google\|ZeroSSL"

===== PAGE: https://docs.tigerdata.com/use-timescale/security/transit-gateway/ =====

Peer your Tiger Cloud services with AWS Transit Gateway

AWS Transit Gateway enables you to securely connect to your Tiger Cloud from AWS, Google Cloud, Microsoft Azure, or any other cloud or on-premise environment.

You use AWS Transit Gateway as a traffic controller for your network. Instead of setting up multiple direct connections to different clouds, on-premise data centers, and other AWS services, you connect everything to AWS Transit Gateway. This simplifies your network and makes it easier to manage and scale.

You can then create a peering connection between your Tiger Cloud services and AWS Transit Gateway in Tiger Cloud. This means that, no matter how big or complex your infrastructure is, you can connect securely to your Tiger Cloud services.

For enhanced security, you can add peering connections to multiple Transit Gateways with overlapping CIDRs—Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID. Otherwise, the existing connection is reused for your services in the same project and region.

To configure this secure connection, you:

1. Connect your infrastructure to AWS Transit Gateway.
2. Create a Tiger Cloud Peering VPC with a peering connection to AWS Transit Gateway.
3. Accept and configure the peering connection on your side.
4. Attach individual services to the Peering VPC.

AWS Transit Gateway enables you to connect from almost any environment, this page provides examples for the most common use cases.

1. Create a Peering VPC in Tiger Cloud Console

   1. In Security > VPC, click Create a VPC:

      

   2. Choose your region and IP range, name your VPC, then click Create VPC:

      

      Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your plan in Tiger Cloud Console.

   3. Add a peering connection:

      1. In the VPC Peering column, click Add.
      2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID.

      

      1. Click Add connection.

2. Accept and configure peering connection in your AWS account

   Once your peering connection appears as Processing, you can accept and configure it in AWS:

   1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as Connected in Tiger Cloud Console.

   2. Configure at least the following in your AWS account networking:

      • Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs.
      • Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs.
      • Security groups to allow outbound TCP 5432.

3. Attach a Tiger Cloud service to the Peering VPC In Tiger Cloud Console

   1. Select the service you want to connect to the Peering VPC.
   2. Click Operations > Security > VPC.
   3. Select the VPC, then click Attach VPC.

   You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time.

4. Connect your infrastructure to AWS Transit Gateway

   Establish connectivity between Azure and AWS. See the AWS architectural documentation for details.

5. Create a Peering VPC in Tiger Cloud Console

   1. In Security > VPC, click Create a VPC:

      

   2. Choose your region and IP range, name your VPC, then click Create VPC:

      

      Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your plan in Tiger Cloud Console.

   3. Add a peering connection:

      1. In the VPC Peering column, click Add.
      2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID.

      

      1. Click Add connection.

6. Accept and configure peering connection in your AWS account

   Once your peering connection appears as Processing, you can accept and configure it in AWS:

   1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as Connected in Tiger Cloud Console.

   2. Configure at least the following in your AWS account networking:

      • Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs.
      • Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs.
      • Security groups to allow outbound TCP 5432.

7. Attach a Tiger Cloud service to the Peering VPC In Tiger Cloud Console

   1. Select the service you want to connect to the Peering VPC.
   2. Click Operations > Security > VPC.
   3. Select the VPC, then click Attach VPC.

   You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time.

8. Connect your infrastructure to AWS Transit Gateway

   Establish connectivity between Google Cloud and AWS. See Connect HA VPN to AWS peer gateways.

9. Create a Peering VPC in Tiger Cloud Console

   1. In Security > VPC, click Create a VPC:

      

   2. Choose your region and IP range, name your VPC, then click Create VPC:

      

      Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your plan in Tiger Cloud Console.

   3. Add a peering connection:

      1. In the VPC Peering column, click Add.
      2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID.

      

      1. Click Add connection.

10. Accept and configure peering connection in your AWS account

    Once your peering connection appears as Processing, you can accept and configure it in AWS:

    1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as Connected in Tiger Cloud Console.

    2. Configure at least the following in your AWS account networking:

       • Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs.
       • Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs.
       • Security groups to allow outbound TCP 5432.

11. Attach a Tiger Cloud service to the Peering VPC In Tiger Cloud Console

    1. Select the service you want to connect to the Peering VPC.
    2. Click Operations > Security > VPC.
    3. Select the VPC, then click Attach VPC.

    You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time.

12. Connect your infrastructure to AWS Transit Gateway

    Establish connectivity between your on-premise infrastructure and AWS. See the Centralize network connectivity using AWS Transit Gateway.

13. Create a Peering VPC in Tiger Cloud Console

    1. In Security > VPC, click Create a VPC:

       

    2. Choose your region and IP range, name your VPC, then click Create VPC:

       

       Your service and Peering VPC must be in the same AWS region. The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your plan in Tiger Cloud Console.

    3. Add a peering connection:

       1. In the VPC Peering column, click Add.
       2. Provide your AWS account ID, Transit Gateway ID, CIDR ranges, and AWS region. Tiger Cloud creates a new isolated connection for every unique Transit Gateway ID.

       

       1. Click Add connection.

14. Accept and configure peering connection in your AWS account

    Once your peering connection appears as Processing, you can accept and configure it in AWS:

    1. Accept the peering request coming from Tiger Cloud. The request can take up to 5 min to arrive. Within 5 more minutes after accepting, the peering should appear as Connected in Tiger Cloud Console.

    2. Configure at least the following in your AWS account networking:

       • Your subnet route table to route traffic to your Transit Gateway for the Peering VPC CIDRs.
       • Your Transit Gateway route table to route traffic to the newly created Transit Gateway peering attachment for the Peering VPC CIDRs.
       • Security groups to allow outbound TCP 5432.

15. Attach a Tiger Cloud service to the Peering VPC In Tiger Cloud Console

    1. Select the service you want to connect to the Peering VPC.
    2. Click Operations > Security > VPC.
    3. Select the VPC, then click Attach VPC.

    You cannot attach a Tiger Cloud service to multiple Tiger Cloud VPCs at the same time.

You can now securely access your services in Tiger Cloud.

===== PAGE: https://docs.tigerdata.com/use-timescale/security/ip-allow-list/ =====

IP allow list

You can restrict access to your Tiger Cloud services to trusted IP addresses only. This prevents unauthorized connections without the need for a Virtual Private Cloud. Creating IP allow lists helps comply with security standards such as SOC 2 or HIPAA that require IP filtering. This is especially useful in regulated industries like finance, healthcare, and government.

For a more fine-grained control, you create separate IP allow lists for the ops mode and the data mode.

Create and attach an IP allow list in the ops mode

You create an IP allow list at the project level, then attach your service to it.

You attach a service to either one VPC, or one IP allow list. You cannot attach a service to a VPC and an IP allow list at the same time.

1. In Tiger Cloud Console, select Security > IP Allow List, then click Create IP Allow List

   

2. Enter your trusted IP addresses

   The number of IP addresses that you can include in one list depends on your pricing plan.

   

3. Name your allow list and click Create IP Allow List

   Click + Create IP Allow List to create another list. The number of IP allow lists you can create depends on your pricing plan.

4. Select a Tiger Cloud service, then click Operations > Security > IP Allow List

   

5. Select the list in the drop-down and click Apply

6. Type Apply in the confirmation popup

You have created and attached an IP allow list for the operations available in the ops mode. You can unattach or change the list attached to a service from the same tab.

Create an IP allow list in the data mode

You create an IP allow list in the data mode settings.

1. In Tiger Cloud Console, toggle Data

2. Click the project name in the upper left corner, then select Settings

3. Scroll down and toggle IP Allowlist

4. Add IP addresses

   1. Click Add entry.
   2. Enter an IP address or a range of IP addresses.
   3. Click Add.
   4. When all the IP addresses have been added, click Apply.
   5. Click Confirm.

You have successfully added an IP allow list for querying your service in the data mode.

===== PAGE: https://docs.tigerdata.com/use-timescale/security/multi-factor-authentication/ =====

Multi-factor user authentication

You can use two-factor authentication to log in to your Tiger Data account. Two-factor authentication, also known as two-step verification or 2FA, enables secure logins that require an authentication code in addition to your user password. The code is provided by an authenticator app on your mobile device. There are multiple authenticator apps available.

This page describes how to configure two-factor authentication with Google Authenticator.

Prerequisites

Before you begin, make sure you have:

• Installed the Google Authenticator application on your mobile device.

Configure two-factor authentication with Google Authenticator

Take the following steps to configure two-factor authentication:

1. Log in to Tiger Cloud Console with your username and password. 2FA is not available if you log in with Google SSO.
2. Click the User name icon in the bottom left of Tiger Cloud Console and select Account.
3. In Account, click Add two-factor authentication.
4. On your mobile device, open Google Authenticator, tap +, and select Scan a QR code.
5. Scan the QR code provided by Tiger Cloud Console in Connect to an authenticator app and click Next.
6. In Tiger Cloud Console, enter the verification code provided by Google Authenticator, and click Next.
7. In Save your recovery codes, copy, download, or print the recovery codes. These are used to recover your account if you lose your device.
8. Verify that you have saved your recovery codes, by clicking OK, I saved my recovery codes.
9. If two-factor authentication is enabled correctly, an email notification is sent to you.

If you lose access to the mobile device you use for multi-factor authentication, and you do not have access to your recovery codes, you cannot sign in to your Tiger Data account. To regain access to your account, contact support@tigerdata.com.

Regenerate recovery codes

If you do not have access to your authenticator app and need to log in to Tiger Cloud Console, you can use your recovery codes. Recovery codes are single-use. If you've used all 10 recovery codes, or lost access to them, you can generate another list. Generating a new list invalidates all previously generated codes.

1. Log in to Tiger Cloud Console with your username and password.
2. Click the User name icon in the bottom left and select Account.
3. In Account, navigate to Two-factor authentication.
4. Click Regenerate recovery codes.
5. In Two-factor authentication, enter the verification code from your authenticator app. Alternatively, if you do not have access to the authenticator app, click Use recovery code instead to enter a recovery code.
6. Click Next.
7. In Save your recovery codes, copy, download, or print the recovery codes. These are used to recover your account if you lose your device.
8. Verify that you have saved your recovery codes, by clicking OK, I saved my recovery codes.

Remove two-factor authentication

If you need to enroll a new device for two-factor authentication, you can remove two-factor authentication from your account and then add it again with your new device.

1. Log in to Tiger Cloud Console with your username and password.
2. Click the User name icon in the bottom left of Tiger Cloud Console and select Account.
3. In Account, navigate to Two-factor authentication.
4. Click Remove two-factor authentication.
5. Enter the verification code from your authenticator app to confirm. Alternatively click Use recovery code instead to type the recovery code.
6. Click Remove.

===== PAGE: https://docs.tigerdata.com/use-timescale/security/client-credentials/ =====

Client credentials

You can use client credentials to programmatically access resources instead of using your username and password. You can generate multiple client credentials for different applications or use cases rather than a single set of user credentials for everything.

Create client credentials

When you create client credentials, a public key and a private key are generated. These keys act as the username and password for programmatic client applications. It is important that you save these keys in a safe place. You can also delete these client credentials when the client applications no longer need access to Tiger Cloud resources. For more information about obtaining an access token programmatically, see the Tiger Cloud Terraform provider documentation.

Creating client credentials

1. Log in to your Tiger Data account.
2. Navigate to the Project Settings page to create client credentials for your project.
3. In the Project Settings page, click Create credentials.
4. In the New client credentials dialog, you can view the Public key and the Secret Key. Copy your secret key and store it in a secure place. You won't be able to view the Secret Key again in the console.
5. Click Done. You can use these keys in your client applications to access Tiger Cloud resources inside the respective project. Tiger Cloud generates a default Name for the client credentials.
6. Click the ⋮ menu and select Rename credentials.
7. In the Edit credential name dialog, type the new name and click Accept.

Deleting client credentials

1. Log in to your Tiger Data account.
2. Navigate to the Project Settings page to view client credentials for your project.
3. In the Project Settings page, click the ⋮ menu of the client credential, and select Delete.
4. In the Are you sure dialog, type the name of the client credential, and click Delete.

===== PAGE: https://docs.tigerdata.com/use-timescale/security/members/ =====

Control access to Tiger Cloud projects

When you sign up for a 30-day free trial, Tiger Cloud creates a project with built-in role-based access.

This includes the following roles:

• Owner: Tiger Cloud assigns this role to you when your project is created. As the Owner, you can add and delete other users, transfer project ownership, administer services, and edit project settings.
• Admin: the Owner assigns this role to other users in the project. A user with the Admin role has the same scope of rights as the Owner but cannot transfer project ownership.
• Developer: the Owner and Admins assign this role to other users in the project. A Developer can build, deploy, and operate services across projects, but does not have administrative privileges over users, roles, or billing. A Developer can invite other users to the project, but only with the Viewer role.
• Viewer: the Owner and Admins assign this role to other users in the project. A Viewer has limited, read-only access to Tiger Cloud Console. This means that a Viewer cannot modify services and their configurations in any way. A Viewer has no access to the data mode and has read-queries-only access to SQL editor.

If you have the Enterprise pricing plan, you can use your company SAML identity provider to log in to Console.

User roles in a Tiger Cloud project do not overlap with the database-level roles for the individual services. This page describes the project roles available in Console. For the database-level user roles, see Manage data security in your Tiger Cloud service.

Add a user to your project

New users do not need to have a Tiger Data account before you add them, they are prompted to create one when they respond to the confirmation email. Existing users join a project in addition to the other projects they are already members of.

To add a user to a project:

1. In Tiger Cloud Console, click Invite users, then click Add new user.

2. Type the email address of the person that you want to add, select their role, and click Invite user.

   

   Enterprise pricing plan and SAML users receive a notification in Console. Users in the other pricing plans receive a confirmation email. The new user then joins the project.

Join a project

When you are asked to join a project, Tiger Cloud Console sends you an invitation email. Follow the instructions in the invitation email to join the project:

1. In the invitation email, click Accept Invite

   Tiger Cloud opens.

2. Follow the setup wizard and create a new account

   You are added to the project you were invited to.

3. In the invitation email, click Accept Invite

   Tiger Cloud Console opens, and you are added to the project.

4. Log in to Console using your company's identity provider

5. Click Notifications, then accept the invitation

   Tiger Cloud Console opens, and you are added to the project. As you are now included in more than one project, you can easily change projects.

Resend a project invitation

Project invitations are valid for 7 days. To resend a project invitation:

1. In Tiger Cloud Console, click Invite users.

2. Next to the person you want to invite to your project, click Resend invitation.

   

Change your current project

To change the project you are currently working in:

1. In Tiger Cloud Console, click the project name > Current project in the top left.

   

2. Select the project you want to use.

Transfer project ownership

Each Tiger Cloud project has one Owner. As the project Owner, you have rights to add and delete users, edit project settings, and transfer the Owner role to another user. When you transfer ownership to another user, you lose your ownership rights.

To transfer project ownership:

1. In Tiger Cloud Console, click Invite users.

2. Next to the person you want to transfer project ownership to, click ⋮ > Transfer project ownership.

   

   If you are unable to transfer ownership, hover over the greyed out button to see the details.

3. Enter your password, and click Verify.

4. Complete the two-factor authentication challenge and click Confirm.

If you have the Enterprise pricing plan, and log in to Tiger Cloud using SAML authentication or have not enabled two-factor authentication, contact support to transfer project ownership.

Leave a project

To stop working in a project:

1. In Tiger Cloud Console, click Invite users.

2. Click ⋮ > Leave project, then click Leave.

Your account is removed from the project immediately, you can no longer access this project.

Change roles of other users in a project

The Owner can change the roles of all users in the project. An Admin can change the roles of all users other than the Owner. Developer and Viewer cannot change the roles of other users.

To change the role for another user:

1. In Tiger Cloud Console, click Invite users.

2. Next to the corresponding user, select another role in the dropdown.

   

   The user role is changed immediately.

Remove users from a project

To remove a user's access to a project:

1. In Tiger Cloud Console, click Invite users.
2. Next to the person you want to remove, click ⋮ > Remove.
3. In Remove user, click Remove.

The user is deleted immediately, they can no longer access your project.

===== PAGE: https://docs.tigerdata.com/use-timescale/security/vpc/ =====

Virtual Private Cloud

You use Virtual Private Cloud (VPC) peering to ensure that your Tiger Cloud services are only accessible through your secured AWS infrastructure. This reduces the potential attack vector surface and improves security.

The data isolation architecture that ensures a highly secure connection between your apps and Tiger Cloud is:

Your customer apps run inside your AWS Customer VPC, your Tiger Cloud services always run inside the secure Tiger Cloud VPC. You control secure communication between apps in your VPC and your services using a dedicated Peering VPC. The AWS PrivateLink connecting Tiger Cloud VPC to the dedicated Peering VPC gives the same level of protection as using a direct AWS PrivateLink connection. It only enables communication to be initiated from your Customer VPC to services running in the Tiger Cloud VPC. Tiger Cloud cannot initiate communication with your Customer VPC.

To configure this secure connection, you first create a Peering VPC with AWS PrivateLink in Tiger Cloud Console. After you have accepted and configured the peering connection to your Customer VPC, you use AWS Security Groups to restrict the apps in your Customer VPC that are visible to the Peering VPC. The last step is to attach individual services to the Peering VPC in Tiger Cloud Console.

• You create each Peering VPC on a Tiger Cloud project level.

• You can attach:

  • Up to 50 Customer VPCs to a Peering VPC.
  • A Tiger Cloud service to a single Peering VPC at a time. The service and the Peering VPC must be in the same AWS region. However, you can peer a Customer VPC and a Peering VPC that are in different regions.
  • Multiple Tiger Cloud services to the same Peering VPC.

• You cannot attach a Tiger Cloud service to multiple Peering VPCs at the same time.

  The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your pricing plan in Tiger Cloud Console.

Prerequisites

To set up VPC peering, you need the following permissions in your AWS account:

• Accept VPC peering requests
• Configure route table rules
• Configure security group and firewall rules

Set up a secured connection between Tiger Cloud and AWS

To connect to a Tiger Cloud service using VPC peering, your apps and infrastructure must be already running in an Amazon Web Services (AWS) VPC. You can peer your VPC from any AWS region. However, your Peering VPC must be within one of the Cloud-supported regions.

The stages to create a secured connection between Tiger Cloud services and your AWS infrastructure are:

1. Create a Peering VPC in Tiger Cloud Console
2. Complete the VPC connection in your AWS
3. Set up security groups in your AWS
4. Attach a Tiger Cloud service to the Peering VPC

Create a Peering VPC in Tiger Cloud Console

Create the VPC and the peering connection that enables you to securely route traffic between Tiger Cloud and your Customer VPC in a logically isolated virtual network.

1. In Tiger Cloud Console > Security > VPC, click Create a VPC

   

2. Choose your region and IP range, name your VPC, then click Create VPC

   

   The IP ranges of the Peering VPC and Customer VPC should not overlap.

3. For as many peering connections as you need:

   1. In the VPC Peering column, click Add.

   2. Enter information about your existing Customer VPC, then click Add Connection.

      

   • You can attach:

• Up to 50 Customer VPCs to a Peering VPC.

• A Tiger Cloud service to a single Peering VPC at a time. The service and the Peering VPC must be in the same AWS region. However, you can peer a Customer VPC and a Peering VPC that are in different regions.

• Multiple Tiger Cloud services to the same Peering VPC.

• You cannot attach a Tiger Cloud service to multiple Peering VPCs at the same time.

  The number of Peering VPCs you can create in your project depends on your pricing plan. If you need another Peering VPC, either contact support@tigerdata.com or change your pricing plan in Tiger Cloud Console.

Tiger Cloud sends a peering request to your AWS account so you can complete the VPC connection in AWS.

Complete the VPC connection in AWS

When you receive the Tiger Cloud peering request in AWS, edit your routing table to match the IP Range and CIDR block between your Customer and Peering VPCs.

When you peer a VPC with multiple CIDRs, all CIDRs are added to the Tiger Cloud rules automatically. After you have finished peering, further changes in your VPC's CIDRs are not detected automatically. If you need to refresh the CIDRs, recreate the peering connection.

The request acceptance process is an important safety mechanism. Do not accept a peering request from an unknown account.

1. In AWS > VPC Dashboard > Peering connections, select the peering connection request from Tiger Cloud

   Copy the peering connection ID to the clipboard. The connection request starts with pcx-.

2. In the peering connection, click Route Tables, then select the Route Table ID that corresponds to your VPC

3. In Routes, click Edit routes

   You see the list of existing destinations.

   .

   If you do not already have a destination that corresponds to the IP range / CIDR block of your Peering VPC:

   1. Click Add route, and set:
      • Destination: the CIDR block of your Peering VPC. For example: 10.0.0.7/17.
      • Target: the peering connection ID you copied to your clipboard.
   2. Click Save changes.

Network traffic is secured between your AWS account and Tiger Cloud for this project.

Set up security groups in AWS

Security groups allow specific inbound and outbound traffic at the resource level. You can associate a VPC with one or more security groups, and each instance in your VPC may belong to a different set of security groups. The security group choices for your VPC are:

• Create a security group to use for your Tiger Cloud VPC only.
• Associate your VPC with an existing security group.
• Do nothing, your VPC is automatically associated with the default one.

To create a security group specific to your Tiger Cloud Peering VPC:

1. AWS > VPC Dashboard > Security Groups, click Create security group

2. Enter the rules for this security group:

   

   • VPC: select the VPC that is peered with Tiger Cloud.
   • Inbound rules: leave empty.
   • Outbound rules:
     • Type: Custom TCP
     • Protocol: TCP
     • Port range: 5432
     • Destination: Custom
     • Info: the CIDR block of your Tiger Cloud Peering VPC.

3. Click Add rule, then click Create security group

Attach a Tiger Cloud service to the Peering VPC

Now that Tiger Cloud is communicating securely with your AWS infrastructure, you can attach one or more services to the Peering VPC.

After you attach a service to a Peering VPC, you can only access it through the peered AWS VPC. It is no longer accessible using the public internet.

1. In Tiger Cloud Console > Services select the service you want to connect to the Peering VPC
2. Click Operations > Security > VPC
3. Select the VPC, then click Attach VPC

And that is it, your service is now securely communicating with your AWS account inside a VPC.

Migrate a Tiger Cloud service between VPCs

To ensure that your applications continue to run without interruption, you keep service attached to the Peering VPC. However, you can change the Peering VPC your service is attached to, or disconnect from the Peering VPC and enable access to the service from the public internet.

Tiger Cloud uses a different DNS for services that are attached to a Peering VPC. When you migrate a service between public access and a Peering VPC, you need to update your connection string.

1. In Tiger Cloud Console > Services select the service to migrate

   If you don't have a service, create a new one.

2. Click Operations > Security > VPC

3. Select the VPC, then click Attach VPC

Migration takes a few minutes to complete and requires a change to DNS settings for the service. The service is not accessible during this time. If you receive a DNS error, allow some time for DNS propagation.

===== PAGE: https://docs.tigerdata.com/use-timescale/security/read-only-role/ =====

Manage data security in your Tiger Cloud service

When you create a service, Tiger Cloud assigns you the tsdmadmin role. This role has full permissions to modify data in your service. However, Tiger Cloud does not provide superuser access. tsdmadmin is not a superuser.

As tsdmadmin, you can use standard Postgres means to create other roles or assign individual permissions. This page shows you how to create a read-only role for your database. Adding a read-only role does not provide resource isolation. To restrict the access of a read-only user, as well as isolate resources, create a read replica instead.

The database-level roles for the individual services in your project do not overlap with the Tiger Cloud project user roles. This page describes the database-level roles. For user roles available in Console, see Control user access to Tiger Cloud projects.

Create a read-only user

You can create a read-only user to provide limited access to your database.

1. Connect to your service as the tsdbadmin user.

2. Create the new role:

   CREATE ROLE readaccess;
   

3. Grant the appropriate permissions for the role, as required. For example, to grant SELECT permissions to a specific table, use:

   GRANT SELECT ON  TO readaccess;
   

   To grant SELECT permissions to all tables in a specific schema, use:

   GRANT SELECT ON ALL TABLES IN SCHEMA <SCHEMA_NAME> TO readaccess;
   

4. Create a new user:

   CREATE USER read_user WITH PASSWORD 'read_password';
   

5. Assign the role to the new user:

   GRANT readaccess TO read_user;
   

===== PAGE: https://docs.tigerdata.com/use-timescale/security/saml/ =====

SAML (Security Assertion Markup Language)

Tiger Cloud offers SAML authentication as part of its Enterprise offering. SAML (Security Assertion Markup Language) is an open standard for exchanging authentication and authorization data between parties. With SAML enabled Tiger Cloud customers can log into their Tiger Data account using their existing SSO service provider credentials.

Tiger Cloud supports most SAML providers that can handle IDP-initiated login

SAML offers many benefits for the Enterprise including:

• Improved security: SAML centralizes user authentication with an identity provider (IdP). This makes it more difficult for attackers to gain access to user accounts.
• Reduced IT costs: SAML can help companies reduce IT costs by eliminating the need to manage multiple user accounts and passwords.
• Improved user experience: SAML makes it easier for users to access multiple applications and resources.

Reach out to your CSM/sales contact to get started. The connection process looks like the following:

1. Configure the IdP to support SAML authentication. This will involve creating a new application and configuring the IdP with the settings provided by your contact.
2. Provide your contact with the requested details about your IdP.
3. Test the SAML authentication process to make sure that it is working correctly.

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/alter/ =====

Altering and updating table schemas

To modify the schema of an existing hypertable, you can use the ALTER TABLE command. When you change the hypertable schema, the changes are also propagated to each underlying chunk.

While you can change the schema of an existing hypertable, you cannot change the schema of a continuous aggregate. For continuous aggregates, the only permissible changes are renaming a view, setting a schema, changing the owner, and adjusting other parameters.

For example, to add a new column called address to a table called distributors:

ALTER TABLE distributors
  ADD COLUMN address varchar(30);

This creates the new column, with all existing entries recording NULL for the new column.

Changing the schema can, in some cases, consume a lot of resources. This is especially true if it requires underlying data to be rewritten. If you want to check your schema change before you apply it, you can use a CHECK constraint, like this:

ALTER TABLE distributors
  ADD CONSTRAINT zipchk
  CHECK (char_length(zipcode) = 5);

This scans the table to verify that existing rows meet the constraint, but does not require a table rewrite.

For more information, see the Postgres ALTER TABLE documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/about-constraints/ =====

About constraints

Constraints are rules that apply to your database columns. This prevents you from entering invalid data into your database. When you create, change, or delete constraints on your hypertables, the constraints are propagated to the underlying chunks, and to any indexes.

Hypertables support all standard Postgres constraint types. For foreign keys in particular, the following is supported:

• Foreign key constraints from a hypertable referencing a regular table
• Foreign key constraints from a regular table referencing a hypertable

Foreign keys from a hypertable referencing another hypertable are not supported.

For example, you can create a table that only allows positive device IDs, and non-null temperature readings. You can also check that time values for all devices are unique. To create this table, with the constraints, use this command:

CREATE TABLE conditions (
    time       TIMESTAMPTZ
    temp       FLOAT NOT NULL,
    device_id  INTEGER CHECK (device_id > 0),
    location   INTEGER REFERENCES locations (id),
    PRIMARY KEY(time, device_id)
) WITH (
    tsdb.hypertable,
    tsdb.partition_column='time'
);

If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

This example also references values in another locations table using a foreign key constraint.

Time columns used for partitioning must not allow NULL values. A NOT NULL constraint is added by default to these columns if it doesn't already exist.

For more information on how to manage constraints, see the Postgres docs.

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/about-indexing/ =====

About indexes

Because looking up data can take a long time, especially if you have a lot of data in your hypertable, you can use an index to speed up read operations from non-compressed chunks in the rowstore (which use their own columnar indexes).

You can create an index on any combination of columns. To define an index as a UNIQUE or PRIMARY KEY index, it must include the partitioning column (this is usually the time column).

Which column you choose to create your index on depends on what kind of data you have stored. When you create a hypertable, set the datatype for the time column as timestamptz and not timestamp. For more information, see Postgres timestamp.

While it is possible to add an index that does not include the time column, doing so results in very slow ingest speeds. For time-series data, indexing on the time column allows one index to be created per chunk.

Consider a simple example with temperatures collected from two locations named office and garage:

An index on (location, time DESC) is organized like this:

garage-0940
garage-0930
garage-0920
garage-0910
office-0930
office-0920
office-0910

An index on (time DESC, location) is organized like this:

0940-garage
0930-garage
0930-office
0920-garage
0920-office
0910-garage
0910-office

A good rule of thumb with indexes is to think in layers. Start by choosing the columns that you typically want to run equality operators on, such as location = garage. Then finish by choosing columns you want to use range operators on, such as time > 0930.

As a more complex example, imagine you have a number of devices tracking 1,000 different retail stores. You have 100 devices per store, and 5 different types of devices. All of these devices report metrics as float values, and you decide to store all the metrics in the same table, like this:

CREATE TABLE devices (
     time timestamptz,
     device_id int,
     device_type int,
     store_id int,
     value float
);

When you create this table, an index is automatically generated on the time column, making it faster to query your data based on time.

If you want to query your data on something other than time, you can create different indexes. For example, you might want to query data from the last month for just a given device_id. Or you could query all data for a single store_id for the last three months.

You want to keep the index on time so that you can quickly filter for a given time range, and add another index on device_id and store_id. This creates a composite index. A composite index on (store_id, device_id, time) orders by store_id first. Each unique store_id, will then be sorted by device_id in order. And each entry with the same store_id and device_id are then ordered by time. To create this index, use this command:

CREATE INDEX ON devices (store_id, device_id, time DESC);

When you have this composite index on your hypertable, you can run a range of different queries. Here are some examples:

SELECT * FROM devices WHERE store_id = x

This queries the portion of the list with a specific store_id. The index is effective for this query, but could be a bit bloated; an index on just store_id would probably be more efficient.

SELECT * FROM devices WHERE store_id = x, time > 10

This query is not effective, because it would need to scan multiple sections of the list. This is because the part of the list that contains data for time > 10 for one device would be located in a different section than for a different device. In this case, consider building an index on (store_id, time) instead.

SELECT * FROM devices WHERE device_id = M, time > 10

The index in the example is useless for this query, because the data for device M is located in a completely different section of the list for each store_id.

SELECT * FROM devices WHERE store_id = M, device_id = M, time > 10

This is an accurate query for this index. It narrows down the list to a very specific portion.

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/json/ =====

JSONB support for semi-structured data

You can use JSON and JSONB to provide semi-structured data. This is most useful for data that contains user-defined fields, such as field names that are defined by individual users and vary from user to user. We recommend using this in a semi-structured way, for example:

CREATE TABLE metrics (
  time TIMESTAMPTZ,
  user_id INT,
  device_id INT,
  data JSONB
);

When you are defining a schema using JSON, ensure that common fields, such as time, user_id, and device_id, are pulled outside of the JSONB structure and stored as columns. This is because field accesses are more efficient on table columns than inside JSONB structures. Storage is also more efficient.

You should also use the JSONB data type, that is, JSON stored in a binary format, rather than JSON data type. JSONB data types are more efficient in both storage overhead and lookup performance.

Use JSONB for user-defined data rather than sparse data. This works best for most data sets. For sparse data, use NULLable fields and, if possible, run on top of a compressed file system like ZFS. This will work better than a JSONB data type, unless the data is extremely sparse, for example, more than 95% of fields for a row are empty.

Index the JSONB structure

When you index JSONB data across all fields, it is usually best to use a GIN (generalized inverted) index. In most cases, you can use the default GIN operator, like this:

CREATE INDEX idxgin ON metrics USING GIN (data);

For more information about GIN indexes, see the Postgres documentation.

This index only optimizes queries where the WHERE clause uses the ?, ?&, ?|, or @> operator. For more information about these operators, see the Postgres documentation.

Index individual fields

JSONB columns sometimes have common fields containing values that are useful to index individually. Indexes like this can be useful for ordering operations on field values, multicolumn indexes, and indexes on specialized types, such as a postGIS geography type. Another advantage of indexes on individual field values is that they are often smaller than GIN indexes on the entire JSONB field. To create an index like this, it is usually best to use a partial index on an expression accessing the field. For example:

CREATE INDEX idxcpu
  ON metrics(((data->>'cpu')::double precision))
  WHERE data ? 'cpu';

In this example, the expression being indexed is the cpu field inside the data JSONB object, cast to a double. The cast reduces the size of the index by storing the much smaller double, instead of a string. The WHERE clause ensures that the only rows included in the index are those that contain a cpu field, because the data ? 'cpu' returns true. This also serves to reduce the size of the index by not including rows without a cpu field. Note that in order for a query to use the index, it must have data ? 'cpu' in the WHERE clause.

This expression can also be used with a multi-column index, for example, by adding time DESC as a leading column. Note, however, that to enable index-only scans, you need data as a column, not the full expression ((data->>'cpu')::double precision).

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/about-tablespaces/ =====

About tablespaces

Tablespaces are used to determine the physical location of the tables and indexes in your database. In most cases, you want to use faster storage to store data that is accessed frequently, and slower storage for data that is accessed less often.

Hypertables consist of a number of chunks, and each chunk can be located in a specific tablespace. This allows you to grow your hypertables across many disks. When you create a new chunk, a tablespace is automatically selected to store the chunk's data.

You can attach and detach tablespaces on a hypertable. When a disk runs out of space, you can detach the full tablespace from the hypertable, and than attach a tablespace associated with a new disk. To see the tablespaces for you hypertable, use the show_tablespaces command.

How hypertable chunks are assigned tablespaces

A hypertable can be partitioned in multiple dimensions, but only one of the dimensions is used to determine the tablespace assigned to a particular hypertable chunk. If a hypertable has one or more hash-partitioned, or space, dimensions, it uses the first hash-partitioned dimension. Otherwise, it uses the first time dimension.

This strategy ensures that hash-partitioned hypertables have chunks co-located according to hash partition, as long as the list of tablespaces attached to the hypertable remains the same. Modulo calculation is used to pick a tablespace, so there can be more partitions than tablespaces. For example, if there are two tablespaces, partition number three uses the first tablespace.

Hypertables that are only time-partitioned add new partitions continuously, and therefore have chunks assigned to tablespaces in a way similar to round-robin.

It is possible to attach more tablespaces than there are partitions for the hypertable. In this case, some tablespaces remain unused until others are detached or additional partitions are added. This is especially true for hash-partitioned tables.

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/about-schemas/ =====

Table management

A database schema defines how the tables and indexes in your database are organized. Using a schema that is appropriate for your workload can result in significant performance improvements. Conversely, using a poorly suited schema can result in significant performance degradation.

If you are working with semi-structured data, such as readings from IoT sensors that collect varying measurements, you might need a flexible schema. In this case, you can use Postgres JSON and JSONB data types.

TimescaleDB supports all table objects supported within Postgres, including data types, indexes, and triggers. However, when you create a hypertable, set the datatype for the time column as timestamptz and not timestamp. For more information, see Postgres timestamp.

This section explains how to design your schema, how indexing and tablespaces work, and how to use Postgres constraint types. It also includes examples to help you create your own schema, and learn how to use JSON and JSONB for semi-structured data.

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/indexing/ =====

Indexing data

You can use an index on your database to speed up read operations. You can create an index on any combination of columns. TimescaleDB supports all table objects supported within Postgres, including data types, indexes, and triggers.

You can create an index using the CREATE INDEX command. For example, to create an index that sorts first by location, then by time, in descending order:

CREATE INDEX ON conditions (location, time DESC);

You can run this command before or after you convert a regular Postgres table to a hypertable.

Default indexes

Some indexes are created by default when you perform certain actions on your database.

When you create a hypertable with a call to CREATE TABLE, a time index is created on your data. If you want to manually create a time index, you can use this command:

CREATE INDEX ON conditions (time DESC);

You can also create an additional index on another column and time. For example:

CREATE INDEX ON conditions (location, time DESC);

TimescaleDB also creates sparse indexes per compressed chunk for optimization. You can manually set up those indexes when you call CREATE TABLE or ALTER_TABLE.

For more information about the order to use when declaring indexes, see the about indexing section.

If you do not want to create default indexes, you can set create_default_indexes to false when you create a hypertable. For example:

CREATE TABLE conditions (
  time        TIMESTAMPTZ       NOT NULL,
  location    TEXT              NOT NULL,
  device      TEXT              NOT NULL,
  temperature DOUBLE PRECISION  NULL,
  humidity    DOUBLE PRECISION  NULL
) WITH (
  tsdb.hypertable,
  tsdb.partition_column='time',
  tsdb.create_default_indexes=false
);

OldCreateHypertable

Refer to the installation documentation for detailed setup instructions.

Best practices for indexing

If you have sparse data, with columns that are often NULL, you can add a clause to the index, saying WHERE column IS NOT NULL. This prevents the index from indexing NULL data, which can lead to a more compact and efficient index. For example:

CREATE INDEX ON conditions (time DESC, humidity)
  WHERE humidity IS NOT NULL;

To define an index as a UNIQUE or PRIMARY KEY index, the index must include the time column and the partitioning column, if you are using one. For example, a unique index must include at least the (time, location) columns, in addition to any other columns you want to use. Generally, time-series data uses UNIQUE indexes more rarely than relational data.

If you do not want to create an index in a single transaction, you can use the CREATE_INDEX function. This uses a separate function to create an index on each chunk, instead of a single transaction for the entire hypertable. This means that you can perform other actions on the table while the index is being created, rather than having to wait until index creation is complete.

You can also use the Postgres WITH clause to perform indexing transactions on an individual chunk.

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/triggers/ =====

Triggers

TimescaleDB supports the full range of Postgres triggers. Creating, altering, or dropping triggers on a hypertable propagates the changes to all of the underlying chunks.

Create a trigger

This example creates a new table called error_conditions with the same schema as conditions, but that only stores records which are considered errors. An error, in this case, is when an application sends a temperature or humidity reading with a value that is greater than or equal to 1000.

Creating a trigger

1. Create a function that inserts erroneous data into the error_conditions table:

   CREATE OR REPLACE FUNCTION record_error()
     RETURNS trigger AS $record_error$
   BEGIN
    IF NEW.temperature >= 1000 OR NEW.humidity >= 1000 THEN
      INSERT INTO error_conditions
        VALUES(NEW.time, NEW.location, NEW.temperature, NEW.humidity);
    END IF;
    RETURN NEW;
   END;
   $record_error$ LANGUAGE plpgsql;
   

2. Create a trigger that calls this function whenever a new row is inserted into the hypertable:

   CREATE TRIGGER record_error
     BEFORE INSERT ON conditions
     FOR EACH ROW
     EXECUTE PROCEDURE record_error();
   

3. All data is inserted into the conditions table, but rows that contain errors are also added to the error_conditions table.

TimescaleDB supports the full range of triggers, including BEFORE INSERT, AFTER INSERT, BEFORE UPDATE, AFTER UPDATE, BEFORE DELETE, and AFTER DELETE. For more information, see the Postgres docs.

===== PAGE: https://docs.tigerdata.com/use-timescale/schema-management/foreign-data-wrappers/ =====

Foreign data wrappers

You use Postgres foreign data wrappers (FDWs) to query external data sources from a Tiger Cloud service. These external data sources can be one of the following:

• Other Tiger Cloud services
• Postgres databases outside of Tiger Cloud

If you are using VPC peering, you can create FDWs in your Customer VPC to query a service in your Tiger Cloud project. However, you can't create FDWs in your Tiger Cloud services to query a data source in your Customer VPC. This is because Tiger Cloud VPC peering uses AWS PrivateLink for increased security. See VPC peering documentation for additional details.

Postgres FDWs are particularly useful if you manage multiple Tiger Cloud services with different capabilities, and need to seamlessly access and merge regular and time-series data.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Query another data source

To query another data source:

You create Postgres FDWs with the postgres_fdw extension, which is enabled by default in Tiger Cloud.

1. Connect to your service

   See how to connect.

2. Create a server

   Run the following command using your connection details:

   CREATE SERVER myserver
   FOREIGN DATA WRAPPER postgres_fdw
   OPTIONS (host '<host>', dbname 'tsdb', port '<port>');
   

3. Create user mapping

   Run the following command using your connection details:

   CREATE USER MAPPING FOR tsdbadmin
   SERVER myserver
   OPTIONS (user 'tsdbadmin', password '<password>');
   

4. Import a foreign schema (recommended) or create a foreign table

   • Import the whole schema:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     FROM SERVER myserver
     INTO foreign_stuff ;
     

   • Alternatively, import a limited number of tables:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     LIMIT TO (table1, table2)
     FROM SERVER myserver
     INTO foreign_stuff;
     

   • Create a foreign table. Skip if you are importing a schema:

     CREATE FOREIGN TABLE films (
         code        char(5) NOT NULL,
         title       varchar(40) NOT NULL,
         did         integer NOT NULL,
         date_prod   date,
         kind        varchar(10),
         len         interval hour to minute
     )
     SERVER film_server;
     

A user with the tsdbadmin role assigned already has the required USAGE permission to create Postgres FDWs. You can enable another user, without the tsdbadmin role assigned, to query foreign data. To do so, explicitly grant the permission. For example, for a new grafana user:

CREATE USER grafana;

GRANT grafana TO tsdbadmin;

CREATE SCHEMA fdw AUTHORIZATION grafana;

CREATE SERVER db1 FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host '<host>', dbname 'tsdb', port '<port>');

CREATE USER MAPPING FOR grafana SERVER db1
OPTIONS (user 'tsdbadmin', password '<password>');

GRANT USAGE ON FOREIGN SERVER db1 TO grafana;

SET ROLE grafana;

IMPORT FOREIGN SCHEMA public
       FROM SERVER db1
       INTO fdw;

You create Postgres FDWs with the postgres_fdw extension. See documenation on how to enable it.

1. Connect to your database

   Use psql to connect to your database.

2. Create a server

   Run the following command using your connection details:

   CREATE SERVER myserver
   FOREIGN DATA WRAPPER postgres_fdw
   OPTIONS (host '<host>', dbname '<database_name>', port '<port>');
   

3. Create user mapping

   Run the following command using your connection details:

   CREATE USER MAPPING FOR postgres
   SERVER myserver
   OPTIONS (user 'postgres', password '<password>');
   

4. Import a foreign schema (recommended) or create a foreign table

   • Import the whole schema:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     FROM SERVER myserver
     INTO foreign_stuff ;
     

   • Alternatively, import a limited number of tables:

     CREATE SCHEMA foreign_stuff;
     
     IMPORT FOREIGN SCHEMA public
     LIMIT TO (table1, table2)
     FROM SERVER myserver
     INTO foreign_stuff;
     

   • Create a foreign table. Skip if you are importing a schema:

     CREATE FOREIGN TABLE films (
         code        char(5) NOT NULL,
         title       varchar(40) NOT NULL,
         did         integer NOT NULL,
         date_prod   date,
         kind        varchar(10),
         len         interval hour to minute
     )
     SERVER film_server;
     

===== PAGE: https://docs.tigerdata.com/use-timescale/write-data/insert/ =====

Insert data

Insert data into a hypertable with a standard INSERT SQL command.

Insert a single row

To insert a single row into a hypertable, use the syntax INSERT INTO ... VALUES. For example, to insert data into a hypertable named conditions:

INSERT INTO conditions(time, location, temperature, humidity)
  VALUES (NOW(), 'office', 70.0, 50.0);

Insert multiple rows

You can also insert multiple rows into a hypertable using a single INSERT call. This works even for thousands of rows at a time. This is more efficient than inserting data row-by-row, and is recommended when possible.

Use the same syntax, separating rows with a comma:

INSERT INTO conditions
  VALUES
    (NOW(), 'office', 70.0, 50.0),
    (NOW(), 'basement', 66.5, 60.0),
    (NOW(), 'garage', 77.0, 65.2);

You can insert multiple rows belonging to different chunks within the same INSERT statement. Behind the scenes, TimescaleDB batches the rows by chunk, and writes to each chunk in a single transaction.

Insert and return data

In the same INSERT command, you can return some or all of the inserted data by adding a RETURNING clause. For example, to return all the inserted data, run:

INSERT INTO conditions
  VALUES (NOW(), 'office', 70.1, 50.1)
  RETURNING *;

This returns:

time                          | location | temperature | humidity
------------------------------+----------+-------------+----------
2017-07-28 11:42:42.846621+00 | office   |        70.1 |     50.1
(1 row)

===== PAGE: https://docs.tigerdata.com/use-timescale/write-data/about-writing-data/ =====

About writing data

TimescaleDB supports writing data in the same way as Postgres, using INSERT, UPDATE, INSERT ... ON CONFLICT, and DELETE.

TimescaleDB is optimized for running real-time analytics workloads on time-series data. For this reason, hypertables are optimized for inserts to the most recent time intervals. Inserting data with recent time values gives excellent performance. However, if you need to make frequent updates to older time intervals, you might see lower write throughput.

===== PAGE: https://docs.tigerdata.com/use-timescale/write-data/upsert/ =====

Upsert data

Upserting is an operation that performs both:

• Inserting a new row if a matching row doesn't already exist
• Either updating the existing row, or doing nothing, if a matching row already exists

Upserts only work when you have a unique index or constraint. A matching row is one that has identical values for the columns covered by the index or constraint.

In Postgres, a primary key is a unique index with a NOT NULL constraint. If you have a primary key, you automatically have a unique index.

Create a table with a unique constraint

The examples in this section use a conditions table with a unique constraint on the columns (time, location). To create a unique constraint, use UNIQUE (<COLUMNS>) while defining your table:

CREATE TABLE conditions (
  time        TIMESTAMPTZ       NOT NULL,
  location    TEXT              NOT NULL,
  temperature DOUBLE PRECISION  NULL,
  humidity    DOUBLE PRECISION  NULL,
  UNIQUE (time, location)
);

You can also create a unique constraint after the table is created. Use the syntax ALTER TABLE ... ADD CONSTRAINT ... UNIQUE. In this example, the constraint is named conditions_time_location:

ALTER TABLE conditions
  ADD CONSTRAINT conditions_time_location
    UNIQUE (time, location);

When you add a unique constraint to a table, you can't insert data that violates the constraint. In other words, if you try to insert data that has identical values to another row, within the columns covered by the constraint, you get an error.

Unique constraints must include all partitioning columns. That means unique constraints on a hypertable must include the time column. If you added other partitioning columns to your hypertable, the constraint must include those as well. For more information, see the section on hypertables and unique indexes.

Insert or update data to a table with a unique constraint

You can tell the database to insert new data if it doesn't violate the constraint, and to update the existing row if it does. Use the syntax INSERT INTO ... VALUES ... ON CONFLICT ... DO UPDATE.

For example, to update the temperature and humidity values if a row with the specified time and location already exists, run:

INSERT INTO conditions
  VALUES ('2017-07-28 11:42:42.846621+00', 'office', 70.2, 50.1)
  ON CONFLICT (time, location) DO UPDATE
    SET temperature = excluded.temperature,
        humidity = excluded.humidity;

Insert or do nothing to a table with a unique constraint

You can also tell the database to do nothing if the constraint is violated. The new data is not inserted, and the old row is not updated. This is useful when writing many rows as one batch, to prevent the entire transaction from failing. The database engine skips the row and moves on.

To insert or do nothing, use the syntax INSERT INTO ... VALUES ... ON CONFLICT DO NOTHING:

INSERT INTO conditions
  VALUES ('2017-07-28 11:42:42.846621+00', 'office', 70.1, 50.0)
  ON CONFLICT DO NOTHING;

===== PAGE: https://docs.tigerdata.com/use-timescale/write-data/delete/ =====

Delete data

You can delete data from a hypertable using a standard DELETE SQL command. If you want to delete old data once it reaches a certain age, you can also drop entire chunks or set up a data retention policy.

Delete data with DELETE command

To delete data from a table, use the syntax DELETE FROM .... In this example, data is deleted from the table conditions, if the row's temperature or humidity is below a certain level:

DELETE FROM conditions WHERE temperature < 35 OR humidity < 60;

If you delete a lot of data, run VACUUM or VACUUM FULL to reclaim storage from the deleted or obsolete rows.

Delete data by dropping chunks

TimescaleDB allows you to delete data by age, by dropping chunks from a hypertable. You can do so either manually or by data retention policy.

To learn more, see the data retention section.

===== PAGE: https://docs.tigerdata.com/use-timescale/write-data/update/ =====

Update data

Update data in a hypertable with a standard UPDATE SQL command.

Update a single row

Update a single row with the syntax UPDATE ... SET ... WHERE. For example, to update a row in the conditions hypertable with new temperature and humidity values, run the following. The WHERE clause specifies the row to be updated.

UPDATE conditions
  SET temperature = 70.2, humidity = 50.0
  WHERE time = '2017-07-28 11:42:42.846621+00'
    AND location = 'office';

Update multiple rows at once

You can also update multiple rows at once, by using a WHERE clause that filters for more than one row. For example, run the following to update all temperature values within the given 10-minute span:

UPDATE conditions
  SET temperature = temperature + 0.1
  WHERE time >= '2017-07-28 11:40'
    AND time < '2017-07-28 11:50';

===== PAGE: https://docs.tigerdata.com/use-timescale/hypertables/hypertables-and-unique-indexes/ =====

Enforce constraints with unique indexes

You use unique indexes on a hypertable to enforce constraints. If you have a primary key, you have a unique index. In Postgres, a primary key is a unique index with a NOT NULL constraint.

You do not need to have a unique index on your hypertables. When you create a unique index, it must contain all the partitioning columns of the hypertable.

Create a hypertable and add unique indexes

To create a unique index on a hypertable:

1. Determine the partitioning columns

   Before you create a unique index, you need to determine which unique indexes are allowed on your hypertable. Begin by identifying your partitioning columns.

   TimescaleDB traditionally uses the following columns to partition hypertables:

   • The time column used to create the hypertable. Every TimescaleDB hypertable is partitioned by time.
   • Any space-partitioning columns. Space partitions are optional and not included in every hypertable.

2. Create a hypertable

   Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data. For example:

   CREATE TABLE hypertable_example(
     time TIMESTAMPTZ,
     user_id BIGINT,
     device_id BIGINT,
     value FLOAT
   ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='time',
     tsdb.segmentby = 'device_id',
     tsdb.orderby = 'time DESC'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

3. Create a unique index on the hypertable

   When you create a unique index on a hypertable, it must contain all the partitioning columns. It may contain other columns as well, and they may be arranged in any order. You cannot create a unique index without time, because time is a partitioning column.

   For example:

   • Create a unique index on time and device_id with a call to CREATE UNIQUE INDEX:

     CREATE UNIQUE INDEX idx_deviceid_time
       ON hypertable_example(device_id, time);
     

   • Create a unique index on time, user_id, and device_id.

     device_id is not a partitioning column, but this still works:

     CREATE UNIQUE INDEX idx_userid_deviceid_time
       ON hypertable_example(user_id, device_id, time);
     

   This restriction is necessary to guarantee global uniqueness in the index.

Create a hypertable from an existing table with unique indexes

If you create a unique index on a table before turning it into a hypertable, the same restrictions apply in reverse. You can only partition the table by columns in your unique index.

1. Create a relational table

   CREATE TABLE another_hypertable_example(
     time TIMESTAMPTZ,
     user_id BIGINT,
     device_id BIGINT,
     value FLOAT
   );
   

2. Create a unique index on the table

   For example, on device_id and time:

   CREATE UNIQUE INDEX idx_deviceid_time
     ON another_hypertable_example(device_id, time);
   

3. Turn the table into a partitioned hypertable

   • On time alone:

     SELECT * from create_hypertable('another_hypertable_example', by_range('time'));
     

   • On time and device_id:

     SELECT * FROM create_hypertable('another_hypertable_example', by_range('time'));
     SELECT * FROM add_dimension('another_hypertable_example', by_hash('device_id', 4));
     

   You get an error if you try to turn the relational table into a hypertable partitioned by time and user_id. This is because user_id is not part of the UNIQUE INDEX. To fix the error, add user_id to your unique index.

===== PAGE: https://docs.tigerdata.com/use-timescale/hypertables/hypertable-crud/ =====

Optimize time-series data in hypertables

Hypertables are designed for real-time analytics, they are Postgres tables that automatically partition your data by time. Typically, you partition hypertables on columns that hold time values. Best practice is to use timestamptz column type. However, you can also partition on date, integer, timestamp and UUIDv7 types.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Create a hypertable

Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data:

CREATE TABLE conditions (
   time        TIMESTAMPTZ       NOT NULL,
   location    TEXT              NOT NULL,
   device      TEXT              NOT NULL,
   temperature DOUBLE PRECISION  NULL,
   humidity    DOUBLE PRECISION  NULL
) WITH (
   tsdb.hypertable,
   tsdb.partition_column='time',
   tsdb.segmentby = 'device',
   tsdb.orderby = 'time DESC'
);

If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

To convert an existing table with data in it, call create_hypertable on that table with migrate_data to true. However, if you have a lot of data, this may take a long time.

Speed up data ingestion

When you set timescaledb.enable_direct_compress_copy your data gets compressed in memory during ingestion with COPY statements. By writing the compressed batches immediately in the columnstore, the IO footprint is significantly lower. Also, the columnstore policy you set is less important, INSERT already produces compressed chunks.

Please note that this feature is a tech preview and not production-ready. Using this feature could lead to regressed query performance and/or storage ratio, if the ingested batches are not correctly ordered or are of too high cardinality.

To enable in-memory data compression during ingestion:

SET timescaledb.enable_direct_compress_copy=on;

Important facts

• High cardinality use cases do not produce good batches and lead to degreaded query performance.
• The columnstore is optimized to store 1000 records per batch, which is the optimal format for ingestion per segment by.
• WAL records are written for the compressed batches rather than the individual tuples.
• Currently only COPY is support, INSERT will eventually follow.
• Best results are achieved for batch ingestion with 1000 records or more, upper boundary is 10.000 records.
• Continous Aggregates are not supported at the moment.

Optimize cooling data in the columnstore

As the data cools and becomes more suited for analytics, add a columnstore policy so your data is automatically converted to the columnstore after a specific time interval. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space. In the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads.

To optimize your data, add a columnstore policy:

CALL add_columnstore_policy('conditions', after => INTERVAL '1d');

You can also manually convert chunks in a hypertable to the columnstore.

Alter a hypertable

You can alter a hypertable, for example to add a column, by using the Postgres ALTER TABLE command. This works for both regular and distributed hypertables.

Add a column to a hypertable

You add a column to a hypertable using the ALTER TABLE command. In this example, the hypertable is named conditions and the new column is named humidity:

ALTER TABLE conditions
  ADD COLUMN humidity DOUBLE PRECISION NULL;

If the column you are adding has the default value set to NULL, or has no default value, then adding a column is relatively fast. If you set the default to a non-null value, it takes longer, because it needs to fill in this value for all existing rows of all existing chunks.

Rename a hypertable

You can change the name of a hypertable using the ALTER TABLE command. In this example, the hypertable is called conditions, and is being changed to the new name, weather:

ALTER TABLE conditions
  RENAME TO weather;

Drop a hypertable

Drop a hypertable using a standard Postgres DROP TABLE command:

DROP TABLE weather;

All data chunks belonging to the hypertable are deleted.

===== PAGE: https://docs.tigerdata.com/use-timescale/hypertables/improve-query-performance/ =====

Improve hypertable and query performance

Hypertables are Postgres tables that help you improve insert and query performance by automatically partitioning your data by time. Each hypertable is made up of child tables called chunks. Each chunk is assigned a range of time, and only contains data from that range. When you run a query, TimescaleDB identifies the correct chunk and runs the query on it, instead of going through the entire table. This page shows you how to tune hypertables to increase performance even more.

• Optimize hypertable chunk intervals: choose the optimum chunk size for your data
• Enable chunk skipping: skip chunks on non-partitioning columns in hypertables when you query your data
• Analyze your hypertables: use Postgres ANALYZE to create the best query plan

Optimize hypertable chunk intervals

Adjusting your hypertable chunk interval can improve performance in your database.

1. Choose an optimum chunk interval

   Postgres builds the index on the fly during ingestion. That means that to build a new entry on the index, a significant portion of the index needs to be traversed during every row insertion. When the index does not fit into memory, it is constantly flushed to disk and read back. This wastes IO resources which would otherwise be used for writing the heap/WAL data to disk.

The default chunk interval is 7 days. However, best practice is to set chunk_interval so that prior to processing, the indexes for chunks currently being ingested into fit within 25% of main memory. For example, on a system with 64 GB of memory, if index growth is approximately 2 GB per day, a 1-week chunk interval is appropriate. If index growth is around 10 GB per day, use a 1-day interval.

You set chunk_interval when you create a hypertable, or by calling set_chunk_time_interval on an existing hypertable.

In the following example you create a table called conditions that stores time values in the time column and has chunks that store data for a chunk_interval of one day:

CREATE TABLE conditions (
   time        TIMESTAMPTZ       NOT NULL,
   location    TEXT              NOT NULL,
   device      TEXT              NOT NULL,
   temperature DOUBLE PRECISION  NULL,
   humidity    DOUBLE PRECISION  NULL
) WITH (
   tsdb.hypertable,
   tsdb.partition_column='time',
   tsdb.chunk_interval='1 day'
);

If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

1. Check current setting for chunk intervals

   Query the TimescaleDB catalog for a hypertable. For example:

   SELECT *
     FROM timescaledb_information.dimensions
     WHERE hypertable_name = 'conditions';
   
   

   The result looks like:

   hypertable_schema | hypertable_name | dimension_number | column_name |       column_type        | dimension_type | time_interval | integer_interval | integer_now_func | num_partitions
   -------------------+-----------------+------------------+-------------+--------------------------+----------------+---------------+------------------+------------------+----------------
    public           | metrics          |                1 | recorded    | timestamp with time zone | Time           | 1 day         |                  |                  |
   

   Time-based interval lengths are reported in microseconds.

2. Change the chunk interval length on an existing hypertable

   To change the chunk interval on an already existing hypertable, call set_chunk_time_interval.

   SELECT set_chunk_time_interval('conditions', INTERVAL '24 hours');
   

   The updated chunk interval only applies to new chunks. This means setting an overly long interval might take a long time to correct. For example, if you set chunk_interval to 1 year and start inserting data, you can no longer shorten the chunk for that year. If you need to correct this situation, create a new hypertable and migrate your data.

   While chunk turnover does not degrade performance, chunk creation does take longer lock time than a normal INSERT operation into a chunk that has already been created. This means that if multiple chunks are being created at the same time, the transactions block each other until the first transaction is completed.

If you use expensive index types, such as some PostGIS geospatial indexes, take care to check the total size of the chunk and its index using chunks_detailed_size.

Enable chunk skipping

Early access: TimescaleDB v2.17.1

One of the key purposes of hypertables is to make your analytical queries run with the lowest latency possible. When you execute a query on a hypertable, you do not parse the whole table; you only access the chunks necessary to satisfy the query. This works well when the WHERE clause of a query uses the column by which a hypertable is partitioned. For example, in a hypertable where every day of the year is a separate chunk, a query for September 1 accesses only the chunk for that day.

However, many queries use columns other than the partitioning one. For example, a satellite company might have a table with two columns: one for when data was gathered by a satellite and one for when it was added to the database. If you partition by the date of gathering, a query by the date of adding accesses all chunks in the hypertable and slows the performance.

To improve query performance, TimescaleDB enables you to skip chunks on non-partitioning columns in hypertables.

Chunk skipping only works on chunks converted to the columnstore after you enable_chunk_skipping.

How chunk skipping works

You enable chunk skipping on a column in a hypertable. TimescaleDB tracks the minimum and maximum values for that column in each chunk. These ranges are stored in the start (inclusive) and end (exclusive) format in the chunk_column_stats catalog table. TimescaleDB uses these ranges for dynamic chunk exclusion when the WHERE clause of an SQL query specifies ranges on the column.

You can enable chunk skipping on hypertables compressed into the columnstore for smallint, int, bigint, serial, bigserial, date, timestamp, or timestamptz type columns.

When to enable chunk skipping

You can enable chunk skipping on as many columns as you need. However, best practice is to enable it on columns that are both:

• Correlated, that is, related to the partitioning column in some way.
• Referenced in the WHERE clauses of the queries.

In the satellite example, the time of adding data to a database inevitably follows the time of gathering. Sequential IDs and the creation timestamp for both entities also increase synchronously. This means those two columns are correlated.

For a more in-depth look on chunk skipping, see our blog post.

Enable chunk skipping

To enable chunk skipping on a column, call enable_chunk_skipping on a hypertable for a column_name. For example, the following query enables chunk skipping on the order_id column in the orders table:

SELECT enable_chunk_skipping('orders', 'order_id');

For more details on how to implement chunk skipping, see the API Reference.

Analyze your hypertables

You can use the Postgres ANALYZE command to query all chunks in your hypertable. The statistics collected by the ANALYZE command are used by the Postgres planner to create the best query plan. For more information about the ANALYZE command, see the Postgres documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/extensions/pgvector/ =====

Create a chatbot using pgvector

The pgvector Postgres extension helps you to store and search over machine learning-generated embeddings. It provides different capabilities that allows you to identify both exact and approximate nearest neighbors. It is designed to work seamlessly with other Postgres features, including indexing and querying.

For more information about these functions and the options available, see the pgvector repository.

Use the pgvector extension to create a chatbot

The pgvector Postgres extension allows you to create, store, and query OpenAI vector embeddings in a Postgres database instance. This page shows you how to use retrieval augmented generation (RAG) to create a chatbot that combines your data with ChatGPT using OpenAI and pgvector. RAG provides a solution to the problem that a foundational model such as GPT-3 or GPT-4 could be missing some information needed to give a good answer, because that information was not in the dataset used to train the model. This can happen if the information is stored in private documents or only became available recently.

In this example, you create embeddings, insert the embeddings into a Tiger Cloud service and query the embeddings using pgvector. The content for the embeddings is from the Tiger Data blog, specifically from the Developer Q&A section, which features posts by Tiger Data users talking about their real-world use cases.

Prerequisites

Before you begin, make sure you have:

• Installed Python.

• Created a Tiger Cloud service.

• Downloaded the cheatsheet when you created the service. This sheet contains the connection details for the database you want to use as a vector database.

• Cloned the pgvector repository.

• Signed up for an OpenAI developer account.

• Created an API key and made a note of your OpenAI API key.

  If you are on a free plan there may be rate limiting for your API requests.

Using the pgvector extension to create a chatbot

1. Create and activate a Python virtual environment:

   virtualenv pgvectorenv
   source pgvectorenv/bin/activate
   

2. Set the environment variables for OPENAI_API_KEY and TIMESCALE_CONNECTION_STRING. In this example, to set the environment variables in macOS, open the zshrc profile. Replace <OPENAI_API>, and <SERVICE_URL> with your OpenAI API key and the URL of your Tiger Cloud service:

   nano ~/.zshrc
   export OPENAI_API_KEY='<OPENAI_API>'
   export TIMESCALE_CONNECTION_STRING='<SERVICE_URL>'
   
   Update the shell with the new variables using `source ~/.zshrc`
   
   

3. Confirm that you have set the environment variables using:

   echo $OPENAI_API_KEY
   echo $TIMESCALE_CONNECTION_STRING
   

4. Install the required modules and packages using the requirements.txt. This file is located in the vector-cookbook\openai_pgvector_helloworld directory:

   pip install -r requirements.txt
   

5. To create embeddings for your data using the OpenAI API, open an editor of your choice and create the create_embeddings.py file.

   ###############################################################################
   ###############################################################################
   import openai
   import os
   import pandas as pd
   import numpy as np
   import json
   import tiktoken
   
   from dotenv import load_dotenv, find_dotenv
   _ = load_dotenv(find_dotenv())
   openai.api_key  = os.environ['OPENAI_API_KEY']
   
   df = pd.read_csv('blog_posts_data.csv')
   df.head()
   
   ###############################################################################
   ###############################################################################
   def num_tokens_from_string(string: str, encoding_name = "cl100k_base") -> int:
       if not string:
           return 0
       encoding = tiktoken.get_encoding(encoding_name)
       num_tokens = len(encoding.encode(string))
       return num_tokens
   
   def get_embedding_cost(num_tokens):
       return num_tokens/1000*0.0001
   
   def get_total_embeddings_cost():
       total_tokens = 0
       for i in range(len(df.index)):
           text = df['content'][i]
           token_len = num_tokens_from_string(text)
           total_tokens = total_tokens + token_len
       total_cost = get_embedding_cost(total_tokens)
       return total_cost
   ###############################################################################
   
   total_cost = get_total_embeddings_cost()
   print("Estimated price to embed this content = $" + str(total_cost))
   
   ###############################################################################
   ###############################################################################
   new_list = []
   for i in range(len(df.index)):
       text = df['content'][i]
       token_len = num_tokens_from_string(text)
       if token_len <= 512:
           new_list.append([df['title'][i], df['content'][i], df['url'][i], token_len])
       else:
           start = 0
           ideal_token_size = 512
           ideal_size = int(ideal_token_size // (4/3))
           end = ideal_size
           #split text by spaces into words
           words = text.split()
   
           #remove empty spaces
           words = [x for x in words if x != ' ']
   
           total_words = len(words)
   
           #calculate iterations
           chunks = total_words // ideal_size
           if total_words % ideal_size != 0:
               chunks += 1
   
           new_content = []
           for j in range(chunks):
               if end > total_words:
                   end = total_words
               new_content = words[start:end]
               new_content_string = ' '.join(new_content)
               new_content_token_len = num_tokens_from_string(new_content_string)
               if new_content_token_len > 0:
                   new_list.append([df['title'][i], new_content_string, df['url'][i], new_content_token_len])
               start += ideal_size
               end += ideal_size
   
   def get_embeddings(text):
      response = openai.Embedding.create(
          model="text-embedding-ada-002",
          input = text.replace("\n"," ")
      )
      embedding = response['data'][0]['embedding']
      return embedding
   
   for i in range(len(new_list)):
      text = new_list[i][1]
      embedding = get_embeddings(text)
      new_list[i].append(embedding)
   
   df_new = pd.DataFrame(new_list, columns=['title', 'content', 'url', 'tokens', 'embeddings'])
   df_new.head()
   
   df_new.to_csv('blog_data_and_embeddings.csv', index=False)
   
   print("Done! Check the file blog_data_and_embeddings.csv for your results.")
   

6. Run the script using the python create_embeddings.py command. You should see an output that looks a bit like this:

   Estimated price to embed this content = $0.0060178
   Done! Check the file blog_data_and_embeddings.csv for your results.
   

7. To insert these embeddings into your Tiger Cloud service using the pgvector extension, open an editor of your choice and create the insert_embeddings.py file.

   ###############################################################################
   ###############################################################################
   import openai
   import os
   import pandas as pd
   import numpy as np
   import psycopg2
   import ast
   import pgvector
   import math
   from psycopg2.extras import execute_values
   from pgvector.psycopg2 import register_vector
   
   ###############################################################################
   ###############################################################################
   connection_string  = os.environ['TIMESCALE_CONNECTION_STRING']
   
   conn = psycopg2.connect(connection_string)
   cur = conn.cursor()
   
   #install pgvector in your database
   cur.execute("CREATE EXTENSION IF NOT EXISTS vector;");
   conn.commit()
   
   register_vector(conn)
   table_create_command = """
   CREATE TABLE embeddings (
               id bigserial primary key,
               title text,
               url text,
               content text,
               tokens integer,
               embedding vector(1536)
               );
               """
   
   cur.execute(table_create_command)
   cur.close()
   conn.commit()
   ###############################################################################
   
   df = pd.read_csv('blog_data_and_embeddings.csv')
   titles = df['title']
   urls = df['url']
   contents = df['content']
   tokens = df['tokens']
   embeds = [list(map(float, ast.literal_eval(embed_str))) for embed_str in df['embeddings']]
   
   df_new = pd.DataFrame({
       'title': titles,
       'url': urls,
       'content': contents,
       'tokens': tokens,
       'embeddings': embeds
   })
   
   print(df_new.head())
   
   ###############################################################################
   ###############################################################################
   register_vector(conn)
   cur = conn.cursor()
   
   data_list = [(row['title'], row['url'], row['content'], int(row['tokens']), np.array(row['embeddings'])) for index, row in df_new.iterrows()]
   execute_values(cur, "INSERT INTO embeddings (title, url, content, tokens, embedding) VALUES %s", data_list)
   conn.commit()
   
   cur.execute("SELECT COUNT(*) as cnt FROM embeddings;")
   num_records = cur.fetchone()[0]
   print("Number of vector records in table: ", num_records,"\n")
   
   cur.execute("SELECT * FROM embeddings LIMIT 1;")
   records = cur.fetchall()
   print("First record in table: ", records)
   
   #calculate the index parameters according to best practices
   num_lists = num_records / 1000
   if num_lists < 10:
      num_lists = 10
   if num_records > 1000000:
      num_lists = math.sqrt(num_records)
   
   #use the cosine distance measure, which is what we'll later use for querying
   cur.execute(f'CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops) WITH (lists = {num_lists});')
   conn.commit()
   print("Index created on embeddings table")
   

8. Run the script using the python insert_embeddings.py command. You should see an output that looks a bit like this:

   0  How to Build a Weather Station With Elixir, Ne...  ...  [0.021399984136223793, 0.021850213408470154, -...
   1  How to Build a Weather Station With Elixir, Ne...  ...  [0.01620873250067234, 0.011362895369529724, 0....
   2  How to Build a Weather Station With Elixir, Ne...  ...  [0.022517921403050423, -0.0019158280920237303,...
   3  CloudQuery on Using Postgres for Cloud Asset...  ...  [0.008915113285183907, -0.004873732570558786, ...
   4  CloudQuery on Using PostgreSQL for Cloud Asset...  ...  [0.0204352755099535, 0.010087345726788044, 0.0...
   
   [5 rows x 5 columns]
   Number of vector records in table:  129
   
   First record in table:  [(1, 'How to Build a Weather Station With Elixir, Nerves, and TimescaleDB', 'https://www.timescale.com/blog/how-to-build-a-weather-station-with-elixir-nerves-and-timescaledb/', 'This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Alexander Koutmos, author of the Build a Weather Station with Elixir and Nerves book, joins us to share how he uses Grafana and TimescaleDB to store and visualize weather data collected from IoT sensors.About the teamThe bookBuild a Weather Station with Elixir and Nerveswas a joint effort between Bruce Tate, Frank Hunleth, and me.I have been writing software professionally for almost a decade and have been working primarily with Elixir since 2016. I currently maintain a few Elixir libraries onHexand also runStagira, a software consultancy company.Bruce Tateis a kayaker, programmer, and father of two from Chattanooga, Tennessee. He is the author of more than ten books and has been around Elixir from the beginning. He is the founder ofGroxio, a company that trains Elixir developers.Frank Hunlethis an embedded systems programmer, OSS maintainer, and Nerves core team member. When not in front of a computer, he loves running and spending time with his family.About the projectIn the Pragmatic Bookshelf book,Build a Weather Station with Elixir and Nerves, we take a project-based approach and guide the reader to create a Nerves-powered IoT weather station.For those unfamiliar with the Elixir ecosystem,Nervesis an IoT framework that allows you to build and deploy IoT applications on a wide array of embedded devices. At a high level, Nerves allows you to focus on building your project and takes care of a lot of the boilerplate associated with running Elixir on embedded devices.The goal of the book is to guide the reader through the process of building an end-to-end IoT solution for capturing, persisting, and visualizing weather data.Assembled weather station hooked up to development machine.One of the motivating factors for this book was to create a real-world project where readers could get hands-on experience with hardware without worrying too much about the nitty-gritty of soldering components together. Experimenting with hardware can often feel intimidating and confusing, but with Elixir and Nerves, we feel confident that even beginners get comfortable and productive quickly. As a result, in the book, we leverage a Raspberry Pi Zero W along with a few I2C enabled sensors to', 501, array([ 0.02139998,  0.02185021, -0.00537814, ..., -0.01257126,
      -0.02165324, -0.03714396], dtype=float32))]
   Index created on embeddings table
   

9. To query the embeddings that you inserted in to your Tiger Cloud service, open an editor of your choice and create the query_embeddings.py file. Here, the query is How does Density use TimescaleDB?.

   ###############################################################################
   ###############################################################################
   import openai
   import os
   import pandas as pd
   import numpy as np
   import json
   import tiktoken
   import psycopg2
   import ast
   import pgvector
   import math
   from psycopg2.extras import execute_values
   from pgvector.psycopg2 import register_vector
   
   from dotenv import load_dotenv, find_dotenv
   _ = load_dotenv(find_dotenv())
   openai.api_key  = os.environ['OPENAI_API_KEY']
   
   connection_string  = os.environ['TIMESCALE_CONNECTION_STRING']
   
   conn = psycopg2.connect(connection_string)
   
   ###############################################################################
   ###############################################################################
   def get_top3_similar_docs(query_embedding, conn):
       embedding_array = np.array(query_embedding)
       register_vector(conn)
       cur = conn.cursor()
       cur.execute("SELECT content FROM embeddings ORDER BY embedding <=> %s LIMIT 3", (embedding_array,))
       top3_docs = cur.fetchall()
       return top3_docs
   
   def get_completion_from_messages(messages, model="gpt-3.5-turbo-0613", temperature=0,   max_tokens=1000):
       response = openai.ChatCompletion.create(
           model=model,
           messages=messages,
           temperature=temperature,
           max_tokens=max_tokens,
       )
       return response.choices[0].message["content"]
   
   def get_embeddings(text):
       response = openai.Embedding.create(
           model="text-embedding-ada-002",
           input = text.replace("\n"," ")
       )
       embedding = response['data'][0]['embedding']
       return embedding
   ###############################################################################
   
   ###############################################################################
   ###############################################################################
   def process_input_with_retrieval(user_input):
       delimiter = "```"
   
       #Step 1: Get documents related to the user input from database
       related_docs = get_top3_similar_docs(get_embeddings(user_input), conn)
   
       system_message = f"""
       You are a friendly chatbot. \
       You can answer questions about timescaledb, its features and its use cases. \
       You respond in a concise, technically credible tone. \
       """
   
       messages = [
           {"role": "system", "content": system_message},
           {"role": "user", "content": f"{delimiter}{user_input}{delimiter}"},
           {"role": "assistant", "content": f"Relevant Tiger Data case studies information: \n {related_docs[0] [0]} \n {related_docs[1][0]} {related_docs[2][0]}"}
       ]
   
       final_response = get_completion_from_messages(messages)
       return final_response
   ###############################################################################
   
   input = "How does Density use TimescaleDB?"
   response = process_input_with_retrieval(input)
   print(input)
   print(response)
   

10. Run the script using the python query_embeddings.py command. You should see an output that looks a bit like this:

    How does Density use TimescaleDB?
    Density uses TimescaleDB as the main database in their smart city system.
    They store counts of people in spaces over time and derive metrics such as dwell time and space usage.
    TimescaleDB's flexibility and ability to handle time-series data efficiently allows Density to slice, dice, and compose queries in various ways.
    They also leverage TimescaleDB's continuous aggregates feature to roll up high-resolution data to lower resolutions, improving query performance.
    Additionally, TimescaleDB's support for percentile calculations has helped Density deliver accurate percentile values for their data.
    Overall, TimescaleDB has significantly improved the performance and scalability of Density's analytics workload.
    

===== PAGE: https://docs.tigerdata.com/use-timescale/extensions/pgcrypto/ =====

Encrypt data using pgcrypto

The pgcrypto Postgres extension provides cryptographic functions such as:

• General hashing
• Password hashing
• PGP encryption
• Raw encryption
• Random-data

For more information about these functions and the options available, see the pgcrypto documentation.

Use the pgcrypto extension to encrypt inserted data

The pgcrypto extension allows you to encrypt, decrypt, hash, and create digital signatures within your database. Tiger Data understands how precious your data is and safeguards sensitive information.

Using the pgcrypto extension to encrypt inserted data

1. Install the pgcrypto extension:

   CREATE EXTENSION IF NOT EXISTS pgcrypto;
   

2. You can confirm if the extension is installed using the \dx command. The installed extensions are listed:

       List of installed extensions
           Name         | Version |   Schema   |                                      Description
   ---------------------+---------+------------+---------------------------------------------------------------------------------------
    pg_stat_statements  | 1.10    | public     | track planning and execution statistics of all SQL statements executed
    pgcrypto            | 1.3     | public     | cryptographic functions
    plpgsql             | 1.0     | pg_catalog | PL/pgSQL procedural language
    timescaledb         | 2.11.0  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
    timescaledb_toolkit | 1.16.0  | public     | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities
   

3. Create a table named user_passwords:

   CREATE TABLE user_passwords (username varchar(100) PRIMARY KEY, crypttext text);
   

4. Insert the values in the user_passwords table and replace <Password_Key> with a password key of your choice:

      INSERT INTO tbl_sym_crypt (username, crypttext)
       VALUES ('user1', pgp_sym_encrypt('user1_password','<Password_Key>')),
          ('user2', pgp_sym_encrypt('user2_password','<Password_Key>'));
   

5. You can confirm that the password is encrypted using the command:

   SELECT * FROM user_passwords;
   

   The encrypted passwords are listed:

          username |                                                                              crypttext
   ----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------
    user1      | \xc30d040703025caa37f9d1c731d169d240018529d6f0002b2948905a87e4787efaa0046e58fd3f04ee95594bea1803807063321f62c9651cbf0422b04508093df9644a76684b504b317cf633552fcf164f
    user2   | \xc30d0407030279bbcf760b81d3de73d23c01c04142632fc8527c0c1b17cc954c77f16df46022acddc565fd18f0f0f761ddb2f31b21c4ebe47a48039d685287d64506029e027cf29b5493b574df
   (2 rows)
   

6. To view the decrypted passwords, replace <Password_Key> with the password key that you created:

    SELECT username, pgp_sym_decrypt(crypttext::bytea, '<Password_Key>')
    FROM user_passwords;
   

   The decrypted passwords are listed:

    username | pgp_sym_decrypt
    ----------+-----------------
     user1   | user1_password
     user2   | user2_password
    (2 rows)
   

===== PAGE: https://docs.tigerdata.com/use-timescale/extensions/postgis/ =====

Analyse geospatial data with postgis

The postgis Postgres extension provides storing, indexing, and querying geographic data. It helps in spatial data analysis, the study of patterns, anomalies, and theories within spatial or geographical data.

For more information about these functions and the options available, see the [PostGIS documentation] postgis-docs.

Use the postgis extension to analyze geospatial data

The postgis Postgres extension allows you to conduct complex analyses of your geospatial time-series data. Tiger Data understands that you have a multitude of data challenges and helps you discover when things happened, and where they occurred. In this example you can query when the covid cases were reported, where they were reported, and how many were reported around a particular location.

Using the postgis extension to analyze geospatial data

1. Install the postgis extension:

   CREATE EXTENSION postgis;
   

2. You can confirm if the extension is installed using the \dx command. The extensions that are installed are listed:

                                       List of installed extensions
   Name         | Version |   Schema   |                                      Description
   ---------------------+---------+------------+---------------------------------------------------------------------------------------
    pg_stat_statements  | 1.10    | public     | track planning and execution statistics of all SQL statements executed
    pgcrypto            | 1.3     | public     | cryptographic functions
    plpgsql             | 1.0     | pg_catalog | PL/pgSQL procedural language
    postgis             | 3.3.3   | public     | PostGIS geometry and geography spatial types and functions
    timescaledb         | 2.11.0  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
    timescaledb_toolkit | 1.16.0  | public     | Library of analytical hyperfunctions,     time-series pipelining, and other SQL utilities
   (6 rows)
   

3. Create a hypertable named covid_location, where, location is a GEOGRAPHY type column that stores GPS coordinates using the 4326/WGS84 coordinate system, and time records the time the GPS coordinate was logged for a specific state_id. This hypertable is partitioned on the time column:

   CREATE TABLE covid_location (
     time TIMESTAMPTZ NOT NULL,
     state_id INT NOT NULL,
     location GEOGRAPHY(POINT, 4326),
     cases INT NOT NULL,
     deaths INT NOT NULL
   ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='time'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

4. To support efficient queries, create an index on the state_id column:

   CREATE INDEX ON covid_location (state_id, time DESC);
   

5. Insert some randomly generated values in the covid_location table. The longitude and latitude coordinates of New Jersey are (-73.935242 40.730610), and New York are (-74.871826 39.833851):

   INSERT INTO covid_location VALUES
   ('2023-06-28 20:00:00',34,'POINT(-74.871826 39.833851)',5,2),
   ('2023-06-28 20:00:00',36,'POINT(-73.935242 40.730610)',7,1),
   ('2023-06-29 20:00:00',34,'POINT(-74.871826 39.833851)',14,0),
   ('2023-06-29 20:00:00',36,'POINT(-73.935242 40.730610)',12,1),
   ('2023-06-30 20:00:00',34,'POINT(-74.871826 39.833851)',10,4);
   

6. To fetch all cases of a specific state during a specific period, use:

   SELECT * FROM covid_location
   WHERE state_id = 34 AND time BETWEEN '2023-06-28 00:00:00' AND '2023-06-30 23:59:59';
   

   The data you get back looks a bit like this:

                        time          | state_id |                      location                      | cases | deaths
   ------------------------+----------+----------------------------------------------------+-------+--------
    2023-06-28 20:00:00+00 |       34 | 0101000020E61000005C7347FFCBB752C0535E2BA1BBEA4340 |     5 |      2
    2023-06-29 20:00:00+00 |       34 | 0101000020E61000005C7347FFCBB752C0535E2BA1BBEA4340 |    14 |      0
    2023-06-30 20:00:00+00 |       34 | 0101000020E61000005C7347FFCBB752C0535E2BA1BBEA4340 |    10 |      4
   (3 rows)
   

7. To fetch the latest logged cases of all states using the Tiger Data SkipScan feature, replace <Interval_Time> with the number of days between the day you are running the query and the day the last report was logged in the table, in this case 30, June, 2023:

   SELECT DISTINCT ON (state_id) state_id, ST_AsText(location) AS location
   FROM covid_location
   WHERE time > now() - INTERVAL '<Interval_Time>'
   ORDER BY state_id,
   time DESC;
   

   The ST_AsText(location) function converts the binary geospatial data into human-readable format. The data you get back looks a bit like this:

   state_id |          location
   ----------+-----------------------------
   34 | POINT(-74.871826 39.833851)
   (1 row)
   

8. To fetch all cases and states that were within 10000 meters of Manhattan at any time:

   SELECT DISTINCT cases, state_id
   FROM covid_location
   WHERE ST_DWithin(
   location,
   ST_GeogFromText('POINT(-73.9851 40.7589)'),
   10000
   );
   

   The data you get back looks a bit like this:

   cases | state_id
   -------+----------
    7 |       36
   12 |       36
   (2 rows)
   

===== PAGE: https://docs.tigerdata.com/use-timescale/extensions/pg-textsearch/ =====

Optimize full text search with BM25

Postgres full-text search at scale consistently hits a wall where performance degrades catastrophically. Tiger Data's pg_textsearch brings modern BM25-based full-text search directly into Postgres, with a memtable architecture for efficient indexing and ranking. pg_textsearch integrates seamlessly with SQL and provides better search quality and performance than the Postgres built-in full-text search.

BM25 scores in pg_textsearch are returned as negative values, where lower (more negative) numbers indicate better matches. pg_textsearch implements the following:

• Corpus-aware ranking: BM25 uses inverse document frequency to weight rare terms higher
• Term frequency saturation: prevents documents with excessive term repetition from dominating results
• Length normalization: adjusts scores based on document length relative to corpus average
• Relative ranking: focuses on rank order rather than absolute score values

This page shows you how to install pg_textsearch, configure BM25 indexes, and optimize your search capabilities using the following best practice:

• Memory planning: size your index_memory_limit based on corpus vocabulary and document count
• Language configuration: choose appropriate text search configurations for your data language
• Hybrid search: combine with pgvector or pgvectorscale for applications requiring both semantic and keyword search
• Query optimization: use score thresholds to filter low-relevance results
• Index monitoring: regularly check index usage and memory consumption

Early access: October 2025 this preview release is designed for development and staging environments. It is not recommended for use with hypertables.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with the Real-time analytics capability.

  You need your connection details. This procedure also works for self-hosted TimescaleDB.

Install pg_textsearch

To install this Postgres extension:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Enable the extension on your Tiger Cloud service

   • For new services, simply enable the extension:

     CREATE EXTENSION pg_textsearch;
     

   • For existing services, update your instance, then enable the extension:

     The extension may not be available until after your next scheduled maintenance window. To pick up the update immediately, manually pause and restart your service.

3. Verify the installation

   SELECT * FROM pg_extension WHERE extname = 'pg_textsearch';
   

You have installed pg_textsearch on Tiger Cloud.

Create BM25 indexes on your data

BM25 indexes provide modern relevance ranking that outperforms Postgres's built-in ts_rank functions by using corpus statistics and better algorithmic design.

To create a BM25 index with pg_textsearch:

1. Create a table with text content

   CREATE TABLE products (
       id serial PRIMARY KEY,
       name text,
       description text,
       category text,
       price numeric
   );
   

2. Insert sample data

   INSERT INTO products (name, description, category, price) VALUES
   ('Mechanical Keyboard', 'Durable mechanical switches with RGB backlighting for gaming and productivity', 'Electronics', 149.99),
   ('Ergonomic Mouse', 'Wireless mouse with ergonomic design to reduce wrist strain during long work sessions', 'Electronics', 79.99),
   ('Standing Desk', 'Adjustable height desk for better posture and productivity throughout the workday', 'Furniture', 599.99);
   

3. Create a BM25 index

   CREATE INDEX products_search_idx ON products
   USING bm25(description)
   WITH (text_config='english');
   

   BM25 supports single-column indexes only.

You have created a BM25 index for full-text search.

Optimize search queries for performance

Use efficient query patterns to leverage BM25 ranking and optimize search performance.

1. Perform ranked searches using the distance operator

   SELECT name, description,
          description <@> to_bm25query('ergonomic work', 'products_search_idx') as score
   FROM products
   ORDER BY description <@> to_bm25query('ergonomic work', 'products_search_idx')
   LIMIT 3;
   

2. Filter results by score threshold

   SELECT name,
          description <@> to_bm25query('wireless', 'products_search_idx') as score
   FROM products
   WHERE description <@> to_bm25query('wireless', 'products_search_idx') < -2.0;
   

3. Combine with standard SQL operations

   SELECT category, name,
          description <@> to_bm25query('ergonomic', 'products_search_idx') as score
   FROM products
   WHERE price < 500
     AND description <@> to_bm25query('ergonomic', 'products_search_idx') < -1.0
   ORDER BY description <@> to_bm25query('ergonomic', 'products_search_idx')
   LIMIT 5;
   

4. Verify index usage with EXPLAIN

   EXPLAIN SELECT * FROM products
   ORDER BY description <@> to_bm25query('wireless keyboard', 'products_search_idx')
   LIMIT 5;
   

You have optimized your search queries for BM25 ranking.

Build hybrid search with semantic and keyword search

Combine pg_textsearch with pgvector or pgvectorscale to build powerful hybrid search systems that use both semantic vector search and keyword BM25 search.

1. Enable the vectorscale extension on your Tiger Cloud service

    CREATE EXTENSION IF NOT EXISTS vectorscale CASCADE;
   

2. Create a table with both text content and vector embeddings

   CREATE TABLE articles (
       id serial PRIMARY KEY,
       title text,
       content text,
       embedding vector(1536)  -- OpenAI ada-002 embedding dimension
   );
   

3. Create indexes for both search types

   -- Vector index for semantic search
   CREATE INDEX articles_embedding_idx ON articles
   USING hnsw (embedding vector_cosine_ops);
   
   -- Keyword index for BM25 search
   CREATE INDEX articles_content_idx ON articles
   USING bm25(content)
   WITH (text_config='english');
   

4. Perform hybrid search using reciprocal rank fusion

   WITH vector_search AS (
     SELECT id,
            ROW_NUMBER() OVER (ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector) AS rank
     FROM articles
     ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector
     LIMIT 20
   ),
   keyword_search AS (
     SELECT id,
            ROW_NUMBER() OVER (ORDER BY content <@> to_bm25query('query performance', 'articles_content_idx')) AS rank
     FROM articles
     ORDER BY content <@> to_bm25query('query performance', 'articles_content_idx')
     LIMIT 20
   )
   SELECT a.id,
          a.title,
          COALESCE(1.0 / (60 + v.rank), 0.0) + COALESCE(1.0 / (60 + k.rank), 0.0) AS combined_score
   FROM articles a
   LEFT JOIN vector_search v ON a.id = v.id
   LEFT JOIN keyword_search k ON a.id = k.id
   WHERE v.id IS NOT NULL OR k.id IS NOT NULL
   ORDER BY combined_score DESC
   LIMIT 10;
   

5. Adjust relative weights for different search types

     WITH vector_search AS (
     SELECT id,
            ROW_NUMBER() OVER (ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector) AS rank
     FROM articles
     ORDER BY embedding <=> '[0.1, 0.2, 0.3]'::vector
     LIMIT 20
   ),
   keyword_search AS (
     SELECT id,
            ROW_NUMBER() OVER (ORDER BY content <@> to_bm25query('query performance', 'articles_content_idx')) AS rank
     FROM articles
     ORDER BY content <@> to_bm25query('query performance', 'articles_content_idx')
     LIMIT 20
   )
   SELECT
       a.id,
       a.title,
       0.7 * COALESCE(1.0 / (60 + v.rank), 0.0) +  -- 70% weight to vectors
       0.3 * COALESCE(1.0 / (60 + k.rank), 0.0)    -- 30% weight to keywords
   AS combined_score
   FROM articles a
   LEFT JOIN vector_search v ON a.id = v.id
   LEFT JOIN keyword_search k ON a.id = k.id
   WHERE v.id IS NOT NULL OR k.id IS NOT NULL
   ORDER BY combined_score DESC
   LIMIT 10;
   

You have implemented hybrid search combining semantic and keyword search.

Configuration options

Customize pg_textsearch behavior for your specific use case and data characteristics.

1. Configure the memory limit

   The size of the memtable depends primarily on the number of distinct terms in your corpus. A corpus with longer documents or more varied vocabulary requires more memory per document.

   -- Set memory limit per index (default 64MB)
   SET pg_textsearch.index_memory_limit = '128MB';
   

2. Configure language-specific text processing

   -- French language configuration
   CREATE INDEX products_fr_idx ON products_fr
   USING pg_textsearch(description)
   WITH (text_config='french');
   
   -- Simple tokenization without stemming
   CREATE INDEX products_simple_idx ON products
   USING pg_textsearch(description)
   WITH (text_config='simple');
   

3. Tune BM25 parameters

   -- Adjust term frequency saturation (k1) and length normalization (b)
   CREATE INDEX products_custom_idx ON products
   USING bm25(description)
   WITH (text_config='english', k1=1.5, b=0.8);
   

   1. Monitor index usage and memory consumption

      • Check index usage statistics

        SELECT schemaname, relname, indexrelname, idx_scan, idx_tup_read
        FROM pg_stat_user_indexes
        WHERE indexrelid::regclass::text ~ 'bm25';
        

      • View detailed index information

        SELECT bm25_debug_dump_index('products_search_idx');
        

You have configured pg_textsearch for optimal performance. For production applications, consider implementing result caching and pagination to improve user experience with large result sets.

Current limitations

This preview release focuses on core BM25 functionality. It has the following limitations:

• Memory-only storage: indexes are limited by pg_textsearch.index_memory_limit (default 64MB)
• No phrase queries: cannot search for exact multi-word phrases yet

These limitations will be addressed in upcoming releases with disk-based segments and expanded query capabilities.

===== PAGE: https://docs.tigerdata.com/use-timescale/metrics-logging/datadog/ =====

Export metrics to Datadog

You can export telemetry data from your Tiger Cloud services with the time-series and analytics capability enabled to Datadog. The available metrics include CPU usage, RAM usage, and storage. This integration is available for Scale or Enterprise pricing plans.

This page shows you how to create a Datadog exporter in Tiger Cloud Console, and manage the lifecycle of data exporters.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

Create a data exporter

Tiger Cloud data exporters send telemetry data from a Tiger Cloud service to third-party monitoring tools. You create an exporter on the project level, in the same AWS region as your service:

1. In Tiger Cloud Console, open Exporters

2. Click New exporter

3. Select Metrics for Data type and Datadog for provider

   

4. Choose your AWS region and provide the API key

   The AWS region must be the same for your Tiger Cloud exporter and the Datadog provider.

5. Set Site to your Datadog region, then click Create exporter

Manage a data exporter

This section shows you how to attach, monitor, edit, and delete a data exporter.

Attach a data exporter to a Tiger Cloud service

To send telemetry data to an external monitoring tool, you attach a data exporter to your Tiger Cloud service. You can attach only one exporter to a service.

To attach an exporter:

1. In Tiger Cloud Console, choose the service
2. Click Operations > Exporters
3. Select the exporter, then click Attach exporter
4. If you are attaching a first Logs data type exporter, restart the service

Monitor Tiger Cloud service metrics

You can now monitor your service metrics. Use the following metrics to check the service is running correctly:

• timescale.cloud.system.cpu.usage.millicores
• timescale.cloud.system.cpu.total.millicores
• timescale.cloud.system.memory.usage.bytes
• timescale.cloud.system.memory.total.bytes
• timescale.cloud.system.disk.usage.bytes
• timescale.cloud.system.disk.total.bytes

Additionally, use the following tags to filter your results.

Tag	Example variable	Description
host	us-east-1.timescale.cloud
project-id
service-id
region	us-east-1	AWS region
role	replica or primary	For service with replicas
node-id		For multi-node services

Edit a data exporter

To update a data exporter:

1. In Tiger Cloud Console, open Exporters
2. Next to the exporter you want to edit, click the menu > Edit
3. Edit the exporter fields and save your changes

You cannot change fields such as the provider or the AWS region.

Delete a data exporter

To remove a data exporter that you no longer need:

1. Disconnect the data exporter from your Tiger Cloud services

   1. In Tiger Cloud Console, choose the service.
   2. Click Operations > Exporters.
   3. Click the trash can icon.
   4. Repeat for every service attached to the exporter you want to remove.

   The data exporter is now unattached from all services. However, it still exists in your project.

2. Delete the exporter on the project level

   1. In Tiger Cloud Console, open Exporters
   2. Next to the exporter you want to edit, click menu > Delete
   3. Confirm that you want to delete the data exporter.

Reference

When you create the IAM OIDC provider, the URL must match the region you create the exporter in. It must be one of the following:

Region	Zone	Location	URL
ap-southeast-1	Asia Pacific	Singapore	irsa-oidc-discovery-prod-ap-southeast-1.s3.ap-southeast-1.amazonaws.com
ap-southeast-2	Asia Pacific	Sydney	irsa-oidc-discovery-prod-ap-southeast-2.s3.ap-southeast-2.amazonaws.com
ap-northeast-1	Asia Pacific	Tokyo	irsa-oidc-discovery-prod-ap-northeast-1.s3.ap-northeast-1.amazonaws.com
ca-central-1	Canada	Central	irsa-oidc-discovery-prod-ca-central-1.s3.ca-central-1.amazonaws.com
eu-central-1	Europe	Frankfurt	irsa-oidc-discovery-prod-eu-central-1.s3.eu-central-1.amazonaws.com
eu-west-1	Europe	Ireland	irsa-oidc-discovery-prod-eu-west-1.s3.eu-west-1.amazonaws.com
eu-west-2	Europe	London	irsa-oidc-discovery-prod-eu-west-2.s3.eu-west-2.amazonaws.com
sa-east-1	South America	São Paulo	irsa-oidc-discovery-prod-sa-east-1.s3.sa-east-1.amazonaws.com
us-east-1	United States	North Virginia	irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com
us-east-2	United States	Ohio	irsa-oidc-discovery-prod-us-east-2.s3.us-east-2.amazonaws.com
us-west-2	United States	Oregon	irsa-oidc-discovery-prod-us-west-2.s3.us-west-2.amazonaws.com

===== PAGE: https://docs.tigerdata.com/use-timescale/metrics-logging/metrics-to-prometheus/ =====

Export metrics to Prometheus

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, and a modern alerting approach.

This page shows you how to export your service telemetry to Prometheus:

• For Tiger Cloud, using a dedicated Prometheus exporter in Tiger Cloud Console.
• For self-hosted TimescaleDB, using Postgres Exporter.

Prerequisites

To follow the steps on this page:

• Download and run Prometheus.

• For Tiger Cloud:

  Create a target Tiger Cloud service with the time-series and analytics capability enabled.

• For self-hosted TimescaleDB:

  • Create a target self-hosted TimescaleDB instance. You need your connection details.
  • Install Postgres Exporter. To reduce latency and potential data transfer costs, install Prometheus and Postgres Exporter on a machine in the same AWS region as your Tiger Cloud service.

Export Tiger Cloud service telemetry to Prometheus

To export your data, do the following:

To export metrics from a Tiger Cloud service, you create a dedicated Prometheus exporter in Tiger Cloud Console, attach it to your service, then configure Prometheus to scrape metrics using the exposed URL. The Prometheus exporter exposes the metrics related to the Tiger Cloud service like CPU, memory, and storage. To scrape other metrics, use Postgres Exporter as described for self-hosted TimescaleDB. The Prometheus exporter is available for Scale and Enterprise pricing plans.

1. Create a Prometheus exporter

   1. In Tiger Cloud Console, click Exporters > + New exporter.

   2. Select Metrics for data type and Prometheus for provider.

      

   3. Choose the region for the exporter. Only services in the same project and region can be attached to this exporter.

   4. Name your exporter.

   5. Change the auto-generated Prometheus credentials, if needed. See official documentation on basic authentication in Prometheus.

2. Attach the exporter to a service

   1. Select a service, then click Operations > Exporters.

   2. Select the exporter in the drop-down, then click Attach exporter.

      

   The exporter is now attached to your service. To unattach it, click the trash icon in the exporter list.

   

3. Configure the Prometheus scrape target

   1. Select your service, then click Operations > Exporters and click the information icon next to the exporter. You see the exporter details.

      

   2. Copy the exporter URL.

   3. In your Prometheus installation, update prometheus.yml to point to the exporter URL as a scrape target:

      scrape_configs:
       - job_name: "timescaledb-exporter"
         scheme: https
         static_configs:
           - targets: ["my-exporter-url"]
         basic_auth:
           username: "user"
           password: "pass"
      

      See the Prometheus documentation for details on configuring scrape targets.

      You can now monitor your service metrics. Use the following metrics to check the service is running correctly:

      • timescale.cloud.system.cpu.usage.millicores
      • timescale.cloud.system.cpu.total.millicores
      • timescale.cloud.system.memory.usage.bytes
      • timescale.cloud.system.memory.total.bytes
      • timescale.cloud.system.disk.usage.bytes
      • timescale.cloud.system.disk.total.bytes

      Additionally, use the following tags to filter your results.

      Tag	Example variable	Description
      host	us-east-1.timescale.cloud
      project-id
      service-id
      region	us-east-1	AWS region
      role	replica or primary	For service with replicas

To export metrics from self-hosted TimescaleDB, you import telemetry data about your database to Postgres Exporter, then configure Prometheus to scrape metrics from it. Postgres Exporter exposes metrics that you define, excluding the system metrics.

1. Create a user to access telemetry data about your database

   1. Connect to your database in psql using your connection details.

   2. Create a user named monitoring with a secure password:

      CREATE USER monitoring WITH PASSWORD '<password>';
      

   3. Grant the pg_read_all_stats permission to the monitoring user:

      GRANT pg_read_all_stats to monitoring;
      

2. Import telemetry data about your database to Postgres Exporter

   1. Connect Postgres Exporter to your database:

      Use your connection details to import telemetry data about your database. You connect as the monitoring user:

      • Local installation:

        export DATA_SOURCE_NAME="postgres://<user>:<password>@<host>:<port>/<database>?sslmode=<sslmode>"
        ./postgres_exporter
        

      • Docker:

        docker run -d \
           -e DATA_SOURCE_NAME="postgres://<user>:<password>@<host>:<port>/<database>?sslmode=<sslmode>" \
           -p 9187:9187 \
           prometheuscommunity/postgres-exporter
        

   2. Check the metrics for your database in the Prometheus format:

      • Browser:

        Navigate to http://<exporter-host>:9187/metrics.

      • Command line:

        curl http://<exporter-host>:9187/metrics
        

3. Configure Prometheus to scrape metrics

   1. In your Prometheus installation, update prometheus.yml to point to your Postgres Exporter instance as a scrape target. In the following example, you replace <exporter-host> with the hostname or IP address of the PostgreSQL Exporter.

      global:
        scrape_interval: 15s
      
      scrape_configs:
      - job_name: 'postgresql'
        static_configs:
         - targets: ['<exporter-host>:9187']
      

      If prometheus.yml has not been created during installation, create it manually. If you are using Docker, you can find the IPAddress in Inspect > Networks for the container running Postgres Exporter.

   2. Restart Prometheus.

   3. Check the Prometheus UI at http://<prometheus-host>:9090/targets and http://<prometheus-host>:9090/tsdb-status.

      You see the Postgres Exporter target and the metrics scraped from it.

You can further visualize your data with Grafana. Use the Grafana Postgres dashboard or create a custom dashboard that suits your needs.

===== PAGE: https://docs.tigerdata.com/use-timescale/metrics-logging/monitoring/ =====

Monitor your Tiger Cloud services

Get complete visibility into your service performance with Tiger Cloud's powerful monitoring suite. Whether you're optimizing for peak efficiency or troubleshooting unexpected behavior, Tiger Cloud gives you the tools to quickly identify and resolve issues.

When something doesn't look right, Tiger Cloud provides a complete investigation workflow:

1. Pinpoint the bottleneck: check Metrics to identify exactly when CPU, memory, or storage spiked.
2. Find the root cause: review Logs for errors or warnings that occurred during the incident.
3. Identify the culprit: examine Insights to see which queries were running at that time and how they impacted resources.
4. Check background activity: look at Jobs to see if scheduled tasks triggered the issue.
5. Investigate active connections: use Connections to see what clients were connected and what queries they were running.

Want to save some time? Check out Recommendations for alerts that may have already flagged the problem!

This pages explains what specific data you get at each point.

Metrics

Tiger Cloud shows you CPU, memory, and storage metrics for up to 30 previous days and with down to 10-second granularity. To access metrics, select your service in Tiger Cloud Console, then click Monitoring > Metrics:

The following metrics are represented by graphs:

• CPU, in mCPU
• Memory, in GiB
• Storage used, in GiB
• Storage I/O, in ops/sec
• Storage bandwidth, in MiB/sec

The Free pricing plan only includes storage metrics.

When you hit the limits:

• For CPU and memory: provision more for your service in Operations > Compute and storage.
• For storage, I/O, and bandwidth: these resources depend on your storage type and I/O boost settings. The standard high-performance storage gives you 16TB of compressed data on a single server, regardless of the number of hypertables in your service. See About storage tiers for how to change the available storage, I/O, and bandwidth.

Hover over the graph to view metrics for a specific time point. Select an area in the graph to zoom into a specific period.

Gray bars indicate that metrics have not been collected for the period shown:

Understand high memory usage

It is normal to observe high overall memory usage for your Tiger Cloud services, especially for workloads with active read and write. Tiger Cloud service run on Linux, and high memory usage is a particularity of the Linux page cache. The Linux kernel stores file-backed data in memory to speed up read operations. Postgres, and by extension, Tiger Cloud services rely heavily on disk I/O to access tables, WALs, and indexes. When your service reads these files, the kernel caches them in memory to improve performance for future access.

Page cache entries are not locked memory: they are evictable and are automatically reclaimed by the kernel when actual memory pressure arises. Therefore, high memory usage shown in the monitoring dashboards is often not due to service memory allocation, but the beneficial caching behavior in the Linux kernel. The trick is to distinguish between normal memory utilization and memory pressure.

High memory usage does not necessarily mean a problem, especially on read replicas or after periods of activity. For a more accurate view of database memory consumption, look at Postgres-specific metrics, such as shared_buffers or memory context breakdowns. Only take action if you see signs of real memory pressure—such as OOM (Out Of Memory) events or degraded performance.

Service states

Tiger Cloud Console gives you a visual representation of the state of your service. The following states are represented with the following colors:

State	Color
Configuring	Yellow
Deleted	Yellow
Deleting	Yellow
Optimizing	Green
Paused	Grey
Pausing	Grey
Queued	Yellow
Ready	Green
Resuming	Yellow
Unstable	Yellow
Upgrading	Yellow
Read-only	Red

Logs

Tiger Cloud shows you detailed logs for your service, which you can filter by type, date, and time.

To access logs, select your service in Tiger Cloud Console, then click Monitoring > Logs:

Insights

Insights help you get a comprehensive understanding of how your queries perform over time, and make the most efficient use of your resources.

To view insights, select your service, then click Monitoring > Insights. Search or filter queries by type, maximum execution time, and time frame.

Insights include Metrics, Current lock contention, and Queries.

Metrics provides a visual representation of CPU, memory, and storage input/output usage over time. It also overlays the execution times of the top three queries matching your search. This helps correlate query executions with resource utilization. Select an area of the graph to zoom into a specific time frame.

Current lock contention shows how many queries or transactions are currently waiting for locks held by other queries or transactions.

Queries displays the top 50 queries matching your search. This includes executions, total rows, total time, median time, P95 time, related hypertables, tables in the columnstore, and user name.

Column	Description
Executions	The number of times the query ran during the selected period.
Total rows	The total number of rows scanned, inserted, or updated by the query during the selected period.
Total time	The total time of query execution.
Median time	The median (P50) time of query execution.
P95 time	The ninety-fifth percentile, or the maximum time of query execution.
Hypertables	If the query ran on a hypertable.
Columnar tables	If the query drew results from a chunk in the columnstore.
User name	The user name of the user running the query.

These metrics calculations are based on the entire period you've selected. For example, if you've selected six hours, all the metrics represent an aggregation of the previous six hours of executions.

If you have just completed a query, it can take some minutes for it to show in the table. Wait a little, then refresh the page to see your query. Check out the last update value at the top of the query table to identify the timestamp from the last processed query stat.

Click a query in the list to see the drill-down view. This view not only helps you identify spikes and unexpected behaviors, but also offers information to optimize your query.

This view includes the following graphs:

• Execution time: the median and P95 query execution times over the selected period. This is useful to understand the consistency and efficiency of your query's execution over time.
• EXPLAIN plan: for queries that take more than 10 seconds to execute, there is an EXPLAIN plan collected automatically.
• Rows: the impact of your query on rows over time. If it's a SELECT statement, it shows the number of rows retrieved, while for an INSERT/UPDATE statement, it reflects the rows inserted.
• Plans and executions: the number of query plans and executions over time. You can use this to optimize query performance, helping you assess if you can benefit from prepared statements to reduce planning overhead.
• Shared buffers hit and miss: shared buffers play a critical role in Postgres's performance by caching data in memory. A shared buffer hit occurs when the required data block is found in the shared buffer memory, while a miss indicates that Postgres couldn't locate the block in memory. A miss doesn't necessarily mean a disk read, because Postgres may retrieve the data from the operating system's disk pages cache. If you observe a high number of shared buffer misses, your current shared buffers setting might be insufficient. Increasing the shared buffer size can improve cache hit rates and query speed.
• Cache hit ratio: measures how much of your query's data is read from shared buffers. A 100% value indicates that all the data required by the query was found in the shared buffer, while a 0% value means none of the necessary data blocks were in the shared buffers. This metric provides a clear understanding of how efficiently your query leverages shared buffers, helping you optimize data access and database performance.

Jobs

Tiger Cloud summarizes all jobs set up for your service along with their details like type, target object, and status. This includes native Tiger Cloud jobs as well as custom jobs you configure based on your specific needs.

1. To view jobs, select your service in Tiger Cloud Console, then click Monitoring > Jobs:

   

2. Click a job ID in the list to view its config and run history:

   

3. Click the pencil icon to edit the job config:

   

Connections

Tiger Cloud lists current and past connections to your service. This includes details like the corresponding query, connecting application, username, connection status, start time, and duration.

To view connections, select your service in Tiger Cloud Console, then click Monitoring > Connections. Expand the query underneath each connection to see the full SQL.

Click the trash icon next to a connection in the list to terminate it. A lock icon means that a connection cannot be terminated; hover over the icon to see the reason.

Recommendations

Tiger Cloud offers specific tips on configuring your service. This includes a wide range of actions—from finishing account setup to tuning your service for the best performance. For example, Tiger Cloud may recommend a more suitable chunk interval or draw your attention to consistently failing jobs.

To view recommendations, select your service in Tiger Cloud Console, then click Monitoring > Recommendations:

Query-level statistics with pg_stat_statements

<Availability products={['cloud', 'self_hosted', 'mst']} />

You can also get query-level statistics for your services with the pg_stat_statements extension. This includes the time spent planning and executing each query; the number of blocks hit, read, and written; and more. pg_stat_statements comes pre-installed with Tiger Cloud.

For more information about pg_stat_statements, see the Postgres documentation.

Query the pg_stat_statements view as you would any Postgres view. The full view includes superuser queries used by Tiger Cloud to manage your service in the background. To view only your queries, filter by the current user.

Connect to your service and run the following command:

SELECT * FROM pg_stat_statements WHERE pg_get_userbyid(userid) = current_user;

For example, to identify the top five longest-running queries by their mean execution time:

SELECT calls,
    mean_exec_time,
    query
FROM pg_stat_statements
WHERE pg_get_userbyid(userid) = current_user
ORDER BY mean_exec_time DESC
LIMIT 5;

Or the top five queries with the highest relative variability in the execution time, expressed as a percentage:

SELECT calls,
    stddev_exec_time/mean_exec_time*100 AS rel_std_dev,
    query
FROM pg_stat_statements
WHERE pg_get_userbyid(userid) = current_user
ORDER BY rel_std_dev DESC
LIMIT 5;

For more examples and detailed explanations, see the blog post on identifying performance bottlenecks with pg_stat_statements.

===== PAGE: https://docs.tigerdata.com/use-timescale/metrics-logging/aws-cloudwatch/ =====

Export metrics to Amazon Cloudwatch

You can export telemetry data from your Tiger Cloud services with the time-series and analytics capability enabled to Amazon CloudWatch. Available metrics include CPU usage, RAM usage, and storage. This integration is available for Scale or Enterprise pricing plans.

This page shows you how to create an Amazon CloudWatch exporter in Tiger Cloud Console, and manage the lifecycle of data exporters.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

Create a data exporter

Tiger Cloud data exporters send telemetry data from a Tiger Cloud service to a third-party monitoring tools. You create an exporter on the project level, in the same AWS region as your service:

1. In Tiger Cloud Console, open Exporters

2. Click New exporter

3. Select the data type and specify AWS CloudWatch for provider

   

4. Provide your AWS CloudWatch configuration

   • The AWS region must be the same for your Tiger Cloud exporter and AWS CloudWatch Log group.
   • The exporter name appears in Tiger Cloud Console, best practice is to make this name easily understandable.
   • For CloudWatch credentials, either use an existing CloudWatch Log group or create a new one. If you're uncertain, use the default values. For more information, see Working with log groups and log streams.

5. Choose the authentication method to use for the exporter

   

   1. In AWS, navigate to IAM > Identity providers, then click Add provider.

   2. Update the new identity provider with your details:

      Set Provider URL to the region where you are creating your exporter.

      

   3. Click Add provider.

   4. In AWS, navigate to IAM > Roles, then click Create role.

   5. Add your identity provider as a Web identity role and click Next.

      

   6. Set the following permission and trust policies:

      • Permission policy:

        {
          "Version": "2012-10-17",
          "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
                     "logs:PutLogEvents",
                     "logs:CreateLogGroup",
                     "logs:CreateLogStream",
                     "logs:DescribeLogStreams",
                     "logs:DescribeLogGroups",
                     "logs:PutRetentionPolicy",
                     "xray:PutTraceSegments",
                     "xray:PutTelemetryRecords",
                     "xray:GetSamplingRules",
                     "xray:GetSamplingTargets",
                     "xray:GetSamplingStatisticSummaries",
                     "ssm:GetParameters"
                 ],
                 "Resource": "*"
             }
         ]
        }
        

      • Role with a Trust Policy:

        {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "Federated": "arn:aws:iam::12345678910:oidc-provider/irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com"
                  },
                  "Action": "sts:AssumeRoleWithWebIdentity",
                  "Condition": {
                      "StringEquals": {
                          "irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com:aud": "sts.amazonaws.com"
                      }
                  }
              },
              {
                  "Sid": "Statement1",
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::12345678910:role/my-exporter-role"
                  },
                  "Action": "sts:AssumeRole"
              }
          ]
        }
        

   7. Click Add role.

   When you use CloudWatch credentials, you link an Identity and Access Management (IAM) user with access to CloudWatch only with your Tiger Cloud service:

   1. Retrieve the user information from IAM > Users in AWS console.

      If you do not have an AWS user with access restricted to CloudWatch only, create one. For more information, see Creating IAM users (console).

   2. Enter the credentials for the AWS IAM user.

      AWS keys give access to your AWS services. To keep your AWS account secure, restrict users to the minimum required permissions. Always store your keys in a safe location. To avoid this issue, use the IAM role authentication method.

6. Select the AWS Region your CloudWatch services run in, then click Create exporter.

Manage a data exporter

This section shows you how to attach, monitor, edit, and delete a data exporter.

Attach a data exporter to a Tiger Cloud service

To send telemetry data to an external monitoring tool, you attach a data exporter to your Tiger Cloud service. You can attach only one exporter to a service.

To attach an exporter:

1. In Tiger Cloud Console, choose the service
2. Click Operations > Exporters
3. Select the exporter, then click Attach exporter
4. If you are attaching a first Logs data type exporter, restart the service

Monitor Tiger Cloud service metrics

You can now monitor your service metrics. Use the following metrics to check the service is running correctly:

• timescale.cloud.system.cpu.usage.millicores
• timescale.cloud.system.cpu.total.millicores
• timescale.cloud.system.memory.usage.bytes
• timescale.cloud.system.memory.total.bytes
• timescale.cloud.system.disk.usage.bytes
• timescale.cloud.system.disk.total.bytes

Additionally, use the following tags to filter your results.

Tag	Example variable	Description
host	us-east-1.timescale.cloud
project-id
service-id
region	us-east-1	AWS region
role	replica or primary	For service with replicas
node-id		For multi-node services

Edit a data exporter

To update a data exporter:

1. In Tiger Cloud Console, open Exporters
2. Next to the exporter you want to edit, click the menu > Edit
3. Edit the exporter fields and save your changes

You cannot change fields such as the provider or the AWS region.

Delete a data exporter

To remove a data exporter that you no longer need:

1. Disconnect the data exporter from your Tiger Cloud services

   1. In Tiger Cloud Console, choose the service.
   2. Click Operations > Exporters.
   3. Click the trash can icon.
   4. Repeat for every service attached to the exporter you want to remove.

   The data exporter is now unattached from all services. However, it still exists in your project.

2. Delete the exporter on the project level

   1. In Tiger Cloud Console, open Exporters
   2. Next to the exporter you want to edit, click menu > Delete
   3. Confirm that you want to delete the data exporter.

Reference

When you create the IAM OIDC provider, the URL must match the region you create the exporter in. It must be one of the following:

Region	Zone	Location	URL
ap-southeast-1	Asia Pacific	Singapore	irsa-oidc-discovery-prod-ap-southeast-1.s3.ap-southeast-1.amazonaws.com
ap-southeast-2	Asia Pacific	Sydney	irsa-oidc-discovery-prod-ap-southeast-2.s3.ap-southeast-2.amazonaws.com
ap-northeast-1	Asia Pacific	Tokyo	irsa-oidc-discovery-prod-ap-northeast-1.s3.ap-northeast-1.amazonaws.com
ca-central-1	Canada	Central	irsa-oidc-discovery-prod-ca-central-1.s3.ca-central-1.amazonaws.com
eu-central-1	Europe	Frankfurt	irsa-oidc-discovery-prod-eu-central-1.s3.eu-central-1.amazonaws.com
eu-west-1	Europe	Ireland	irsa-oidc-discovery-prod-eu-west-1.s3.eu-west-1.amazonaws.com
eu-west-2	Europe	London	irsa-oidc-discovery-prod-eu-west-2.s3.eu-west-2.amazonaws.com
sa-east-1	South America	São Paulo	irsa-oidc-discovery-prod-sa-east-1.s3.sa-east-1.amazonaws.com
us-east-1	United States	North Virginia	irsa-oidc-discovery-prod.s3.us-east-1.amazonaws.com
us-east-2	United States	Ohio	irsa-oidc-discovery-prod-us-east-2.s3.us-east-2.amazonaws.com
us-west-2	United States	Oregon	irsa-oidc-discovery-prod-us-west-2.s3.us-west-2.amazonaws.com

===== PAGE: https://docs.tigerdata.com/use-timescale/data-retention/create-a-retention-policy/ =====

Create a data retention policy

Automatically drop data once its time value ages past a certain interval. When you create a data retention policy, TimescaleDB automatically schedules a background job to drop old chunks.

Add a data retention policy

Add a data retention policy by using the add_retention_policy function.

Adding a data retention policy

1. Choose which hypertable you want to add the policy to. Decide how long you want to keep data before dropping it. In this example, the hypertable named conditions retains the data for 24 hours.

2. Call add_retention_policy:

   SELECT add_retention_policy('conditions', INTERVAL '24 hours');
   

A data retention policy only allows you to drop chunks based on how far they are in the past. To drop chunks based on how far they are in the future, manually drop chunks.

Remove a data retention policy

Remove an existing data retention policy by using the remove_retention_policy function. Pass it the name of the hypertable to remove the policy from.

SELECT remove_retention_policy('conditions');

See scheduled data retention jobs

To see your scheduled data retention jobs and their job statistics, query the timescaledb_information.jobs and timescaledb_information.job_stats tables. For example:

SELECT j.hypertable_name,
       j.job_id,
       config,
       schedule_interval,
       job_status,
       last_run_status,
       last_run_started_at,
       js.next_start,
       total_runs,
       total_successes,
       total_failures
  FROM timescaledb_information.jobs j
  JOIN timescaledb_information.job_stats js
    ON j.job_id = js.job_id
  WHERE j.proc_name = 'policy_retention';

The results look like this:

-[ RECORD 1 ]-------+-----------------------------------------------
hypertable_name     | conditions
job_id              | 1000
config              | {"drop_after": "5 years", "hypertable_id": 14}
schedule_interval   | 1 day
job_status          | Scheduled
last_run_status     | Success
last_run_started_at | 2022-05-19 16:15:11.200109+00
next_start          | 2022-05-20 16:15:11.243531+00
total_runs          | 1
total_successes     | 1
total_failures      | 0

===== PAGE: https://docs.tigerdata.com/use-timescale/data-retention/manually-drop-chunks/ =====

Manually drop chunks

Drop chunks manually by time value. For example, drop chunks containing data older than 30 days.

Dropping chunks manually is a one-time operation. To automatically drop chunks as they age, set up a data retention policy.

Drop chunks older than a certain date

To drop chunks older than a certain date, use the drop_chunks function. Provide the name of the hypertable to drop chunks from, and a time interval beyond which to drop chunks.

For example, to drop chunks with data older than 24 hours:

SELECT drop_chunks('conditions', INTERVAL '24 hours');

Drop chunks between 2 dates

You can also drop chunks between 2 dates. For example, drop chunks with data between 3 and 4 months old.

Supply a second INTERVAL argument for the newer_than cutoff:

SELECT drop_chunks(
  'conditions',
  older_than => INTERVAL '3 months',
  newer_than => INTERVAL '4 months'
)

Drop chunks in the future

You can also drop chunks in the future, for example, to correct data with the wrong timestamp. To drop all chunks that are more than 3 months in the future, from a hypertable called conditions:

SELECT drop_chunks(
  'conditions',
  newer_than => now() + INTERVAL '3 months'
);

===== PAGE: https://docs.tigerdata.com/use-timescale/data-retention/data-retention-with-continuous-aggregates/ =====

About data retention with continuous aggregates

You can downsample your data by combining a data retention policy with continuous aggregates. If you set your refresh policies correctly, you can delete old data from a hypertable without deleting it from any continuous aggregates. This lets you save on raw data storage while keeping summarized data for historical analysis.

To keep your aggregates while dropping raw data, you must be careful about refreshing your aggregates. You can delete raw data from the underlying table without deleting data from continuous aggregates, so long as you don't refresh the aggregate over the deleted data. When you refresh a continuous aggregate, TimescaleDB updates the aggregate based on changes in the raw data for the refresh window. If it sees that the raw data was deleted, it also deletes the aggregate data. To prevent this, make sure that the aggregate's refresh window doesn't overlap with any deleted data. For more information, see the following example.

As an example, say that you add a continuous aggregate to a conditions hypertable that stores device temperatures:

CREATE MATERIALIZED VIEW conditions_summary_daily (day, device, temp)
WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 day', time), device, avg(temperature)
  FROM conditions
  GROUP BY (1, 2);

SELECT add_continuous_aggregate_policy('conditions_summary_daily', '7 days', '1 day', '1 day');

This creates a conditions_summary_daily aggregate which stores the daily temperature per device. The aggregate refreshes every day. Every time it refreshes, it updates with any data changes from 7 days ago to 1 day ago.

You should not set a 24-hour retention policy on the conditions hypertable. If you do, chunks older than 1 day are dropped. Then the aggregate refreshes based on data changes. Since the data change was to delete data older than 1 day, the aggregate also deletes the data. You end up with no data in the conditions_summary_daily table.

To fix this, set a longer retention policy, for example 30 days:

SELECT add_retention_policy('conditions', INTERVAL '30 days');

Now, chunks older than 30 days are dropped. But when the aggregate refreshes, it doesn't look for changes older than 30 days. It only looks for changes between 7 days and 1 day ago. The raw hypertable still contains data for that time period. So your aggregate retains the data.

Data retention on a continuous aggregate itself

You can also apply data retention on a continuous aggregate itself. For example, you can keep raw data for 30 days, as mentioned earlier. Meanwhile, you can keep daily data for 600 days, and no data beyond that.

===== PAGE: https://docs.tigerdata.com/use-timescale/data-retention/about-data-retention/ =====

About data retention

In modern applications, data grows exponentially. As data gets older, it often becomes less useful in day-to-day operations. However, you still need it for analysis. TimescaleDB elegantly solves this problem with automated data retention policies.

Data retention policies delete raw old data for you on a schedule that you define. By combining retention policies with continuous aggregates, you can downsample your data and keep useful summaries of it instead. This lets you analyze historical data - while also saving on storage.

Drop data by chunk

TimescaleDB data retention works on chunks, not on rows. Deleting data row-by-row, for example, with the Postgres DELETE command, can be slow. But dropping data by the chunk is faster, because it deletes an entire file from disk. It doesn't need garbage collection and defragmentation.

Whether you use a policy or manually drop chunks, TimescaleDB drops data by the chunk. It only drops chunks where all the data is within the specified time range.

For example, consider the setup where you have 3 chunks containing data:

1. More than 36 hours old
2. Between 12 and 36 hours old
3. From the last 12 hours

You manually drop chunks older than 24 hours. Only the oldest chunk is deleted. The middle chunk is retained, because it contains some data newer than 24 hours. No individual rows are deleted from that chunk.

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/refresh-policies/ =====

Refresh continuous aggregates

Continuous aggregates can have a range of different refresh policies. In addition to refreshing the continuous aggregate automatically using a policy, you can also refresh it manually.

Prerequisites

To follow the procedure on this page you need to:

• Create a target Tiger Cloud service.

  This procedure also works for self-hosted TimescaleDB.

Change the refresh policy

Continuous aggregates require a policy for automatic refreshing. You can adjust this to suit different use cases. For example, you can have the continuous aggregate and the hypertable stay in sync, even when data is removed from the hypertable. Alternatively, you could keep source data in the continuous aggregate even after it is removed from the hypertable.

You can change the way your continuous aggregate is refreshed by calling add_continuous_aggregate_policy.

Among others, add_continuous_aggregate_policy takes the following arguments:

• start_offset: the start of the refresh window relative to when the policy runs
• end_offset: the end of the refresh window relative to when the policy runs
• schedule_interval: the refresh interval in minutes or hours. Defaults to 24 hours.

Note the following:

• If you set the start_offset or end_offset to NULL, the range is open-ended and extends to the beginning or end of time.

• If you set end_offset within the current time bucket, this bucket is excluded from materialization. This is done for the following reasons:

  • The current bucket is incomplete and can't be refreshed.
  • The current bucket gets a lot of writes in the timestamp order, and its aggregate becomes outdated very quickly. Excluding it improves performance.

  To include the latest raw data in queries, enable real-time aggregation.

See the API reference for the full list of required and optional arguments and use examples.

The policy in the following example ensures that all data in the continuous aggregate is up to date with the hypertable, except for data written within the last hour of wall-clock time. The policy also does not refresh the last time bucket of the continuous aggregate.

Since the policy in this example runs once every hour (schedule_interval) while also excluding data within the most recent hour (end_offset), it takes up to 2 hours for data written to the hypertable to be reflected in the continuous aggregate. Backfills, which are usually outside the most recent hour of data, will be visible after up to 1 hour depending on when the policy last ran when the data was written.

Because it has an open-ended start_offset parameter, any data that is removed from the table, for example with a DELETE or with drop_chunks, is also removed from the continuous aggregate view. This means that the continuous aggregate always reflects the data in the underlying hypertable.

To changing a refresh policy to use a NULL start_offset:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Create a new policy on conditions_summary_hourly that keeps the continuous aggregate up to date, and runs every hour:

   SELECT add_continuous_aggregate_policy('conditions_summary_hourly',
     start_offset => NULL,
     end_offset => INTERVAL '1 h',
     schedule_interval => INTERVAL '1 h');
   

If you want to keep data in the continuous aggregate even if it is removed from the underlying hypertable, you can set the start_offset to match the data retention policy on the source hypertable. For example, if you have a retention policy that removes data older than one month, set start_offset to one month or less. This sets your policy so that it does not refresh the dropped data.

1. Connect to your Tiger Cloud service.

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Create a new policy on conditions_summary_hourly that keeps data removed from the hypertable in the continuous aggregate, and runs every hour:

   SELECT add_continuous_aggregate_policy('conditions_summary_hourly',
     start_offset => INTERVAL '1 month',
     end_offset => INTERVAL '1 h',
     schedule_interval => INTERVAL '1 h');
   

It is important to consider your data retention policies when you're setting up continuous aggregate policies. If the continuous aggregate policy window covers data that is removed by the data retention policy, the data will be removed when the aggregates for those buckets are refreshed. For example, if you have a data retention policy that removes all data older than two weeks, the continuous aggregate policy will only have data for the last two weeks.

Add concurrent refresh policies

You can add concurrent refresh policies on each continuous aggregate, as long as their start and end offsets don't overlap. For example, to backfill data into older chunks you set up one policy that refreshes recent data, and another that refreshes backfilled data.

The first policy in this example is keeps the continuous aggregate up to date with data that was inserted in the past day. Any data that was inserted or updated for previous days is refreshed by the second policy.

1. Connect to your Tiger Cloud service.

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Create a new policy on conditions_summary_daily to refresh the continuous aggregate with recently inserted data which runs hourly:

   SELECT add_continuous_aggregate_policy('conditions_summary_daily',
     start_offset => INTERVAL '1 day',
     end_offset => INTERVAL '1 h',
     schedule_interval => INTERVAL '1 h');
   

3. At the psql prompt, create a concurrent policy on conditions_summary_daily to refresh the continuous aggregate with backfilled data:

   SELECT add_continuous_aggregate_policy('conditions_summary_daily',
     start_offset => NULL
     end_offset => INTERVAL '1 day',
     schedule_interval => INTERVAL '1 hour');
   

Manually refresh a continuous aggregate

If you need to manually refresh a continuous aggregate, you can use the refresh command. This recomputes the data within the window that has changed in the underlying hypertable since the last refresh. Therefore, if only a few buckets need updating, the refresh runs quickly.

If you have recently dropped data from a hypertable with a continuous aggregate, calling refresh_continuous_aggregate on a region containing dropped chunks recalculates the aggregate without the dropped data. See drop data for more information.

The refresh command takes three arguments:

• The name of the continuous aggregate view to refresh
• The timestamp of the beginning of the refresh window
• The timestamp of the end of the refresh window

Only buckets that are wholly within the specified range are refreshed. For example, if you specify 2021-05-01', '2021-06-01 the only buckets that are refreshed are those up to but not including 2021-06-01. It is possible to specify NULL in a manual refresh to get an open-ended range, but we do not recommend using it, because you could inadvertently materialize a large amount of data, slow down your performance, and have unintended consequences on other policies like data retention.

To manually refresh a continuous aggregate, use the refresh command:

CALL refresh_continuous_aggregate('example', '2021-05-01', '2021-06-01');

Follow the logic used by automated refresh policies and avoid refreshing time buckets that are likely to have a lot of writes. This means that you should generally not refresh the latest incomplete time bucket. To include the latest raw data in your queries, use real-time aggregation instead.

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/drop-data/ =====

Dropping data

When you are working with continuous aggregates, you can drop a view, or you can drop raw data from the underlying hypertable or from the continuous aggregate itself. A combination of refresh and data retention policies can help you downsample your data. This lets you keep historical data at a lower granularity than recent data.

However, you should be aware if a retention policy is likely to drop raw data from your hypertable that you need in your continuous aggregate.

To simplify the process of setting up downsampling, you can use the visualizer and code generator.

Drop a continuous aggregate view

You can drop a continuous aggregate view using the DROP MATERIALIZED VIEW command. This command also removes refresh policies defined on the continuous aggregate. It does not drop the data from the underlying hypertable.

Dropping a continuous aggregate view

1. From the psqlprompt, drop the view:

   DROP MATERIALIZED VIEW view_name;
   

Drop raw data from a hypertable

If you drop data from a hypertable used in a continuous aggregate it can lead to problems with your continuous aggregate view. In many cases, dropping underlying data replaces the aggregate with NULL values, which can lead to unexpected results in your view.

You can drop data from a hypertable using drop_chunks in the usual way, but before you do so, always check that the chunk is not within the refresh window of a continuous aggregate that still needs the data. This is also important if you are manually refreshing a continuous aggregate. Calling refresh_continuous_aggregate on a region containing dropped chunks recalculates the aggregate without the dropped data.

If a continuous aggregate is refreshing when data is dropped because of a retention policy, the aggregate is updated to reflect the loss of data. If you need to retain the continuous aggregate after dropping the underlying data, set the start_offset value of the aggregate policy to a smaller interval than the drop_after parameter of the retention policy.

For more information, see the data retention documentation.

PolicyVisualizerDownsampling

Refer to the installation documentation for detailed setup instructions.

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/migrate/ =====

Migrate a continuous aggregate to the new form

In TimescaleDB v2.7 and later, continuous aggregates use a new format that improves performance and makes them compatible with more SQL queries. Continuous aggregates created in older versions of TimescaleDB, or created in a new version with the option timescaledb.finalized set to false, use the old format.

To migrate a continuous aggregate from the old format to the new format, you can use this procedure. It automatically copies over your data and policies. You can continue to use the continuous aggregate while the migration is happening.

Connect to your database and run:

CALL cagg_migrate('<CONTINUOUS_AGGREGATE_NAME>');

There are known issues with cagg_migrate() in version 2.8.0. Upgrade to version 2.8.1 or later before using it.

Configure continuous aggregate migration

The migration procedure provides two boolean configuration parameters, override and drop_old. By default, the name of your new continuous aggregate is the name of your old continuous aggregate, with the suffix _new.

Set override to true to rename your new continuous aggregate with the original name. The old continuous aggregate is renamed with the suffix _old.

To both rename and drop the old continuous aggregate entirely, set both parameters to true. Note that drop_old must be used together with override.

Check on continuous aggregate migration status

To check the progress of the continuous aggregate migration, query the migration planning table:

SELECT * FROM _timescaledb_catalog.continuous_agg_migrate_plan_step;

Troubleshooting

Permissions error when migrating a continuous aggregate

You might get a permissions error when migrating a continuous aggregate from old to new format using cagg_migrate. The user performing the migration must have the following permissions:

• Select, insert, and update permissions on the tables _timescale_catalog.continuous_agg_migrate_plan and _timescale_catalog.continuous_agg_migrate_plan_step
• Usage permissions on the sequence _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq

To solve the problem, change to a user capable of granting permissions, and grant the following permissions to the user performing the migration:

GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan TO <USER>;
GRANT SELECT, INSERT, UPDATE ON TABLE _timescaledb_catalog.continuous_agg_migrate_plan_step TO <USER>;
GRANT USAGE ON SEQUENCE _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq TO <USER>;

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/compression-on-continuous-aggregates/ =====

Compress continuous aggregates

To save on storage costs, you use hypercore to downsample historical data stored in continuous aggregates. After you enable columnstore on a MATERIALIZED VIEW, you set a columnstore policy. This policy defines the intervals when chunks in a continuous aggregate are compressed as they are converted from the rowstore to the columnstore.

Columnstore works in the same way on hypertables and continuous aggregates. When you enable columnstore with no other options, your data is segmented by the groupby columns in the continuous aggregate, and ordered by the time column. Real-time aggregation is disabled by default.

Since TimescaleDB v2.20.0 For the old API, see Compress continuous aggregates.

Configure columnstore on continuous aggregates

For an existing continuous aggregate:

1. Enable columnstore on a continuous aggregate

   To enable the columnstore compression on a continuous aggregate, set timescaledb.enable_columnstore = true when you alter the view:

   ALTER MATERIALIZED VIEW <cagg_name> set (timescaledb.enable_columnstore = true);
   

   To disable the columnstore compression, set timescaledb.enable_columnstore = false:

2. Set columnstore policies on the continuous aggregate

   Before you set up a columnstore policy on a continuous aggregate, you first set the refresh policy. To prevent refresh policies from failing, you set the columnstore policy interval so that actively refreshed regions are not compressed. For example:

   1. Set the refresh policy

      SELECT add_continuous_aggregate_policy('<cagg_name>',
        start_offset => INTERVAL '30 days',
        end_offset => INTERVAL '1 day',
        schedule_interval => INTERVAL '1 hour');
      

   2. Set the columnstore policy

      For this refresh policy, the after parameter must be greater than the value of start_offset in the refresh policy:

      CALL add_columnstore_policy('<cagg_name>', after => INTERVAL '45 days');
      

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/create-index/ =====

Create an index on a continuous aggregate

By default, some indexes are automatically created when you create a continuous aggregate. You can change this behavior. You can also manually create and drop indexes.

Automatically created indexes

When you create a continuous aggregate, an index is automatically created for each GROUP BY column. The index is a composite index, combining the GROUP BY column with the time_bucket column.

For example, if you define a continuous aggregate view with GROUP BY device, location, bucket, two composite indexes are created: one on {device, bucket} and one on {location, bucket}.

Turn off automatic index creation

To turn off automatic index creation, set timescaledb.create_group_indexes to false when you create the continuous aggregate.

For example:

CREATE MATERIALIZED VIEW conditions_daily
  WITH (timescaledb.continuous, timescaledb.create_group_indexes=false)
  AS
  ...

Manually create and drop indexes

You can use a regular Postgres statement to create or drop an index on a continuous aggregate.

For example, to create an index on avg_temp for a materialized hypertable named weather_daily:

CREATE INDEX avg_temp_idx ON weather_daily (avg_temp);

Indexes are created under the _timescaledb_internal schema, where the continuous aggregate data is stored. To drop the index, specify the schema. For example, to drop the index avg_temp_idx, run:

DROP INDEX _timescaledb_internal.avg_temp_idx

Limitations on created indexes

In TimescaleDB v2.7 and later, you can create an index on any column in the materialized view. This includes aggregated columns, such as those storing sums and averages. In earlier versions of TimescaleDB, you can't create an index on an aggregated column.

You can't create unique indexes on a continuous aggregate, in any of the TimescaleDB versions.

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/about-continuous-aggregates/ =====

About continuous aggregates

In modern applications, data usually grows very quickly. This means that aggregating it into useful summaries can become very slow. If you are collecting data very frequently, you might want to aggregate your data into minutes or hours instead. For example, if an IoT device takes temperature readings every second, you might want to find the average temperature for each hour. Every time you run this query, the database needs to scan the entire table and recalculate the average. TimescaleDB makes aggregating data lightning fast, accurate, and easy with continuous aggregates.

Continuous aggregates in TimescaleDB are a kind of hypertable that is refreshed automatically in the background as new data is added, or old data is modified. Changes to your dataset are tracked, and the hypertable behind the continuous aggregate is automatically updated in the background.

Continuous aggregates have a much lower maintenance burden than regular Postgres materialized views, because the whole view is not created from scratch on each refresh. This means that you can get on with working your data instead of maintaining your database.

Because continuous aggregates are based on hypertables, you can query them in exactly the same way as your other tables. This includes continuous aggregates in the rowstore, compressed into the columnstore, or tiered to object storage. You can even create continuous aggregates on top of your continuous aggregates, for an even more fine-tuned aggregation.

Real-time aggregation enables you to combine pre-aggregated data from the materialized view with the most recent raw data. This gives you up-to-date results on every query. In TimescaleDB v2.13 and later, real-time aggregates are DISABLED by default. In earlier versions, real-time aggregates are ENABLED by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data.

Types of aggregation

There are three main ways to make aggregation easier: materialized views, continuous aggregates, and real-time aggregates.

Materialized views are a standard Postgres function. They are used to cache the result of a complex query so that you can reuse it later on. Materialized views do not update regularly, although you can manually refresh them as required.

Continuous aggregates are a TimescaleDB-only feature. They work in a similar way to a materialized view, but they are updated automatically in the background, as new data is added to your database. Continuous aggregates are updated continuously and incrementally, which means they are less resource intensive to maintain than materialized views. Continuous aggregates are based on hypertables, and you can query them in the same way as you do your other tables.

Real-time aggregates are a TimescaleDB-only feature. They are the same as continuous aggregates, but they add the most recent raw data to the previously aggregated data to provide accurate and up-to-date results, without needing to aggregate data as it is being written.

Continuous aggregates on continuous aggregates

You can create a continuous aggregate on top of another continuous aggregate. This allows you to summarize data at different granularity. For example, you might have a raw hypertable that contains second-by-second data. Create a continuous aggregate on the hypertable to calculate hourly data. To calculate daily data, create a continuous aggregate on top of your hourly continuous aggregate.

For more information, see the documentation about continuous aggregates on continuous aggregates.

Continuous aggregates with a JOIN clause

Continuous aggregates support the following JOIN features:

Feature	TimescaleDB < 2.10.x	TimescaleDB <= 2.15.x	TimescaleDB >= 2.16.x
INNER JOIN	❌	✅	✅
LEFT JOIN	❌	❌	✅
LATERAL JOIN	❌	❌	✅
Joins between ONE hypertable and ONE standard Postgres table	❌	✅	✅
Joins between ONE hypertable and MANY standard Postgres tables	❌	❌	✅
Join conditions must be equality conditions, and there can only be ONE JOIN condition	❌	✅	✅
Any join conditions	❌	❌	✅

JOINS in TimescaleDB must meet the following conditions:

• Only the changes to the hypertable are tracked, and they are updated in the continuous aggregate when it is refreshed. Changes to standard Postgres table are not tracked.
• You can use an INNER, LEFT, and LATERAL joins; no other join type is supported.
• Joins on the materialized hypertable of a continuous aggregate are not supported.
• Hierarchical continuous aggregates can be created on top of a continuous aggregate with a JOIN clause, but cannot themselves have a JOIN clause.

JOIN examples

Given the following schema:

CREATE TABLE locations (
  id TEXT PRIMARY KEY,
  name TEXT
);

CREATE TABLE devices (
  id SERIAL PRIMARY KEY,
  location_id TEXT,
  name TEXT
);

CREATE TABLE conditions (
  "time" TIMESTAMPTZ,
  device_id INTEGER,
  temperature FLOAT8
) WITH (
  tsdb.hypertable,
  tsdb.partition_column='time'
);

See the following JOIN examples on continuous aggregates:

• INNER JOIN on a single equality condition, using the ON clause:

  CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature)
  FROM conditions
  JOIN devices ON devices.id = conditions.device_id
  GROUP BY bucket, devices.name
  WITH NO DATA;
  

• INNER JOIN on a single equality condition, using the ON clause, with a further condition added in the WHERE clause:

  CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature)
  FROM conditions
  JOIN devices ON devices.id = conditions.device_id
  WHERE devices.location_id = 'location123'
  GROUP BY bucket, devices.name
  WITH NO DATA;
  

• INNER JOIN on a single equality condition specified in WHERE clause:

  CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature)
  FROM conditions, devices
  WHERE devices.id = conditions.device_id
  GROUP BY bucket, devices.name
  WITH NO DATA;
  

• INNER JOIN on multiple equality conditions:

  CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature)
  FROM conditions
  JOIN devices ON devices.id = conditions.device_id AND devices.location_id = 'location123'
  GROUP BY bucket, devices.name
  WITH NO DATA;
  

  TimescaleDB v2.16.x and higher.

• INNER JOIN with a single equality condition specified in WHERE clause can be combined with further conditions in the WHERE clause:

  CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature)
  FROM conditions, devices
  WHERE devices.id = conditions.device_id
  AND devices.location_id = 'location123'
  GROUP BY bucket, devices.name
  WITH NO DATA;
  

  TimescaleDB v2.16.x and higher.

• INNER JOIN between a hypertable and multiple Postgres tables:

  CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 day', time) AS bucket, devices.name AS device, locations.name AS location, MIN(temperature), MAX(temperature)
  FROM conditions
  JOIN devices ON devices.id = conditions.device_id
  JOIN locations ON locations.id = devices.location_id
  GROUP BY bucket, devices.name, locations.name
  WITH NO DATA;
  

  TimescaleDB v2.16.x and higher.

• LEFT JOIN between a hypertable and a Postgres table:

  CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature)
  FROM conditions
  LEFT JOIN devices ON devices.id = conditions.device_id
  GROUP BY bucket, devices.name
  WITH NO DATA;
  

  TimescaleDB v2.16.x and higher.

• LATERAL JOIN between a hypertable and a subquery:

  CREATE MATERIALIZED VIEW conditions_by_day WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 day', time) AS bucket, devices.name, MIN(temperature), MAX(temperature)
  FROM conditions,
  LATERAL (SELECT * FROM devices WHERE devices.id = conditions.device_id) AS devices
  GROUP BY bucket, devices.name
  WITH NO DATA;
  

  TimescaleDB v2.16.x and higher.

Function support

In TimescaleDB v2.7 and later, continuous aggregates support all Postgres aggregate functions. This includes both parallelizable aggregates, such as SUM and AVG, and non-parallelizable aggregates, such as RANK.

In TimescaleDB v2.10.0 and later, the FROM clause supports JOINS, with some restrictions. For more information, see the JOIN support section.

In older versions of TimescaleDB, continuous aggregates only support aggregate functions that can be parallelized by Postgres. You can work around this by aggregating the other parts of your query in the continuous aggregate, then using the window function to query the aggregate.

The following table summarizes the aggregate functions supported in continuous aggregates:

Function, clause, or feature	TimescaleDB 2.6 and earlier	TimescaleDB 2.7, 2.8, and 2.9	TimescaleDB 2.10 and later
Parallelizable aggregate functions	✅	✅	✅
Non-parallelizable SQL aggregates	❌	✅	✅
ORDER BY	❌	✅	✅
Ordered-set aggregates	❌	✅	✅
Hypothetical-set aggregates	❌	✅	✅
DISTINCT in aggregate functions	❌	✅	✅
FILTER in aggregate functions	❌	✅	✅
FROM clause supports JOINS	❌	❌	✅

DISTINCT works in aggregate functions, not in the query definition. For example, for the table:

CREATE TABLE public.candle(
symbol_id uuid                     NOT NULL,
symbol    text                     NOT NULL,
"time"    timestamp with time zone NOT NULL,
open      double precision         NOT NULL,
high      double precision         NOT NULL,
low       double precision         NOT NULL,
close     double precision         NOT NULL,
volume    double precision         NOT NULL
);

• The following works:

  CREATE MATERIALIZED VIEW candles_start_end
  WITH (timescaledb.continuous) AS
  SELECT time_bucket('1 hour', "time"), COUNT(DISTINCT symbol), first(time, time) as first_candle, last(time, time) as last_candle
  FROM candle
  GROUP BY 1;
  

• This does not:

  CREATE MATERIALIZED VIEW candles_start_end
  WITH (timescaledb.continuous) AS
  SELECT DISTINCT ON (symbol)
  symbol,symbol_id, first(time, time) as first_candle, last(time, time) as last_candle
  FROM candle
  GROUP BY symbol_id;
  

If you want the old behavior in later versions of TimescaleDB, set the timescaledb.finalized parameter to false when you create your continuous aggregate.

Components of a continuous aggregate

Continuous aggregates consist of:

• Materialization hypertable to store the aggregated data in
• Materialization engine to aggregate data from the raw, underlying, table to the materialization hypertable
• Invalidation engine to determine when data needs to be re-materialized, due to changes in the data
• Query engine to access the aggregated data

Materialization hypertable

Continuous aggregates take raw data from the original hypertable, aggregate it, and store the aggregated data in a materialization hypertable. When you query the continuous aggregate view, the aggregated data is returned to you as needed.

Using the same temperature example, the materialization table looks like this:

day	location	chunk	avg temperature
2021/01/01	New York	1	73
2021/01/01	Stockholm	1	70
2021/01/02	New York	2
2021/01/02	Stockholm	2	69

The materialization table is stored as a TimescaleDB hypertable, to take advantage of the scaling and query optimizations that hypertables offer. Materialization tables contain a column for each group-by clause in the query, and an aggregate column for each aggregate in the query.

For more information, see materialization hypertables.

Materialization engine

The materialization engine performs two transactions. The first transaction blocks all INSERTs, UPDATEs, and DELETEs, determines the time range to materialize, and updates the invalidation threshold. The second transaction unblocks other transactions, and materializes the aggregates. The first transaction is very quick, and most of the work happens during the second transaction, to ensure that the work does not interfere with other operations.

Invalidation engine

Any change to the data in a hypertable could potentially invalidate some materialized rows. The invalidation engine checks to ensure that the system does not become swamped with invalidations.

Fortunately, time-series data means that nearly all INSERTs and UPDATEs have a recent timestamp, so the invalidation engine does not materialize all the data, but to a set point in time called the materialization threshold. This threshold is set so that the vast majority of INSERTs contain more recent timestamps. These data points have never been materialized by the continuous aggregate, so there is no additional work needed to notify the continuous aggregate that they have been added. When the materializer next runs, it is responsible for determining how much new data can be materialized without invalidating the continuous aggregate. It then materializes the more recent data and moves the materialization threshold forward in time. This ensures that the threshold lags behind the point-in-time where data changes are common, and that most INSERTs do not require any extra writes.

When data older than the invalidation threshold is changed, the maximum and minimum timestamps of the changed rows is logged, and the values are used to determine which rows in the aggregation table need to be recalculated. This logging does cause some write load, but because the threshold lags behind the area of data that is currently changing, the writes are small and rare.

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/time/ =====

Time and continuous aggregates

Functions that depend on a local timezone setting inside a continuous aggregate are not supported. You cannot adjust to a local time because the timezone setting changes from user to user.

To manage this, you can use explicit timezones in the view definition. Alternatively, you can create your own custom aggregation scheme for tables that use an integer time column.

Declare an explicit timezone

The most common method of working with timezones is to declare an explicit timezone in the view query.

1. At the psqlprompt, create the view and declare the timezone:

   CREATE MATERIALIZED VIEW device_summary
   WITH (timescaledb.continuous)
   AS
   SELECT
     time_bucket('1 hour', observation_time) AS bucket,
     min(observation_time AT TIME ZONE 'EST') AS min_time,
     device_id,
     avg(metric) AS metric_avg,
     max(metric) - min(metric) AS metric_spread
   FROM
     device_readings
   GROUP BY bucket, device_id;
   

2. Alternatively, you can cast to a timestamp after the view using SELECT:

   SELECT min_time::timestamp FROM device_summary;
   

Integer-based time

Date and time is usually expressed as year-month-day and hours:minutes:seconds. Most TimescaleDB databases use a date/time-type column to express the date and time. However, in some cases, you might need to convert these common time and date formats to a format that uses an integer. The most common integer time is Unix epoch time, which is the number of seconds since the Unix epoch of 1970-01-01, but other types of integer-based time formats are possible.

These examples use a hypertable called devices that contains CPU and disk usage information. The devices measure time using the Unix epoch.

To create a hypertable that uses an integer-based column as time, you need to provide the chunk time interval. In this case, each chunk is 10 minutes.

1. At the psql prompt, create a hypertable and define the integer-based time column and chunk time interval:

   CREATE TABLE devices(
     time BIGINT,        -- Time in minutes since epoch
     cpu_usage INTEGER,  -- Total CPU usage
     disk_usage INTEGER, -- Total disk usage
     PRIMARY KEY (time)
   ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='time',
     tsdb.chunk_interval='10'
   );
   

   If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

To define a continuous aggregate on a hypertable that uses integer-based time, you need to have a function to get the current time in the correct format, and set it for the hypertable. You can do this with the set_integer_now_func function. It can be defined as a regular Postgres function, but needs to be STABLE, take no arguments, and return an integer value of the same type as the time column in the table. When you have set up the time-handling, you can create the continuous aggregate.

1. At the psql prompt, set up a function to convert the time to the Unix epoch:

   CREATE FUNCTION current_epoch() RETURNS BIGINT
   LANGUAGE SQL STABLE AS $$
   SELECT EXTRACT(EPOCH FROM CURRENT_TIMESTAMP)::bigint;$$;
   
    SELECT set_integer_now_func('devices', 'current_epoch');
   

2. Create the continuous aggregate for the devices table:

   CREATE MATERIALIZED VIEW devices_summary
   WITH (timescaledb.continuous) AS
   SELECT time_bucket('500', time) AS bucket,
      avg(cpu_usage) AS avg_cpu,
      avg(disk_usage) AS avg_disk
   FROM devices
   GROUP BY bucket;
   

3. Insert some rows into the table:

   CREATE EXTENSION tablefunc;
   
   INSERT INTO devices(time, cpu_usage, disk_usage)
   SELECT time,
      normal_rand(1,70,10) AS cpu_usage,
     normal_rand(1,2,1) * (row_number() over()) AS disk_usage
   FROM generate_series(1,10000) AS time;
   

   This command uses the tablefunc extension to generate a normal distribution, and uses the row_number function to turn it into a cumulative sequence.

4. Check that the view contains the correct data:

   postgres=# SELECT * FROM devices_summary ORDER BY bucket LIMIT 10;
   bucket |       avg_cpu       |       avg_disk
   --------+---------------------+----------------------
        0 | 63.0000000000000000 |   6.0000000000000000
        5 | 69.8000000000000000 |   9.6000000000000000
       10 | 70.8000000000000000 |  24.0000000000000000
       15 | 75.8000000000000000 |  37.6000000000000000
       20 | 71.6000000000000000 |  26.8000000000000000
       25 | 67.6000000000000000 |  56.0000000000000000
       30 | 68.8000000000000000 |  90.2000000000000000
       35 | 71.6000000000000000 |  88.8000000000000000
       40 | 66.4000000000000000 |  81.2000000000000000
       45 | 68.2000000000000000 | 106.0000000000000000
   (10 rows)
   

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/materialized-hypertables/ =====

Materialized hypertables

Continuous aggregates take raw data from the original hypertable, aggregate it, and store the aggregated data in a materialization hypertable. You can modify this materialized hypertable in the same way as any other hypertable.

Discover the name of a materialized hypertable

To change a materialized hypertable, you need to use its fully qualified name. To find the correct name, use the timescaledb_information.continuous_aggregates view). You can then use the name to modify it in the same way as any other hypertable.

Discovering the name of a materialized hypertable

1. At the psqlprompt, query timescaledb_information.continuous_aggregates:

   SELECT view_name, format('%I.%I', materialization_hypertable_schema,
           materialization_hypertable_name) AS materialization_hypertable
       FROM timescaledb_information.continuous_aggregates;
   

2. Locate the name of the hypertable you want to adjust in the results of the query. The results look like this:

            view_name         |            materialization_hypertable
   ---------------------------+---------------------------------------------------
   conditions_summary_hourly | _timescaledb_internal._materialized_hypertable_30
   conditions_summary_daily  | _timescaledb_internal._materialized_hypertable_31
   (2 rows)
   

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/real-time-aggregates/ =====

Real-time aggregates

Rapidly growing data means you need more control over what to aggregate and how to aggregate it. With this in mind, Tiger Data equips you with tools for more fine-tuned data analysis.

By default, continuous aggregates do not include the most recent data chunk from the underlying hypertable. Real-time aggregates, however, use the aggregated data and add the most recent raw data to it. This provides accurate and up-to-date results, without needing to aggregate data as it is being written.

In TimescaleDB v2.13 and later, real-time aggregates are DISABLED by default. In earlier versions, real-time aggregates are ENABLED by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data.

For more detail on the comparison between continuous and real-time aggregates, see our real-time aggregate blog post.

Use real-time aggregates

You can enable and disable real-time aggregation by setting the materialized_only parameter when you create or alter the view.

1. Enable real-time aggregation for an existing continuous aggregate:

   ALTER MATERIALIZED VIEW table_name set (timescaledb.materialized_only = false);
   

2. Disable real-time aggregation:

   ALTER MATERIALIZED VIEW table_name set (timescaledb.materialized_only = true);
   

Real-time aggregates and refreshing historical data

Real-time aggregates automatically add the most recent data when you query your continuous aggregate. In other words, they include data more recent than your last materialized bucket.

If you add new historical data to an already-materialized bucket, it won't be reflected in a real-time aggregate. You should wait for the next scheduled refresh, or manually refresh by calling refresh_continuous_aggregate. You can think of real-time aggregates as being eventually consistent for historical data.

For more information, see the troubleshooting section.

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/create-a-continuous-aggregate/ =====

Create a continuous aggregate

Creating a continuous aggregate is a two-step process. You need to create the view first, then enable a policy to keep the view refreshed. You can create the view on a hypertable, or on top of another continuous aggregate. You can have more than one continuous aggregate on each source table or view.

Continuous aggregates require a time_bucket on the time partitioning column of the hypertable.

By default, views are automatically refreshed. You can adjust this by setting the WITH NO DATA option. Additionally, the view can not be a security barrier view.

Continuous aggregates use hypertables in the background, which means that they also use chunk time intervals. By default, the continuous aggregate's chunk time interval is 10 times what the original hypertable's chunk time interval is. For example, if the original hypertable's chunk time interval is 7 days, the continuous aggregates that are on top of it have a 70 day chunk time interval.

Create a continuous aggregate

In this example, we are using a hypertable called conditions, and creating a continuous aggregate view for daily weather data. The GROUP BY clause must include a time_bucket expression which uses time dimension column of the hypertable. Additionally, all functions and their arguments included in SELECT, GROUP BY, and HAVING clauses must be immutable.

Creating a continuous aggregate

1. At the psqlprompt, create the materialized view:

   CREATE MATERIALIZED VIEW conditions_summary_daily
   WITH (timescaledb.continuous) AS
   SELECT device,
      time_bucket(INTERVAL '1 day', time) AS bucket,
      AVG(temperature),
      MAX(temperature),
      MIN(temperature)
   FROM conditions
   GROUP BY device, bucket;
   

   To create a continuous aggregate within a transaction block, use the WITH NO DATA option.

   To improve continuous aggregate performance, set timescaledb.invalidate_using = 'wal' Since TimescaleDB v2.22.0.

2. Create a policy to refresh the view every hour:

   SELECT add_continuous_aggregate_policy('conditions_summary_daily',
     start_offset => INTERVAL '1 month',
     end_offset => INTERVAL '1 day',
     schedule_interval => INTERVAL '1 hour');
   

You can use most Postgres aggregate functions in continuous aggregations. To see what Postgres features are supported, check the function support table.

Choosing an appropriate bucket interval

Continuous aggregates require a time_bucket on the time partitioning column of the hypertable. The time bucket allows you to define a time interval, instead of having to use specific timestamps. For example, you can define a time bucket as five minutes, or one day.

You can't use time_bucket_gapfill directly in a continuous aggregate. This is because you need access to previous data to determine the gapfill content, which isn't yet available when you create the continuous aggregate. You can work around this by creating the continuous aggregate using time_bucket, then querying the continuous aggregate using time_bucket_gapfill.

Using the WITH NO DATA option

By default, when you create a view for the first time, it is populated with data. This is so that the aggregates can be computed across the entire hypertable. If you don't want this to happen, for example if the table is very large, or if new data is being continuously added, you can control the order in which the data is refreshed. You can do this by adding a manual refresh with your continuous aggregate policy using the WITH NO DATA option.

The WITH NO DATA option allows the continuous aggregate to be created instantly, so you don't have to wait for the data to be aggregated. Data begins to populate only when the policy begins to run. This means that only data newer than the start_offset time begins to populate the continuous aggregate. If you have historical data that is older than the start_offset interval, you need to manually refresh the history up to the current start_offset to allow real-time queries to run efficiently.

Creating a continuous aggregate with the WITH NO DATA option

1. At the psql prompt, create the view:

   CREATE MATERIALIZED VIEW cagg_rides_view
   WITH (timescaledb.continuous) AS
   SELECT vendor_id,
   time_bucket('1h', pickup_datetime) AS hour,
     count(*) total_rides,
     avg(fare_amount) avg_fare,
     max(trip_distance) as max_trip_distance,
     min(trip_distance) as min_trip_distance
   FROM rides
   GROUP BY vendor_id, time_bucket('1h', pickup_datetime)
   WITH NO DATA;
   

2. Manually refresh the view:

   CALL refresh_continuous_aggregate('cagg_rides_view', NULL, localtimestamp - INTERVAL '1 week');
   

3. Add the policy:

   SELECT add_continuous_aggregate_policy('cagg_rides_view',
     start_offset => INTERVAL '1 week',
     end_offset   => INTERVAL '1 hour',
     schedule_interval => INTERVAL '30 minutes');
   

Create a continuous aggregate with a JOIN

In TimescaleDB V2.10 and later, with Postgres v12 or later, you can create a continuous aggregate with a query that also includes a JOIN. For example:

CREATE MATERIALIZED VIEW conditions_summary_daily_3
WITH (timescaledb.continuous) AS
SELECT time_bucket(INTERVAL '1 day', day) AS bucket,
   AVG(temperature),
   MAX(temperature),
   MIN(temperature),
   name
FROM devices JOIN conditions USING (device_id)
GROUP BY name, bucket;

For more information about creating a continuous aggregate with a JOIN, including some additional restrictions, see the about continuous aggregates section.

Query continuous aggregates

When you have created a continuous aggregate and set a refresh policy, you can query the view with a SELECT query. You can only specify a single hypertable in the FROM clause. Including more hypertables, tables, views, or subqueries in your SELECT query is not supported. Additionally, make sure that the hypertable you are querying does not have row-level-security policies enabled.

Querying a continuous aggregate

1. At the psql prompt, query the continuous aggregate view called conditions_summary_hourly for the average, minimum, and maximum temperatures for the first quarter of 2021 recorded by device 5:

   SELECT *
     FROM conditions_summary_hourly
     WHERE device = 5
     AND bucket >= '2020-01-01'
     AND bucket < '2020-04-01';
   

2. Alternatively, query the continuous aggregate view called conditions_summary_hourly for the top 20 largest metric spreads in that quarter:

   SELECT *
     FROM conditions_summary_hourly
     WHERE max - min > 1800
     AND bucket >= '2020-01-01' AND bucket < '2020-04-01'
     ORDER BY bucket DESC, device DESC LIMIT 20;
   

Use continuous aggregates with mutable functions: experimental

Mutable functions have experimental supported in the continuous aggregate query definition. Mutable functions are enabled by default. However, if you use them in a materialized query a warning is returned.

When using non-immutable functions you have to ensure these functions produce consistent results across continuous aggregate refresh runs. For example, if a function depends on the current time zone you have to ensure all your continuous aggregate refreshes run with a consistent setting for this.

Use continuous aggregates with window functions: experimental

Window functions have experimental supported in the continuous aggregate query definition. Window functions are disabled by default. To enable them, set timescaledb.enable_cagg_window_functions to true.

Support is experimental, there is a risk of data inconsistency. For example, in backfill scenarios, buckets could be missed.

Create a window function

To use a window function in a continuous aggregate:

1. Create a simple table with to store a value at a specific time:

   CREATE TABLE example (
     time       TIMESTAMPZ        NOT NULL,
     value      TEXT              NOT NULL,
   );
   

2. Enable window functions.

   As window functions are experimental, in order to create continuous aggregates with window functions. you have to enable_cagg_window_functions.

    SET timescaledb.enable_cagg_window_functions TO TRUE;
   

3. Bucket your data by time and calculate the delta between time buckets using the lag window function:

   Window functions must stay within the time bucket. Any query that tries to look beyond the current time bucket will produce incorrect results around the refresh boundaries.

   CREATE MATERIALIZED VIEW example_aggregate
     WITH (timescaledb.continuous) AS
       SELECT
         time_bucket('1d', time),
         customer_id,
         sum(amount) AS amount,
         sum(amount) - LAG(sum(amount),1,NULL) OVER (PARTITION BY time_bucket('1d', time) ORDER BY sum(amount) DESC) AS amount_diff,
         ROW_NUMBER() OVER (PARTITION BY time_bucket('1d', time) ORDER BY sum(amount) DESC)
       FROM sales GROUP BY 1,2;
   

   Window functions that partition by time_bucket should be safe even with LAG()/LEAD()

Window function workaround for older versions of TimescaleDB

For TimescaleDB v2.19.3 and below, continuous aggregates do not support window functions. To work around this:

1. Create a simple table with to store a value at a specific time:

   CREATE TABLE example (
     time       TIMESTAMPZ        NOT NULL,
     value      TEXT              NOT NULL,
   );
   

2. Create a continuous aggregate that does not use a window function:

   CREATE MATERIALIZED VIEW example_aggregate
     WITH (timescaledb.continuous) AS
       SELECT
         time_bucket('10 minutes', time) AS bucket,
         first(value, time) AS value
       FROM example GROUP BY bucket;
   

3. Use the lag window function on your continuous aggregate at query time:

   This speeds up your query by calculating the aggregation ahead of time. The delta is calculated at query time.

   SELECT
     bucket,
     value - lag(value, 1) OVER (ORDER BY bucket) AS delta
   FROM example_aggregate;
   

===== PAGE: https://docs.tigerdata.com/use-timescale/continuous-aggregates/hierarchical-continuous-aggregates/ =====

Continuous aggregates on continuous aggregates

The more data you have, the more likely you are to run a more sophisticated analysis on it. When a simple one-level aggregation is not enough, TimescaleDB lets you create continuous aggregates on top of other continuous aggregates. This way, you summarize data at different levels of granularity, while still saving resources with precomputing.

For example, you might have an hourly continuous aggregate that summarizes minute-by-minute data. To get a daily summary, you can create a new continuous aggregate on top of your hourly aggregate. This is more efficient than creating the daily aggregate on top of the original hypertable, because you can reuse the calculations from the hourly aggregate.

This feature is available in TimescaleDB v2.9 and later.

Create a continuous aggregate on top of another continuous aggregate

Creating a continuous aggregate on top of another continuous aggregate works the same way as creating it on top of a hypertable. In your query, select from a continuous aggregate rather than from the hypertable, and use the time-bucketed column from the existing continuous aggregate as your time column.

For more information, see the instructions for creating a continuous aggregate.

Use real-time aggregation with hierarchical continuous aggregates

In TimescaleDB v2.13 and later, real-time aggregates are DISABLED by default. In earlier versions, real-time aggregates are ENABLED by default; when you create a continuous aggregate, queries to that view include the results from the most recent raw data.

Real-time aggregates always return up-to-date data in response to queries. They accomplish this by joining the materialized data in the continuous aggregate with unmaterialized raw data from the source table or view.

When continuous aggregates are stacked, each continuous aggregate is only aware of the layer immediately below. The joining of unmaterialized data happens recursively until it reaches the bottom layer, giving you access to recent data down to that layer.

If you keep all continuous aggregates in the stack as real-time aggregates, the bottom layer is the source hypertable. That means every continuous aggregate in the stack has access to all recent data.

If there is a non-real-time continuous aggregate somewhere in the stack, the recursive joining stops at that non-real-time continuous aggregate. Higher-level continuous aggregates don't receive any unmaterialized data from lower levels.

For example, say you have the following continuous aggregates:

• A real-time hourly continuous aggregate on the source hypertable
• A real-time daily continuous aggregate on the hourly continuous aggregate
• A non-real-time, or materialized-only, monthly continuous aggregate on the daily continuous aggregate
• A real-time yearly continuous aggregate on the monthly continuous aggregate

Queries on the hourly and daily continuous aggregates include real-time, non-materialized data from the source hypertable. Queries on the monthly continuous aggregate only return already-materialized data. Queries on the yearly continuous aggregate return materialized data from the yearly continuous aggregate itself, plus more recent data from the monthly continuous aggregate. However, the data is limited to what is already materialized in the monthly continuous aggregate, and doesn't get even more recent data from the source hypertable. This happens because the materialized-only continuous aggregate provides a stopping point, and the yearly continuous aggregate is unaware of any layers beyond that stopping point. This is similar to how stacked views work in Postgres.

To make queries on the yearly continuous aggregate access all recent data, you can either:

• Make the monthly continuous aggregate real-time, or
• Redefine the yearly continuous aggregate on top of the daily continuous aggregate.

Roll up calculations

When summarizing already-summarized data, be aware of how stacked calculations work. Not all calculations return the correct result if you stack them.

For example, if you take the maximum of several subsets, then take the maximum of the maximums, you get the maximum of the entire set. But if you take the average of several subsets, then take the average of the averages, that can result in a different figure than the average of all the data.

To simplify such calculations when using continuous aggregates on top of continuous aggregates, you can use the hyperfunctions from TimescaleDB Toolkit, such as the statistical aggregates. These hyperfunctions are designed with a two-step aggregation pattern that allows you to roll them up into larger buckets. The first step creates a summary aggregate that can be rolled up, just as a maximum can be rolled up. You can store this aggregate in your continuous aggregate. Then, you can call an accessor function as a second step when you query from your continuous aggregate. This accessor takes the stored data from the summary aggregate and returns the final result.

For example, you can create an hourly continuous aggregate using percentile_agg over a hypertable, like this:

CREATE MATERIALIZED VIEW response_times_hourly
WITH (timescaledb.continuous)
AS SELECT
    time_bucket('1 h'::interval, ts) as bucket,
    api_id,
    avg(response_time_ms),
    percentile_agg(response_time_ms) as percentile_hourly
FROM response_times
GROUP BY 1, 2;

To then stack another daily continuous aggregate over it, you can use a rollup function, like this:

CREATE MATERIALIZED VIEW response_times_daily
WITH (timescaledb.continuous)
AS SELECT
    time_bucket('1 d'::interval, bucket) as bucket_daily,
    api_id,
    mean(rollup(percentile_hourly)) as mean,
    rollup(percentile_hourly) as percentile_daily
FROM response_times_hourly
GROUP BY 1, 2;

The mean function of the TimescaleDB Toolkit is used to calculate the concrete mean value of the rolled up values. The additional percentile_daily attribute contains the raw rolled up values, which can be used in an additional continuous aggregate on top of this continuous aggregate (for example a continuous aggregate for the daily values).

For more information and examples about using rollup functions to stack calculations, see the percentile approximation API documentation.

Restrictions

There are some restrictions when creating a continuous aggregate on top of another continuous aggregate. In most cases, these restrictions are in place to ensure valid time-bucketing:

• You can only create a continuous aggregate on top of a finalized continuous aggregate. This new finalized format is the default for all continuous aggregates created since TimescaleDB 2.7. If you need to create a continuous aggregate on top of a continuous aggregate in the old format, you need to migrate your continuous aggregate to the new format first.

• The time bucket of a continuous aggregate should be greater than or equal to the time bucket of the underlying continuous aggregate. It also needs to be a multiple of the underlying time bucket. For example, you can rebucket an hourly continuous aggregate into a new continuous aggregate with time buckets of 6 hours. You can't rebucket the hourly continuous aggregate into a new continuous aggregate with time buckets of 90 minutes, because 90 minutes is not a multiple of 1 hour.

• A continuous aggregate with a fixed-width time bucket can't be created on top of a continuous aggregate with a variable-width time bucket. Fixed-width time buckets are time buckets defined in seconds, minutes, hours, and days, because those time intervals are always the same length. Variable-width time buckets are time buckets defined in months or years, because those time intervals vary by the month or on leap years. This limitation prevents a case such as trying to rebucket monthly buckets into 61 day buckets, where there is no good mapping between time buckets for month combinations such as July/August (62 days).

  Note that even though weeks are fixed-width intervals, you can't use monthly or yearly time buckets on top of weekly time buckets for the same reason. The number of weeks in a month or year is usually not an integer.

  However, you can stack a variable-width time bucket on top of a fixed-width time bucket. For example, creating a monthly continuous aggregate on top of a daily continuous aggregate works, and is the one of the main use cases for this feature.

===== PAGE: https://docs.tigerdata.com/use-timescale/hypercore/secondary-indexes/ =====

Improve query and upsert performance

Real-time analytics applications require more than fast inserts and analytical queries. They also need high performance when retrieving individual records, enforcing constraints, or performing upserts, something that OLAP/columnar databases lack. This pages explains how to improve performance by segmenting and ordering data.

To improve query performance using indexes, see About indexes and Indexing data.

Segmenting and ordering data

To optimize query performance, TimescaleDB enables you to explicitly control the way your data is physically organized in the columnstore. By structuring data effectively, queries can minimize disk reads and execute more efficiently, using vectorized execution for parallel batch processing where possible.

• Group related data together to improve scan efficiency: organizing rows into logical segments ensures that queries filtering by a specific value only scan relevant data sections. For example, in the above, querying for a specific ID is particularly fast.
• Sort data within segments to accelerate range queries: defining a consistent order reduces the need for post-query sorting, making time-based queries and range scans more efficient.
• Reduce disk reads and maximize vectorized execution: a well-structured storage layout enables efficient batch processing (Single Instruction, Multiple Data, or SIMD vectorization) and parallel execution, optimizing query performance.

By combining segmentation and ordering, TimescaleDB ensures that columnar queries are not only fast but also resource-efficient, enabling high-performance real-time analytics.

Improve performance in the columnstore by segmenting and ordering data

Ordering data in the columnstore has a large impact on the compression ratio and performance of your queries. Rows that change over a dimension should be close to each other. As hypertables contain time-series data, they are partitioned by time. This makes the time column a perfect candidate for ordering your data since the measurements evolve as time goes on.

If you use orderby as your only columnstore setting, you get a good enough compression ratio to save a lot of storage and your queries are faster. However, if you only use orderby, you always have to access your data using the time dimension, then filter the rows returned on other criteria.

Accessing the data effectively depends on your use case and your queries. You segment data in the columnstore to match the way you want to access it. That is, in a way that makes it easier for your queries to fetch the right data at the right time. When you segment your data to access specific columns, your queries are optimized and yield even better performance.

For example, to access information about a single device with a specific device_id, you segment on the device_id column. This enables you to run analytical queries on compressed data in the columnstore much faster.

For example for the following hypertable:

CREATE TABLE metrics (
  time TIMESTAMPTZ,
  user_id INT,
  device_id INT,
  data JSONB
) WITH (
  tsdb.hypertable,
  tsdb.partition_column='time'
);

1. Execute a query on a regular hypertable

   1. Query your data

      SELECT device_id, AVG(cpu) AS avg_cpu, AVG(disk_io) AS avg_disk_io
      FROM metrics
      WHERE device_id = 5
      GROUP BY device_id;
      

      Gives the following result:

      device_id |      avg_cpu       |     avg_disk_io
      -----------+--------------------+---------------------
      5 | 0.4972598866221261 | 0.49820356730280524
      (1 row)
      Time: 177,399 ms
      

2. Execute a query on the same data segmented and ordered in the columnstore

   1. Control the way your data is ordered in the columnstore:

      ALTER TABLE metrics SET (
        timescaledb.enable_columnstore = true,
        timescaledb.orderby = 'time',
        timescaledb.segmentby = 'device_id'
      );
      

   2. Query your data

       select avg(cpu) from metrics where time >= '2024-03-01 00:00:00+01' and time < '2024-03-02 00:00:00+01';
      

      Gives the following result:

      device_id |      avg_cpu      |     avg_disk_io
      -----------+-------------------+---------------------
      5 | 0.497259886622126 | 0.49820356730280535
      (1 row)
      Time: 42,139 ms
      

   As you see, using orderby and segmentby not only reduces the amount of space taken by your data, but also vastly improves query speed.

The number of rows that are compressed together in a single batch (like the ones we see above) is 1000. If your chunk does not contain enough data to create big enough batches, your compression ratio will be reduced. This needs to be taken into account when you define your columnstore settings.

===== PAGE: https://docs.tigerdata.com/use-timescale/hypercore/modify-data-in-hypercore/ =====

Modify data in hypercore

Old API since TimescaleDB v2.20.0 TimescaleDB is optimized for fast updates on compressed data in the columnstore. To modify data in the columnstore, use standard SQL.

You set up hypercore to automatically convert data between the rowstore and columnstore when it reaches a certain age. After you have optimized data in the columnstore, you may need to modify it. For example, to make small changes, or backfill large amounts of data. You may even have to update the schema to accommodate these changes to the data.

This page shows you how to update small and large amounts of new data, and update the schema in the columnstore.

Prerequisites

To follow the procedure on this page you need to:

• Create a target Tiger Cloud service.

  This procedure also works for self-hosted TimescaleDB.

• Optimize your data for real-time analytics.

Modify small amounts of data

You can INSERT, UPDATE, and DELETE data in the columnstore, even if the data you are inserting has unique constraints. When you insert data into a chunk in the columnstore, a small amount of data is decompressed to allow a speculative insertion, and block any inserts that could violate the constraints.

When you DELETE whole segments of data, filter your deletes using the column you segment_by instead of separate deletes. This considerably increases performance.

Modify large amounts of data

If you need to modify or add a lot of data to a chunk in the columnstore, best practice is to stop any jobs moving chunks to the columnstore, convert the chunk back to the rowstore, then modify the data. After the update, convert the chunk to the columnstore and restart the jobs. This workflow is especially useful if you need to backfill old data.

1. Stop the jobs that are automatically adding chunks to the columnstore

   Retrieve the list of jobs from the timescaledb_information.jobs view to find the job you need to alter_job.

   SELECT alter_job(JOB_ID, scheduled => false);
   

2. Convert a chunk to update back to the rowstore

   CALL convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk');
   

3. Update the data in the chunk you added to the rowstore

   Best practice is to structure your INSERT statement to include appropriate partition key values, such as the timestamp. TimescaleDB adds the data to the correct chunk:

   INSERT INTO metrics (time, value)
   VALUES ('2025-01-01T00:00:00', 42);
   

4. Convert the updated chunks back to the columnstore

   CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk');
   

5. Restart the jobs that are automatically converting chunks to the columnstore

   SELECT alter_job(JOB_ID, scheduled => true);
   

Modify a table schema for data in the columnstore

You can modify the schema of a table in the columnstore. To do this, you need to:

1. Stop the jobs that are automatically adding chunks to the columnstore

   Retrieve the list of jobs from the timescaledb_information.jobs view to find the job you need to alter_job.

   SELECT alter_job(JOB_ID, scheduled => false);
   

2. Convert a chunk to update back to the rowstore

   CALL convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk');
   

3. Modify the schema:

   Possible modifications are:

   • Add a nullable column:

     ALTER TABLE <hypertable> ADD COLUMN <column_name> <datatype>;

   • Add a column with a default value and a NOT NULL constraint:

     ALTER TABLE <hypertable> ADD COLUMN <column_name> <datatype> NOT NULL DEFAULT <default_value>;

   • Rename a column:

     ALTER TABLE <hypertable> RENAME <column_name> TO <new_name>;

   • Drop a column:

     ALTER TABLE <hypertable> DROP COLUMN <column_name>;

   You cannot change the data type of an existing column.

4. Convert the updated chunks back to the columnstore

   CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk');
   

5. Restart the jobs that are automatically converting chunks to the columnstore

   SELECT alter_job(JOB_ID, scheduled => true);
   

===== PAGE: https://docs.tigerdata.com/use-timescale/hypercore/real-time-analytics-in-hypercore/ =====

Optimize your data for real-time analytics

Hypercore is the hybrid row-columnar storage engine in TimescaleDB used by hypertables. Traditional databases force a trade-off between fast inserts (row-based storage) and efficient analytics (columnar storage). Hypercore eliminates this trade-off, allowing real-time analytics without sacrificing transactional capabilities.

Hypercore dynamically stores data in the most efficient format for its lifecycle:

• Row-based storage for recent data: the most recent chunk (and possibly more) is always stored in the rowstore, ensuring fast inserts, updates, and low-latency single record queries. Additionally, row-based storage is used as a writethrough for inserts and updates to columnar storage.
• Columnar storage for analytical performance: chunks are automatically compressed into the columnstore, optimizing storage efficiency and accelerating analytical queries.

Unlike traditional columnar databases, hypercore allows data to be inserted or modified at any stage, making it a flexible solution for both high-ingest transactional workloads and real-time analytics—within a single database.

When you convert chunks from the rowstore to the columnstore, multiple records are grouped into a single row. The columns of this row hold an array-like structure that stores all the data. For example, data in the following rowstore chunk:

Timestamp	Device ID	Device Type	CPU	Disk IO
12:00:01	A	SSD	70.11	13.4
12:00:01	B	HDD	69.70	20.5
12:00:02	A	SSD	70.12	13.2
12:00:02	B	HDD	69.69	23.4
12:00:03	A	SSD	70.14	13.0
12:00:03	B	HDD	69.70	25.2

Is converted and compressed into arrays in a row in the columnstore:

Timestamp	Device ID	Device Type	CPU	Disk IO
[12:00:01, 12:00:01, 12:00:02, 12:00:02, 12:00:03, 12:00:03]	[A, B, A, B, A, B]	[SSD, HDD, SSD, HDD, SSD, HDD]	[70.11, 69.70, 70.12, 69.69, 70.14, 69.70]	[13.4, 20.5, 13.2, 23.4, 13.0, 25.2]

Because a single row takes up less disk space, you can reduce your chunk size by up to 98%, and can also speed up your queries. This saves on storage costs, and keeps your queries operating at lightning speed.

For an in-depth explanation of how hypertables and hypercore work, see the Data model.

This page shows you how to get the best results when you set a policy to automatically convert chunks in a hypertable from the rowstore to the columnstore.

Prerequisites

To follow the steps on this page:

• Create a target Tiger Cloud service with real-time analytics enabled.

  You need your connection details.

The code samples in this page use the crypto_sample.zip data from this key features tutorial.

Optimize your data with columnstore policies

The compression ratio and query performance of data in the columnstore is dependent on the order and structure of your data. Rows that change over a dimension should be close to each other. With time-series data, you orderby the time dimension. For example, Timestamp:

Timestamp	Device ID	Device Type	CPU	Disk IO
12:00:01	A	SSD	70.11	13.4

This ensures that records are compressed and accessed in the same order. However, you would always have to access the data using the time dimension, then filter all the rows using other criteria. To make your queries more efficient, you segment your data based on the following:

• The way you want to access it. For example, to rapidly access data about a single device, you segmentby the Device ID column. This enables you to run much faster analytical queries on data in the columnstore.
• The compression rate you want to achieve. The lower the cardinality of the segmentby column, the better compression results you get.

When TimescaleDB converts a chunk to the columnstore, it automatically creates a different schema for your data. It also creates and uses custom indexes to incorporate the segmentby and orderby parameters when you write to and read from the columnstore.

To set up your hypercore automation:

1. Connect to your Tiger Cloud service

   In Tiger Cloud Console open an SQL editor. You can also connect to your service using psql.

2. Enable columnstore on a hypertable

   Create a hypertable for your time-series data using CREATE TABLE. For efficient queries on data in the columnstore, remember to segmentby the column you will use most often to filter your data. For example:

   • Use CREATE TABLE for a hypertable

     CREATE TABLE crypto_ticks (
        "time" TIMESTAMPTZ,
        symbol TEXT,
        price DOUBLE PRECISION,
        day_volume NUMERIC
     ) WITH (
       tsdb.hypertable,
       tsdb.partition_column='time',
       tsdb.segmentby='symbol',
       tsdb.orderby='time DESC'
     );
     

     If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

   • Use ALTER MATERIALIZED VIEW for a continuous aggregate

     ALTER MATERIALIZED VIEW assets_candlestick_daily set (
        timescaledb.enable_columnstore = true,
        timescaledb.segmentby = 'symbol' );
     

     Before you say huh, a continuous aggregate is a specialized hypertable.

3. Add a policy to convert chunks to the columnstore at a specific time interval

   Create a columnstore_policy that automatically converts chunks in a hypertable to the columnstore at a specific time interval. For example, convert yesterday's crypto trading data to the columnstore:

   CALL add_columnstore_policy('crypto_ticks', after => INTERVAL '1d');
   

   TimescaleDB is optimized for fast updates on compressed data in the columnstore. To modify data in the columnstore, use standard SQL.

4. Check the columnstore policy

   1. View your data space saving:

      When you convert data to the columnstore, as well as being optimized for analytics, it is compressed by more than 90%. This helps you save on storage costs and keeps your queries operating at lightning speed. To see the amount of space saved:

      SELECT
        pg_size_pretty(before_compression_total_bytes) as before,
        pg_size_pretty(after_compression_total_bytes) as after
      FROM hypertable_columnstore_stats('crypto_ticks');
      

      You see something like:

      before	after
      194 MB	24 MB

   2. View the policies that you set or the policies that already exist:

      SELECT * FROM timescaledb_information.jobs
      WHERE proc_name='policy_compression';
      

      See timescaledb_information.jobs.

5. Pause a columnstore policy

   SELECT * FROM timescaledb_information.jobs where
      proc_name = 'policy_compression' AND relname = 'crypto_ticks'
   
   -- Select the JOB_ID from the results
   
   SELECT alter_job(JOB_ID, scheduled => false);
   

   See alter_job.

6. Restart a columnstore policy

   SELECT alter_job(JOB_ID, scheduled => true);
   

   See alter_job.

7. Remove a columnstore policy

   CALL remove_columnstore_policy('crypto_ticks');
   

   See remove_columnstore_policy.

8. Disable columnstore

   If your table has chunks in the columnstore, you have to convert the chunks back to the rowstore before you disable the columnstore.

   ALTER TABLE crypto_ticks SET (timescaledb.enable_columnstore = false);
   

   See alter_table_hypercore.

Reference

For integers, timestamps, and other integer-like types, data is compressed using delta encoding, delta-of-delta, simple-8b, and run-length encoding. For columns with few repeated values, XOR-based and dictionary compression is used. For all other types, dictionary compression is used.

===== PAGE: https://docs.tigerdata.com/use-timescale/hypercore/compression-methods/ =====

About compression methods

Depending on the data type that is compressed when your data is converted from the rowstore to the columnstore, TimescaleDB uses the following compression algorithms:

• Integers, timestamps, boolean and other integer-like types: a combination of the following compression methods is used: delta encoding, delta-of-delta, simple-8b, and run-length encoding.
• Columns that do not have a high amount of repeated values: XOR-based compression with some dictionary compression.
• All other types: dictionary compression.

This page gives an in-depth explanation of the compression methods used in hypercore.

Integer compression

For integers, timestamps, and other integer-like types TimescaleDB uses a combination of delta encoding, delta-of-delta, simple 8-b, and run-length encoding.

The simple-8b compression method has been extended so that data can be decompressed in reverse order. Backward scanning queries are common in time-series workloads. This means that these types of queries run much faster.

Delta encoding

Delta encoding reduces the amount of information required to represent a data object by only storing the difference, sometimes referred to as the delta, between that object and one or more reference objects. These algorithms work best where there is a lot of redundant information, and it is often used in workloads like versioned file systems. For example, this is how Dropbox keeps your files synchronized. Applying delta-encoding to time-series data means that you can use fewer bytes to represent a data point, because you only need to store the delta from the previous data point.

For example, imagine you had a dataset that collected CPU, free memory, temperature, and humidity over time. If you time column was stored as an integer value, like seconds since UNIX epoch, your raw data would look a little like this:

time	cpu	mem_free_bytes	temperature	humidity
2023-04-01 10:00:00	82	1,073,741,824	80	25
2023-04-01 10:00:05	98	858,993,459	81	25
2023-04-01 10:00:10	98	858,904,583	81	25

With delta encoding, you only need to store how much each value changed from the previous data point, resulting in smaller values to store. So after the first row, you can represent subsequent rows with less information, like this:

time	cpu	mem_free_bytes	temperature	humidity
2023-04-01 10:00:00	82	1,073,741,824	80	25
5 seconds	16	-214,748,365	1	0
5 seconds	0	-88,876	0	0

Applying delta encoding to time-series data takes advantage of the fact that most time-series datasets are not random, but instead represent something that is slowly changing over time. The storage savings over millions of rows can be substantial, especially if the value changes very little, or doesn't change at all.

Delta-of-delta encoding

Delta-of-delta encoding takes delta encoding one step further and applies delta-encoding over data that has previously been delta-encoded. With time-series datasets where data collection happens at regular intervals, you can apply delta-of-delta encoding to the time column, which results in only needing to store a series of zeroes.

In other words, delta encoding stores the first derivative of the dataset, while delta-of-delta encoding stores the second derivative of the dataset.

Applied to the example dataset from earlier, delta-of-delta encoding results in this:

time	cpu	mem_free_bytes	temperature	humidity
2020-04-01 10:00:00	82	1,073,741,824	80	25
5 seconds	16	-214,748,365	1	0
0 seconds	0	-88,876	0	0

In this example, delta-of-delta further compresses 5 seconds in the time column down to 0 for every entry in the time column after the second row, because the five second gap remains constant for each entry. Note that you see two entries in the table before the delta-delta 0 values, because you need two deltas to compare.

This compresses a full timestamp of 8 bytes, or 64 bits, down to just a single bit, resulting in 64x compression.

Simple-8b

With delta and delta-of-delta encoding, you can significantly reduce the number of digits you need to store. But you still need an efficient way to store the smaller integers. The previous examples used a standard integer datatype for the time column, which needs 64 bits to represent the value of 0 when delta-delta encoded. This means that even though you are only storing the integer 0, you are still consuming 64 bits to store it, so you haven't actually saved anything.

Simple-8b is one of the simplest and smallest methods of storing variable-length integers. In this method, integers are stored as a series of fixed-size blocks. For each block, every integer within the block is represented by the minimal bit-length needed to represent the largest integer in that block. The first bits of each block denotes the minimum bit-length for the block.

This technique has the advantage of only needing to store the length once for a given block, instead of once for each integer. Because the blocks are of a fixed size, you can infer the number of integers in each block from the size of the integers being stored.

For example, if you wanted to store a temperature that changed over time, and you applied delta encoding, you might end up needing to store this set of integers:

temperature (deltas)
1
10
11
13
9
100
22
11

With a block size of 10 digits, you could store this set of integers as two blocks: one block storing 5 2-digit numbers, and a second block storing 3 3-digit numbers, like this:

<CodeBlock canCopy={false} showLineNumbers={false} children={{2: [01, 10, 11, 13, 09]} {3: [100, 022, 011]}} />

In this example, both blocks store about 10 digits worth of data, even though some of the numbers have to be padded with a leading 0. You might also notice that the second block only stores 9 digits, because 10 is not evenly divisible by 3.

Simple-8b works in this way, except it uses binary numbers instead of decimal, and it usually uses 64-bit blocks. In general, the longer the integer, the fewer number of integers that can be stored in each block.

Run-length encoding

Simple-8b compresses integers very well, however, if you have a large number of repeats of the same value, you can get even better compression with run-length encoding. This method works well for values that don't change very often, or if an earlier transformation removes the changes.

Run-length encoding is one of the classic compression algorithms. For time-series data with billions of contiguous zeroes, or even a document with a million identically repeated strings, run-length encoding works incredibly well.

For example, if you wanted to store a temperature that changed minimally over time, and you applied delta encoding, you might end up needing to store this set of integers:

temperature (deltas)
11
12
12
12
12
12
12
1
12
12
12
12

For values like these, you do not need to store each instance of the value, but rather how long the run, or number of repeats, is. You can store this set of numbers as {run; value} pairs like this:

<CodeBlock canCopy={false} showLineNumbers={false} children={{1; 11}, {6; 12}, {1; 1}, {4; 12}} />

This technique uses 11 digits of storage (1, 1, 1, 6, 1, 2, 1, 1, 4, 1, 2), rather than 23 digits that an optimal series of variable-length integers requires (11, 12, 12, 12, 12, 12, 12, 1, 12, 12, 12, 12).

Run-length encoding is also used as a building block for many more advanced algorithms, such as Simple-8b RLE, which is an algorithm that combines run-length and Simple-8b techniques. TimescaleDB implements a variant of Simple-8b RLE. This variant uses different sizes to standard Simple-8b, in order to handle 64-bit values, and RLE.

Floating point compression

For columns that do not have a high amount of repeated values, TimescaleDB uses XOR-based compression.

The standard XOR-based compression method has been extended so that data can be decompressed in reverse order. Backward scanning queries are common in time-series workloads. This means that queries that use backwards scans run much faster.

XOR-based compression

Floating point numbers are usually more difficult to compress than integers. Fixed-length integers often have leading zeroes, but floating point numbers usually use all of their available bits, especially if they are converted from decimal numbers, which can't be represented precisely in binary.

Techniques like delta-encoding don't work well for floats, because they do not reduce the number of bits sufficiently. This means that most floating-point compression algorithms tend to be either complex and slow, or truncate significant digits. One of the few simple and fast lossless floating-point compression algorithms is XOR-based compression, built on top of Facebook's Gorilla compression.

XOR is the binary function exclusive or. In this algorithm, successive floating point numbers are compared with XOR, and a difference results in a bit being stored. The first data point is stored without compression, and subsequent data points are represented using their XOR'd values.

Data-agnostic compression

For values that are not integers or floating point, TimescaleDB uses dictionary compression.

Dictionary compression

One of the earliest lossless compression algorithms, dictionary compression is the basis of many popular compression methods. Dictionary compression can also be found in areas outside of computer science, such as medical coding.

Instead of storing values directly, dictionary compression works by making a list of the possible values that can appear, and then storing an index into a dictionary containing the unique values. This technique is quite versatile, can be used regardless of data type, and works especially well when you have a limited set of values that repeat frequently.

For example, if you had the list of temperatures shown earlier, but you wanted an additional column storing a city location for each measurement, you might have a set of values like this:

City
New York
San Francisco
San Francisco
Los Angeles

Instead of storing all the city names directly, you can instead store a dictionary, like this:

<CodeBlock canCopy={false} showLineNumbers={false} children={{0: "New York", 1: "San Francisco", 2: "Los Angeles",}} />

You can then store just the indices in your column, like this:

City
0
1
1
2

For a dataset with a lot of repetition, this can offer significant compression. In the example, each city name is on average 11 bytes in length, while the indices are never going to be more than 4 bytes long, reducing space usage nearly 3 times. In TimescaleDB, the list of indices is compressed even further with the Simple-8b+RLE method, making the storage cost even smaller.

Dictionary compression doesn't always result in savings. If your dataset doesn't have a lot of repeated values, then the dictionary is the same size as the original data. TimescaleDB automatically detects this case, and falls back to not using a dictionary in that scenario.

===== PAGE: https://docs.tigerdata.com/use-timescale/compression/modify-a-schema/ =====

Schema modifications

You can modify the schema of compressed hypertables in recent versions of TimescaleDB.

Schema modification	Before TimescaleDB 2.1	TimescaleDB 2.1 to 2.5	TimescaleDB 2.6 and above
Add a nullable column	❌	✅	✅
Add a column with a default value and a NOT NULL constraint	❌	❌	✅
Rename a column	❌	✅	✅
Drop a column	❌	❌	✅
Change the data type of a column	❌	❌	❌

To perform operations that aren't supported on compressed hypertables, first decompress the table.

Add a nullable column

To add a nullable column:

ALTER TABLE <hypertable> ADD COLUMN <column_name> <datatype>;

For example:

ALTER TABLE conditions ADD COLUMN device_id integer;

Note that adding constraints to the new column is not supported before TimescaleDB v2.6.

Add a column with a default value and a NOT NULL constraint

To add a column with a default value and a not-null constraint:

ALTER TABLE <hypertable> ADD COLUMN <column_name> <datatype>
    NOT NULL DEFAULT <default_value>;

For example:

ALTER TABLE conditions ADD COLUMN device_id integer
    NOT NULL DEFAULT 1;

Rename a column

To rename a column:

ALTER TABLE <hypertable> RENAME <column_name> TO <new_name>;

For example:

ALTER TABLE conditions RENAME device_id TO devid;

Drop a column

You can drop a column from a compressed hypertable, if the column is not an orderby or segmentby column. To drop a column:

ALTER TABLE <hypertable> DROP COLUMN <column_name>;

For example:

ALTER TABLE conditions DROP COLUMN temperature;

===== PAGE: https://docs.tigerdata.com/use-timescale/compression/decompress-chunks/ =====

Decompression

Old API since TimescaleDB v2.18.0 Replaced by convert_to_rowstore.

When compressing your data, you can reduce the amount of storage space used. But you should always leave some additional storage capacity. This gives you the flexibility to decompress chunks when necessary, for actions such as bulk inserts.

This section describes commands to use for decompressing chunks. You can filter by time to select the chunks you want to decompress.

Decompress chunks manually

Before decompressing chunks, stop any compression policy on the hypertable you are decompressing. The database automatically recompresses your chunks in the next scheduled job. If you accumulate a large amount of chunks that need to be compressed, the troubleshooting guide shows how to compress a backlog of chunks. For more information on how to stop and run compression policies using alter_job(), see the API reference.

There are several methods for selecting chunks and decompressing them.

Decompress individual chunks

To decompress a single chunk by name, run this command:

SELECT decompress_chunk('_timescaledb_internal.<chunk_name>');

where, <chunk_name> is the name of the chunk you want to decompress.

Decompress chunks by time

To decompress a set of chunks based on a time range, you can use the output of show_chunks to decompress each one:

SELECT decompress_chunk(c, true)
    FROM show_chunks('table_name', older_than, newer_than) c;

For more information about the decompress_chunk function, see the decompress_chunk API reference.

Decompress chunks on more precise constraints

If you want to use more precise matching constraints, for example space partitioning, you can construct a command like this:

SELECT tableoid::regclass FROM metrics
  WHERE time = '2000-01-01' AND device_id = 1
  GROUP BY tableoid;

                 tableoid
------------------------------------------
 _timescaledb_internal._hyper_72_37_chunk

===== PAGE: https://docs.tigerdata.com/use-timescale/compression/compression-on-continuous-aggregates/ =====

Convert continuous aggregates to the columnstore

Continuous aggregates are often used to downsample historical data. If the data is only used for analytical queries and never modified, you can compress the aggregate to save on storage.

Old API since TimescaleDB v2.18.0 Replaced by Convert continuous aggregates to the columnstore.

Before version 2.18.1, you can't refresh the compressed regions of a continuous aggregate. To avoid conflicts between compression and refresh, make sure you set compress_after to a larger interval than the start_offset of your refresh policy.

Compression on continuous aggregates works similarly to compression on hypertables. When compression is enabled and no other options are provided, the segment_by value will be automatically set to the group by columns of the continuous aggregate and the time_bucket column will be used as the order_by column in the compression configuration.

Enable compression on continuous aggregates

You can enable and disable compression on continuous aggregates by setting the compress parameter when you alter the view.

Enabling and disabling compression on continuous aggregates

1. For an existing continuous aggregate, at the psql prompt, enable compression:

   ALTER MATERIALIZED VIEW cagg_name set (timescaledb.compress = true);
   

2. Disable compression:

   ALTER MATERIALIZED VIEW cagg_name set (timescaledb.compress = false);
   

Disabling compression on a continuous aggregate fails if there are compressed chunks associated with the continuous aggregate. In this case, you need to decompress the chunks, and then drop any compression policy on the continuous aggregate, before you disable compression. For more detailed information, see the decompress chunks section:

SELECT decompress_chunk(c, true) FROM show_chunks('cagg_name') c;

Compression policies on continuous aggregates

Before setting up a compression policy on a continuous aggregate, you should set up a refresh policy. The compression policy interval should be set so that actively refreshed regions are not compressed. This is to prevent refresh policies from failing. For example, consider a refresh policy like this:

SELECT add_continuous_aggregate_policy('cagg_name',
  start_offset => INTERVAL '30 days',
  end_offset => INTERVAL '1 day',
  schedule_interval => INTERVAL '1 hour');

With this kind of refresh policy, the compression policy needs the compress_after parameter greater than the start_offset parameter of the continuous aggregate policy:

SELECT add_compression_policy('cagg_name', compress_after=>'45 days'::interval);

===== PAGE: https://docs.tigerdata.com/use-timescale/compression/manual-compression/ =====

Manual compression

In most cases, an automated compression policy is sufficient to automatically compress your chunks. However, if you want more control, you can also use manual synchronous compression of specific chunks.

Before you start, you need a list of chunks to compress. In this example, you use a hypertable called example, and compress chunks older than three days.

Selecting chunks to compress

1. At the psql prompt, select all chunks in the table example that are older than three days:

   SELECT show_chunks('example', older_than => INTERVAL '3 days');
   

2. This returns a list of chunks. Take note of the chunks' names:

   	show_chunks
   1	_timescaledb_internal_hyper_1_2_chunk
   2	_timescaledb_internal_hyper_1_3_chunk

When you are happy with the list of chunks, you can use the chunk names to manually compress each one.

Compressing chunks manually

1. At the psql prompt, compress the chunk:

   SELECT compress_chunk( '<chunk_name>');
   

2. Check the results of the compression with this command:

   SELECT *
   FROM chunk_compression_stats('example');
   

   The results show the chunks for the given hypertable, their compression status, and some other statistics:

   chunk_schema	chunk_name	compression_status	before_compression_table_bytes	before_compression_index_bytes	before_compression_toast_bytes	before_compression_total_bytes	after_compression_table_bytes	after_compression_index_bytes	after_compression_toast_bytes	after_compression_total_bytes	node_name
   _timescaledb_internal	_hyper_1_1_chunk	Compressed	8192 bytes	16 kB	8192 bytes	32 kB	8192 bytes	16 kB	8192 bytes	32 kB
   _timescaledb_internal	_hyper_1_20_chunk	Uncompressed

3. Repeat for all chunks you want to compress.

Manually compress chunks in a single command

Alternatively, you can select the chunks and compress them in a single command by using the output of the show_chunks command to compress each one. For example, use this command to compress chunks between one and three weeks old if they are not already compressed:

SELECT compress_chunk(i, if_not_compressed => true)
    FROM show_chunks(
        'example',
        now()::timestamp - INTERVAL '1 week',
        now()::timestamp - INTERVAL '3 weeks'
    ) i;

Roll up uncompressed chunks when compressing

In TimescaleDB v2.9 and later, you can roll up multiple uncompressed chunks into a previously compressed chunk as part of your compression procedure. This allows you to have much smaller uncompressed chunk intervals, which reduces the disk space used for uncompressed data. For example, if you have multiple smaller uncompressed chunks in your data, you can roll them up into a single compressed chunk.

To roll up your uncompressed chunks into a compressed chunk, alter the compression settings to set the compress chunk time interval and run compression operations to roll up the chunks while compressing.

The default setting of compress_orderby is 'time DESC' (the descending or DESC command is used to sort the data returned in ascending order), which causes chunks to be re-compressed many times during the rollup, possibly leading to a steep performance penalty. Set timescaledb.compress_orderby = 'time ASC' to avoid this penalty.

ALTER TABLE example SET (timescaledb.compress_chunk_time_interval = '<time_interval>',
                            timescaledb.compress_orderby = 'time ASC');
SELECT compress_chunk(c, if_not_compressed => true)
    FROM show_chunks(
        'example',
        now()::timestamp - INTERVAL '1 week'
    ) c;

The time interval you choose must be a multiple of the uncompressed chunk interval. For example, if your uncompressed chunk interval is one week, your <time_interval> of the compressed chunk could be two weeks or six weeks, but not one month.

===== PAGE: https://docs.tigerdata.com/use-timescale/compression/about-compression/ =====

About compression

Old API since TimescaleDB v2.18.0 Replaced by hypercore.

Compressing your time-series data allows you to reduce your chunk size by more than 90%. This saves on storage costs, and keeps your queries operating at lightning speed.

When you enable compression, the data in your hypertable is compressed chunk by chunk. When the chunk is compressed, multiple records are grouped into a single row. The columns of this row hold an array-like structure that stores all the data. This means that instead of using lots of rows to store the data, it stores the same data in a single row. Because a single row takes up less disk space than many rows, it decreases the amount of disk space required, and can also speed up your queries.

For example, if you had a table with data that looked a bit like this:

Timestamp	Device ID	Device Type	CPU	Disk IO
12:00:01	A	SSD	70.11	13.4
12:00:01	B	HDD	69.70	20.5
12:00:02	A	SSD	70.12	13.2
12:00:02	B	HDD	69.69	23.4
12:00:03	A	SSD	70.14	13.0
12:00:03	B	HDD	69.70	25.2

You can convert this to a single row in array form, like this:

Timestamp	Device ID	Device Type	CPU	Disk IO
[12:00:01, 12:00:01, 12:00:02, 12:00:02, 12:00:03, 12:00:03]	[A, B, A, B, A, B]	[SSD, HDD, SSD, HDD, SSD, HDD]	[70.11, 69.70, 70.12, 69.69, 70.14, 69.70]	[13.4, 20.5, 13.2, 23.4, 13.0, 25.2]

This section explains how to enable native compression, and then goes into detail on the most important settings for compression, to help you get the best possible compression ratio.

Key aspects of compression

Every table has a different schema but they do share some commonalities that you need to think about.

Consider the table metrics with the following attributes:

Column	Type	Collation	Nullable	Default
time	timestamp with time zone		not null
device_id	integer		not null
device_type	integer		not null
cpu	double precision
disk_io	double precision

All hypertables have a primary dimension which is used to partition the table into chunks. The primary dimension is given when the hypertable is created. In the example below, you can see a classic time-series use case with a time column as the primary dimension. In addition, there are two columns cpu and disk_io containing the values that are captured over time, and a column device_id for the device that captured the values. Columns can be used in a few different ways:

• You can use values in a column as a lookup key, in the example above device_id is a typical example of such a column.
• You can use a column for partitioning a table. This is typically a time column like time in the example above, but it is possible to partition the table using other types as well.
• You can use a column as a filter to narrow down on what data you select. The column device_type is an example of where you can decide to look at, for example, only solid state drives (SSDs). The remaining columns are typically the values or metrics you are collecting. These are typically aggregated or presented in other ways. The columns cpu and disk_io are typical examples of such columns.

<CodeBlock canCopy={false} showLineNumbers={false} children={SELECT avg(cpu), sum(disk_io) FROM metrics WHERE device_type = ‘SSD’ AND time >= now() - ‘1 day’::interval;} />

When chunks are compressed in a hypertable, data stored in them is reorganized and stored in column-order rather than row-order. As a result, it is not possible to use the same uncompressed schema version of the chunk and a different schema must be created. This is automatically handled by TimescaleDB, but it has a few implications: The compression ratio and query performance is very dependent on the order and structure of the compressed data, so some considerations are needed when setting up compression. Indexes on the hypertable cannot always be used in the same manner for the compressed data.

Indexes set on the hypertable are used only on chunks containing uncompressed data. TimescaleDB creates and uses custom indexes to incorporate the segmentby and orderby parameters during compression which are used when reading compressed data. More on this in the next section.

Based on the previous schema, filtering of data should happen over a certain time period and analytics are done on device granularity. This pattern of data access lends itself to organizing the data layout suitable for compression.

Ordering and segmenting.

Ordering the data will have a great impact on the compression ratio and performance of your queries. Rows that change over a dimension should be close to each other. Since we are mostly dealing with time-series data, time dimension is a great candidate. Most of the time data changes in a predictable fashion, following a certain trend. We can exploit this fact to encode the data so it takes less space to store. For example, if you order the records over time, they will get compressed in that order and subsequently also accessed in the same order.

Using the following configuration setup on our example table: <CodeBlock canCopy={false} showLineNumbers={false} children={ALTER TABLE metrics SET (timescaledb.compress, timescaledb.compress_orderby='time');} />

would produce the following data layout.

|Timestamp|Device ID|Device Type|CPU|Disk IO| |-|-|-|-| |[12:00:01, 12:00:01, 12:00:02, 12:00:02, 12:00:03, 12:00:03]|[A, B, A, B, A, B]|[SSD, HDD, SSD, HDD, SSD, HDD]|[70.11, 69.70, 70.12, 69.69, 70.14, 69.70]|[13.4, 20.5, 13.2, 23.4, 13.0, 25.2]|

time column is used for ordering data, which makes filtering it using time column much more efficient.

<CodeBlock canCopy={false} showLineNumbers={false} children={` postgres=# select avg(cpu) from metrics where time >= '2024-03-01 00:00:00+01' and time < '2024-03-02 00:00:00+01'; avg

0.4996848437842719 (1 row) Time: 87,218 ms postgres=# ALTER TABLE metrics SET ( timescaledb.compress, timescaledb.compress_segmentby = 'device_id', timescaledb.compress_orderby='time' ); ALTER TABLE Time: 6,607 ms postgres=# SELECT compress_chunk(c) FROM show_chunks('metrics') c; compress_chunk

_timescaledb_internal._hyper_2_4_chunk _timescaledb_internal._hyper_2_5_chunk _timescaledb_internal._hyper_2_6_chunk (3 rows) Time: 3070,626 ms (00:03,071) postgres=# select avg(cpu) from metrics where time >= '2024-03-01 00:00:00+01' and time < '2024-03-02 00:00:00+01'; avg

0.49968484378427 (1 row) Time: 45,384 ms `} />

This makes the time column a perfect candidate for ordering your data since the measurements evolve as time goes on. If you were to use that as your only compression setting, you would most likely get a good enough compression ratio to save a lot of storage. However, accessing the data effectively depends on your use case and your queries. With this setup, you would always have to access the data by using the time dimension and subsequently filter all the rows based on any other criteria.

Segmenting the compressed data should be based on the way you access the data. Basically, you want to segment your data in such a way that you can make it easier for your queries to fetch the right data at the right time. That is to say, your queries should dictate how you segment the data so they can be optimized and yield even better query performance.

For example, If you want to access a single device using a specific device_id value (either all records or maybe for a specific time range), you would need to filter all those records one by one during row access time. To get around this, you can use device_id column for segmenting. This would allow you to run analytical queries on compressed data much faster if you are looking for specific device IDs.

Consider the following query:

<CodeBlock canCopy={false} showLineNumbers={false} children={SELECT device_id, AVG(cpu) AS avg_cpu, AVG(disk_io) AS avg_disk_io FROM metrics WHERE device_id = 5 GROUP BY device_id;} />

As you can see, the query does a lot of work based on the device_id identifier by grouping all its values together. We can use this fact to speed up these types of queries by setting up compression to segment the data around the values in this column.

Using the following configuration setup on our example table: <CodeBlock canCopy={false} showLineNumbers={false} children={ALTER TABLE metrics SET ( timescaledb.compress, timescaledb.compress_segmentby='device_id', timescaledb.compress_orderby='time' );} />

would produce the following data layout.

time	device_id	device_type	cpu	disk_io	energy_consumption
[12:00:02, 12:00:01]	1	[SSD,SSD]	[88.2, 88.6]	[20, 25]	[0.8, 0.85]
[12:00:02, 12:00:01]	2	[HDD,HDD]	[300.5, 299.1]	[30, 40]	[0.9, 0.95]
...	...	...	...	...	...

Segmenting column device_id is used for grouping data points together based on the value of that column. This makes accessing a specific device much more efficient.

<CodeBlock canCopy={false} showLineNumbers={false} children={` postgres=# \timing Timing is on. postgres=# SELECT device_id, AVG(cpu) AS avg_cpu, AVG(disk_io) AS avg_disk_io FROM metrics WHERE device_id = 5 GROUP BY device_id; device_id | avg_cpu | avg_disk_io -----------+--------------------+--------------------- 5 | 0.4972598866221261 | 0.49820356730280524 (1 row) Time: 177,399 ms postgres=# ALTER TABLE metrics SET ( timescaledb.compress, timescaledb.compress_segmentby = 'device_id', timescaledb.compress_orderby='time' ); ALTER TABLE Time: 6,607 ms postgres=# SELECT compress_chunk(c) FROM show_chunks('metrics') c; compress_chunk

_timescaledb_internal._hyper_2_4_chunk _timescaledb_internal._hyper_2_5_chunk _timescaledb_internal._hyper_2_6_chunk (3 rows) Time: 3070,626 ms (00:03,071) postgres=# SELECT device_id, AVG(cpu) AS avg_cpu, AVG(disk_io) AS avg_disk_io FROM metrics WHERE device_id = 5 GROUP BY device_id; device_id | avg_cpu | avg_disk_io -----------+-------------------+--------------------- 5 | 0.497259886622126 | 0.49820356730280535 (1 row) Time: 42,139 ms `} />

Number of rows that are compressed together in a single batch (like the ones we see above) is 1000. If your chunk does not contain enough data to create big enough batches, your compression ratio will be reduced. This needs to be taken into account when defining your compression settings.

===== PAGE: https://docs.tigerdata.com/use-timescale/compression/compression-design/ =====

Designing your database for compression

Old API since TimescaleDB v2.18.0 Replaced by hypercore.

Time-series data can be unique, in that it needs to handle both shallow and wide queries, such as "What's happened across the deployment in the last 10 minutes," and deep and narrow, such as "What is the average CPU usage for this server over the last 24 hours." Time-series data usually has a very high rate of inserts as well; hundreds of thousands of writes per second can be very normal for a time-series dataset. Additionally, time-series data is often very granular, and data is collected at a higher resolution than many other datasets. This can result in terabytes of data being collected over time.

All this means that if you need great compression rates, you probably need to consider the design of your database, before you start ingesting data. This section covers some of the things you need to take into consideration when designing your database for maximum compression effectiveness.

Compressing data

TimescaleDB is built on Postgres which is, by nature, a row-based database. Because time-series data is accessed in order of time, when you enable compression, TimescaleDB converts many wide rows of data into a single row of data, called an array form. This means that each field of that new, wide row stores an ordered set of data comprising the entire column.

For example, if you had a table with data that looked a bit like this:

Timestamp	Device ID	Status Code	Temperature
12:00:01	A	0	70.11
12:00:01	B	0	69.70
12:00:02	A	0	70.12
12:00:02	B	0	69.69
12:00:03	A	0	70.14
12:00:03	B	4	69.70

You can convert this to a single row in array form, like this:

Timestamp	Device ID	Status Code	Temperature
[12:00:01, 12:00:01, 12:00:02, 12:00:02, 12:00:03, 12:00:03]	[A, B, A, B, A, B]	[0, 0, 0, 0, 0, 4]	[70.11, 69.70, 70.12, 69.69, 70.14, 69.70]

Even before you compress any data, this format immediately saves storage by reducing the per-row overhead. Postgres typically adds a small number of bytes of overhead per row. So even without any compression, the schema in this example is now smaller on disk than the previous format.

This format arranges the data so that similar data, such as timestamps, device IDs, or temperature readings, is stored contiguously. This means that you can then use type-specific compression algorithms to compress the data further, and each array is separately compressed. For more information about the compression methods used, see the compression methods section.

When the data is in array format, you can perform queries that require a subset of the columns very quickly. For example, if you have a query like this one, that asks for the average temperature over the past day:

<CodeBlock canCopy={false} showLineNumbers={false} children={SELECT time_bucket(‘1 minute’, timestamp) as minute AVG(temperature) FROM table WHERE timestamp > now() - interval ‘1 day’ ORDER BY minute DESC GROUP BY minute;} />

The query engine can fetch and decompress only the timestamp and temperature columns to efficiently compute and return these results.

Finally, TimescaleDB uses non-inline disk pages to store the compressed arrays. This means that the in-row data points to a secondary disk page that stores the compressed array, and the actual row in the main table becomes very small, because it is now just pointers to the data. When data stored like this is queried, only the compressed arrays for the required columns are read from disk, further improving performance by reducing disk reads and writes.

Querying compressed data

In the previous example, the database has no way of knowing which rows need to be fetched and decompressed to resolve a query. For example, the database can't easily determine which rows contain data from the past day, as the timestamp itself is in a compressed column. You don't want to have to decompress all the data in a chunk, or even an entire hypertable, to determine which rows are required.

TimescaleDB automatically includes more information in the row and includes additional groupings to improve query performance. When you compress a hypertable, either manually or through a compression policy, it can help to specify an ORDER BY column.

ORDER BY columns specify how the rows that are part of a compressed batch are ordered. For most time-series workloads, this is by timestamp, so if you don't specify an ORDER BY column, TimescaleDB defaults to using the time column. You can also specify additional dimensions, such as location.

For each ORDER BY column, TimescaleDB automatically creates additional columns that store the minimum and maximum value of that column. This way, the query planner can look at the range of timestamps in the compressed column, without having to do any decompression, and determine whether the row could possibly match the query.

When you compress your hypertable, you can also choose to specify a SEGMENT BY column. This allows you to segment compressed rows by a specific column, so that each compressed row corresponds to a data about a single item such as, for example, a specific device ID. This further allows the query planner to determine if the row could possibly match the query without having to decompress the column first. For example:

Device ID	Timestamp	Status Code	Temperature	Min Timestamp	Max Timestamp
A	[12:00:01, 12:00:02, 12:00:03]	[0, 0, 0]	[70.11, 70.12, 70.14]	12:00:01	12:00:03
B	[12:00:01, 12:00:02, 12:00:03]	[0, 0, 4]	[69.70, 69.69, 69.70]	12:00:01	12:00:03

With the data segmented in this way, a query for device A between a time interval becomes quite fast. The query planner can use an index to find those rows for device A that contain at least some timestamps corresponding to the specified interval, and even a sequential scan is quite fast since evaluating device IDs or timestamps does not require decompression. This means the query executor only decompresses the timestamp and temperature columns corresponding to those selected rows.

===== PAGE: https://docs.tigerdata.com/use-timescale/compression/compression-policy/ =====

Create a compression policy

Old API since TimescaleDB v2.18.0 Replaced by Optimize your data for real-time analytics.

You can enable compression on individual hypertables, by declaring which column you want to segment by.

Enable a compression policy

This page uses an example table, called example, and segments it by the device_id column. Every chunk that is more than seven days old is then marked to be automatically compressed. The source data is organized like this:

time	device_id	cpu	disk_io	energy_consumption
8/22/2019 0:00	1	88.2	20	0.8
8/22/2019 0:05	2	300.5	30	0.9

Enabling compression

1. At the psql prompt, alter the table:

   ALTER TABLE example SET (
     timescaledb.compress,
     timescaledb.compress_segmentby = 'device_id'
   );
   

2. Add a compression policy to compress chunks that are older than seven days:

   SELECT add_compression_policy('example', INTERVAL '7 days');
   

For more information, see the API reference for ALTER TABLE (compression) and add_compression_policy.

View current compression policy

To view the compression policy that you've set:

SELECT * FROM timescaledb_information.jobs
  WHERE proc_name='policy_compression';

For more information, see the API reference for timescaledb_information.jobs.

Pause compression policy

To disable a compression policy temporarily, find the corresponding job ID and then call alter_job to pause it:

SELECT * FROM timescaledb_information.jobs where proc_name = 'policy_compression' AND relname = 'example'

SELECT alter_job(<job_id>, scheduled => false);

To enable it again:

SELECT alter_job(<job_id>, scheduled => true);

Remove compression policy

To remove a compression policy, use remove_compression_policy:

SELECT remove_compression_policy('example');

For more information, see the API reference for remove_compression_policy.

Disable compression

You can disable compression entirely on individual hypertables. This command works only if you don't currently have any compressed chunks:

ALTER TABLE <EXAMPLE> SET (timescaledb.compress=false);

If your hypertable contains compressed chunks, you need to decompress each chunk individually before you can turn off compression.

===== PAGE: https://docs.tigerdata.com/use-timescale/compression/modify-compressed-data/ =====

Inserting or modifying data in the columnstore

In TimescaleDB v2.11.0 and later, you can use the UPDATE and DELETE commands to modify existing rows in compressed chunks. This works in a similar way to INSERT operations. To reduce the amount of decompression, TimescaleDB only attempts to decompress data where it is necessary. However, if there are no qualifiers, or if the qualifiers cannot be used as filters, calls to UPDATE and DELETE may convert large amounts of data to the rowstore and back to the columnstore. To avoid large scale conversion, filter on the columns you use to segementby and orderby. This filters as much data as possible before any data is modified, and reduces the amount of data conversions.

DML operations on the columnstore work if the data you are inserting has unique constraints. Constraints are preserved during the insert operation. TimescaleDB uses a Postgres function that decompresses relevant data during the insert to check if the new data breaks unique checks. This means that any time you insert data into the columnstore, a small amount of data is decompressed to allow a speculative insertion, and block any inserts which could violate constraints.

For TimescaleDB v2.17.0 and later, delete performance is improved on compressed hypertables when a large amount of data is affected. When you delete whole segments of data, filter your deletes by segmentby column(s) instead of separate deletes. This considerably increases performance by skipping the decompression step. Since TimescaleDB v2.21.0 and later, DELETE operations on the columnstore are executed on the batch level, which allows more performant deletion of data of non-segmentby columns and reduces IO usage.

Earlier versions of TimescaleDB (before v2.11.0)

This feature requires Postgres 14 or later

From TimescaleDB v2.3.0, you can insert data into compressed chunks with some limitations. The primary limitation is that you can't insert data with unique constraints. Additionally, newly inserted data needs to be compressed at the same time as the data in the chunk, either by a running recompression policy, or by using recompress_chunk manually on the chunk.

In TimescaleDB v2.2.0 and earlier, you cannot insert data into compressed chunks.

===== PAGE: https://docs.tigerdata.com/use-timescale/jobs/create-and-manage-jobs/ =====

Create and manage jobs

Jobs in TimescaleDB are custom functions or procedures that run on a schedule that you define. This page explains how to create, test, alter, and delete a job.

Prerequisites

To follow the procedure on this page you need to:

• Create a target Tiger Cloud service.

  This procedure also works for self-hosted TimescaleDB.

Create a job

To create a job, create a function or procedure that you want your database to execute, then set it up to run on a schedule.

1. Define a function or procedure in the language of your choice

   Wrap it in a CREATE statement:

   CREATE FUNCTION <function_name> (job_id INT DEFAULT NULL, config JSONB DEFAULT NULL)
   RETURNS VOID
   DECLARE
   	<declaration>;
   BEGIN
   	<function_body>;
   END;
   $<variable_name>$ LANGUAGE <language>;
   

   For example, to create a function that reindexes a table within your database:

   CREATE FUNCTION reindex_mytable(job_id INT DEFAULT NULL, config JSONB DEFAULT NULL)
   RETURNS VOID
   AS $$
   BEGIN
      REINDEX TABLE mytable;
   END;
   $$ LANGUAGE plpgsql;
   

   job_id and config are required arguments in the function signature. This returns CREATE FUNCTION to indicate that the function has successfully been created.

2. Call the function to validate

   For example:

   select reindex_mytable();
   

   The result looks like this:

    reindex_mytable
   -----------------
   
   (1 row)
   

3. Register your job with add_job

   Pass the name of your job, the schedule you want it to run on, and the content of your config. For the config value, if you don't need any special configuration parameters, set to NULL. For example, to run the reindex_mytable function every hour:

   SELECT add_job('reindex_mytable', '1h', config => NULL);
   

   The call returns a job_id and stores it along with config in the TimescaleDB catalog.

   The job runs on the schedule you set. You can also run it manually with run_job passing job_id. When the job runs, job_id and config are passed as arguments.

4. Validate the job

   List all currently registered jobs with timescaledb_information.jobs:

   SELECT * FROM timescaledb_information.jobs;
   

   The result looks like this:

   job_id |      application_name      | schedule_interval | max_runtime | max_retries | retry_period |      proc_schema      |    proc_name     |   owner   | scheduled |         config         |          next_start           | hypertable_schema | hypertable_name
   --------+----------------------------+-------------------+-------------+-------------+--------------+-----------------------+------------------+-----------+-----------+------------------------+-------------------------------+-------------------+-----------------
   1 | Telemetry Reporter [1]     | 24:00:00          | 00:01:40    |          -1 | 01:00:00     | _timescaledb_internal | policy_telemetry | postgres  | t         |                        | 2022-08-18 06:26:39.524065+00 |                   |
   1000 | User-Defined Action [1000] | 01:00:00          | 00:00:00    |          -1 | 00:05:00     | public                | reindex_mytable  | tsdbadmin | t         |                        | 2022-08-17 07:17:24.831698+00 |                   |
   (2 rows)
   

Test and debug a job

To debug a job, increase the log level and run the job manually with run_job in the foreground. Because run_job is a stored procedure and not a function, run it with CALL instead of SELECT.

1. Set the minimum log level to DEBUG1

   SET client_min_messages TO DEBUG1;
   

2. Run the job

   Replace 1000 with your job_id:

   CALL run_job(1000);
   

Alter and delete a job

Alter an existing job with alter_job. You can change both the config and the schedule on which the job runs.

1. Change a job's config

   To replace the entire JSON config for a job, call alter_job with a new config object. For example, replace the JSON config for a job with ID 1000:

   SELECT alter_job(1000, config => '{"hypertable":"metrics"}');
   

2. Turn off job scheduling

   To turn off automatic scheduling of a job, call alter_job and set scheduledto false. You can still run the job manually with run_job. For example, turn off the scheduling for a job with ID 1000:

   SELECT alter_job(1000, scheduled => false);
   

3. Re-enable automatic scheduling of a job

   To re-enable automatic scheduling of a job, call alter_job and set scheduled to true. For example, re-enable scheduling for a job with ID 1000:

   SELECT alter_job(1000, scheduled => true);
   

4. Delete a job with delete_job

   For example, to delete a job with ID 1000:

   SELECT delete_job(1000);
   

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/function-pipelines/ =====

Function pipelines

Function pipelines are an experimental feature, designed to radically improve how you write queries to analyze data in Postgres and SQL. They work by applying principles from functional programming and popular tools like Python Pandas, and PromQL.

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

The timevector() function materializes all its data points in memory. This means that if you use it on a very large dataset, it runs out of memory. Do not use the timevector function on a large dataset, or in production.

SQL is the best language for data analysis, but it is not perfect, and at times it can be difficult to construct the query you want. For example, this query gets data from the last day from the measurements table, sorts the data by the time column, calculates the delta between the values, takes the absolute value of the delta, and then takes the sum of the result of the previous steps:

SELECT device id,
sum(abs_delta) as volatility
FROM (
 SELECT device_id,
abs(val - lag(val) OVER last_day) as abs_delta
FROM measurements
WHERE ts >= now()-'1 day'::interval) calc_delta
GROUP BY device_id;

You can express the same query with a function pipeline like this:

SELECT device_id,
    toolkit_experimental.timevector(ts, val)
        -> toolkit_experimental.sort()
        -> toolkit_experimental.delta()
        -> toolkit_experimental.abs()
        -> toolkit_experimental.sum() as volatility
FROM measurements
WHERE ts >= now()-'1 day'::interval
GROUP BY device_id;

Function pipelines are completely SQL compliant, meaning that any tool that speaks SQL is able to support data analysis using function pipelines.

Anatomy of a function pipeline

Function pipelines are built as a series of elements that work together to create your query. The most important part of a pipeline is a custom data type called a timevector. The other elements then work on the timevector to build your query, using a custom operator to define the order in which the elements are run.

Timevectors

A timevector is a collection of time,value pairs with a defined start and end time, that could something like this:

Your entire database might have time,value pairs that go well into the past and continue into the future, but the timevector has a defined start and end time within that dataset, which could look something like this:

To construct a timevector from your data, use a custom aggregate and pass in the columns to become the time,value pairs. It uses a WHERE clause to define the limits of the subset, and a GROUP BY clause to provide identifying information about the time-series. For example, to construct a timevector from a dataset that contains temperatures, the SQL looks like this:

SELECT device_id,
 toolkit_experimental.timevector(ts, val)
FROM measurements
WHERE ts >= now() - '1 day'::interval
GROUP BY device_id;

Custom operator

Function pipelines use a single custom operator of ->. This operator is used to apply and compose multiple functions. The -> operator takes the inputs on the left of the operator, and applies the operation on the right of the operator. To put it more plainly, you can think of it as "do the next thing."

A typical function pipeline could look something like this:

SELECT device_id,
  toolkit_experimental.timevector(ts, val)
        -> toolkit_experimental.sort()
        -> toolkit_experimental.delta()
        -> toolkit_experimental.abs()
        -> toolkit_experimental.sum() as volatility
FROM measurements
WHERE ts >= now() - '1 day'::interval
GROUP BY device_id;

While it might look at first glance as though timevector(ts, val) operation is an argument to sort(), in a pipeline these are all regular function calls. Each of the calls can only operate on the things in their own parentheses, and don't know about anything to the left of them in the statement.

Each of the functions in a pipeline returns a custom type that describes the function and its arguments, these are all pipeline elements. The -> operator performs one of two different types of actions depending on the types on its right and left sides:

• Applies a pipeline element to the left hand argument: performing the function described by the pipeline element on the incoming data type directly.
• Compose pipeline elements into a combined element that can be applied at some point in the future. This is an optimization that allows you to nest elements to reduce the number of passes that are required.

The operator determines the action to perform based on its left and right arguments.

Pipeline elements

There are two main types of pipeline elements:

• Transforms change the contents of the timevector, returning the updated vector.
• Finalizers finish the pipeline and output the resulting data.

Transform elements take in a timevector and produce a timevector. They are the simplest element to compose, because they produce the same type. For example:

SELECT device_id,
 toolkit_experimental.timevector(ts, val)
     -> toolkit_experimental.sort()
        -> toolkit_experimental.delta()
        -> toolkit_experimental.map($$ ($value^3 + $value^2 + $value * 2) $$)
        -> toolkit_experimental.lttb(100)
FROM measurements

Finalizer elements end the timevector portion of a pipeline. They can produce an output in a specified format. or they can produce an aggregate of the timevector.

For example, a finalizer element that produces an output:

SELECT device_id,
 toolkit_experimental.timevector(ts, val)
    -> toolkit_experimental.sort()
    -> toolkit_experimental.delta()
    -> toolkit_experimental.unnest()
FROM measurements

Or a finalizer element that produces an aggregate:

SELECT device_id,
 toolkit_experimental.timevector(ts, val)
    -> toolkit_experimental.sort()
    -> toolkit_experimental.delta()
    -> toolkit_experimental.time_weight()
FROM measurements

The third type of pipeline elements are aggregate accessors and mutators. These work on a timevector in a pipeline, but they also work in regular aggregate queries. An example of using these in a pipeline:

SELECT percentile_agg(val) -> toolkit_experimental.approx_percentile(0.5)
FROM measurements

Transform elements

Transform elements take a timevector, and produce a timevector.

Vectorized math functions

Vectorized math function elements modify each value inside the timevector with the specified mathematical function. They are applied point-by-point and they produce a one-to-one mapping from the input to output timevector. Each point in the input has a corresponding point in the output, with its value transformed by the mathematical function specified.

Elements are always applied left to right, so the order of operations is not taken into account even in the presence of explicit parentheses. This means for a timevector row ('2020-01-01 00:00:00+00', 20.0), this pipeline works:

timevector('2021-01-01 UTC', 10) -> add(5) -> (mul(2) -> add(1))

And this pipeline works in the same way:

timevector('2021-01-01 UTC', 10) -> add(5) -> mul(2) -> add(1)

Both of these examples produce ('2020-01-01 00:00:00+00', 31.0).

If multiple arithmetic operations are needed and precedence is important, consider using a Lambda instead.

Unary mathematical functions

Unary mathematical function elements apply the corresponding mathematical function to each datapoint in the timevector, leaving the timestamp and ordering the same. The available elements are:

Element	Description
abs()	Computes the absolute value of each value
cbrt()	Computes the cube root of each value
ceil()	Computes the first integer greater than or equal to each value
floor()	Computes the first integer less than or equal to each value
ln()	Computes the natural logarithm of each value
log10()	Computes the base 10 logarithm of each value
round()	Computes the closest integer to each value
sign()	Computes +/-1 for each positive/negative value
sqrt()	Computes the square root for each value
trunc()	Computes only the integer portion of each value

Even if an element logically computes an integer, timevectors only deal with double precision floating point values, so the computed value is the floating point representation of the integer. For example:

-- NOTE: the (pipeline -> unnest()).* allows for time, value columns to be produced without a subselect
SELECT (
    toolkit_experimental.timevector(time, value)
    -> toolkit_experimental.abs()
    -> toolkit_experimental.unnest()).*
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

          time          | value
------------------------+-------
 2021-01-06 00:00:00+00 |     0
 2021-01-01 00:00:00+00 |    25
 2021-01-02 00:00:00+00 |   0.1
 2021-01-04 00:00:00+00 |    10
 2021-01-05 00:00:00+00 |   3.3
(5 rows)

Binary mathematical functions

Binary mathematical function elements run the corresponding mathematical function on the value in each point in the timevector, using the supplied number as the second argument of the function. The available elements are:

Element	Description
add(N)	Computes each value plus N
div(N)	Computes each value divided by N
logn(N)	Computes the logarithm base N of each value
mod(N)	Computes the remainder when each number is divided by N
mul(N)	Computes each value multiplied by N
power(N)	Computes each value taken to the N power
sub(N)	Computes each value less N

These elements calculate vector -> power(2) by squaring all of the values, and vector -> logn(3) gives the log-base-3 of each value. For example:

SELECT (
    toolkit_experimental.timevector(time, value)
    -> toolkit_experimental.power(2)
    -> toolkit_experimental.unnest()).*
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

          time          |        value
------------------------+----------------------
 2021-01-06 00:00:00+00 |                    0
 2021-01-01 00:00:00+00 |                  625
 2021-01-02 00:00:00+00 | 0.010000000000000002
 2021-01-04 00:00:00+00 |                  100
 2021-01-05 00:00:00+00 |   10.889999999999999
(5 rows)

Compound transforms

Mathematical transforms are applied only to the value in each point in a timevector and always produce one-to-one output timevectors. Compound transforms can involve both the time and value parts of the points in the timevector, and they are not necessarily one-to-one. One or more points in the input can be used to produce zero or more points in the output. So, where mathematical transforms always produce timevectors of the same length, compound transforms can produce larger or smaller timevectors as an output.

Delta transforms

A delta() transform calculates the difference between consecutive values in the timevector. The first point in the timevector is omitted as there is no previous value and it cannot have a delta(). Data should be sorted using the sort() element before passing into delta(). For example:

SELECT (
    toolkit_experimental.timevector(time, value)
    -> toolkit_experimental.sort()
    -> toolkit_experimental.delta()
    -> toolkit_experimental.unnest()).*
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

          time          | value
------------------------+-------
 2021-01-02 00:00:00+00 | -24.9
 2021-01-04 00:00:00+00 | -10.1
 2021-01-05 00:00:00+00 |  13.3
 2021-01-06 00:00:00+00 |  -3.3
(4 rows)

The first row of the output is missing, as there is no way to compute a delta without a previous value.

Fill method transform

The fill_to() transform ensures that there is a point at least every interval, if there is not a point, it fills in the point using the method provided. The timevector must be sorted before calling fill_to(). The available fill methods are:

fill_method	description
LOCF	Last object carried forward, fill with last known value prior to the hole
Interpolate	Fill the hole using a collinear point with the first known value on either side
Linear	This is an alias for interpolate
Nearest	Fill with the matching value from the closer of the points preceding or following the hole

For example:

SELECT (
    toolkit_experimental.timevector(time, value)
    -> toolkit_experimental.sort()
    -> toolkit_experimental.fill_to('1 day', 'LOCF')
    -> toolkit_experimental.unnest()).*
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

          time          | value
------------------------+-------
 2021-01-01 00:00:00+00 |    25
 2021-01-02 00:00:00+00 |   0.1
 2021-01-03 00:00:00+00 |   0.1
 2021-01-04 00:00:00+00 |   -10
 2021-01-05 00:00:00+00 |   3.3
 2021-01-06 00:00:00+00 |     0
(6 rows)

Largest triangle three buckets (LTTB) transform

The largest triangle three buckets (LTTB) transform uses the LTTB graphical downsampling algorithm to downsample a timevector to the specified resolution while maintaining visual acuity.

Sort transform

The sort() transform sorts the timevector by time, in ascending order. This transform is ignored if the timevector is already sorted. For example:

SELECT (
    toolkit_experimental.timevector(time, value)
    -> toolkit_experimental.sort()
    -> toolkit_experimental.unnest()).*
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

          time          | value
------------------------+-------
 2021-01-01 00:00:00+00 |    25
 2021-01-02 00:00:00+00 |   0.1
 2021-01-04 00:00:00+00 |   -10
 2021-01-05 00:00:00+00 |   3.3
 2021-01-06 00:00:00+00 |     0
(5 rows)

Lambda elements

The Lambda element functions use the Toolkit's experimental Lambda syntax to transform a timevector. A Lambda is an expression that is applied to the elements of a timevector. It is written as a string, usually $$-quoted, containing the expression to run. For example:

$$
 let $is_relevant = $time > '2021-01-01't and $time < '2021-10-14't;
 let $is_significant = abs(round($value)) >= 0;
 $is_relevant and $is_significant
$$

A Lambda expression can be constructed using these components:

• Variable declarations such as let $foo = 3; $foo * $foo. Variable declarations end with a semicolon. All Lambdas must end with an expression, this does not have a semicolon. Multiple variable declarations can follow one another, for example: let $foo = 3; let $bar = $foo * $foo; $bar * 10
• Variable names such as $foo. They must start with a $ symbol. The variables $time and $value are reserved; they refer to the time and value of the point in the vector the Lambda expression is being called on.
• Function calls such as abs($foo). Most mathematical functions are supported.
• Binary operations containing the arithmetic binary operators and, or, =, !=, <, <=, >, >=, ^, *, /, +, and - are supported.
• Interval literals are expressed with a trailing i. For example, '1 day'i. Except for the trailing i, these follow the Postgres INTERVAL input format.
• Time literals such as '2021-01-02 03:00:00't expressed with a trailing t. Except for the trailing t these follow the Postgres TIMESTAMPTZ input format.
• Number literals such as 42, 0.0, -7, or 1e2.

Lambdas follow a grammar that is roughly equivalent to EBNF. For example:

Expr     = ('let' Variable '=' Tuple ';')* Tuple
Tuple    = Binops (',' Binops)*
Binops   = Unaryops (Binop Unaryops)*
UnaryOps = ('-' | 'not') UnaryOps | Term
Term     = Variable | Time | Interval | Number | Function | '(' Expr ')'
Function = FunctionName '(' (Binops ',')* ')'
Variable = ? described above ?
Time     = ? described above ?
Interval = ? described above ?
Number   = ? described above ?

Map Lambda

The map() Lambda maps each element of the timevector. This Lambda must return either a DOUBLE PRECISION, where only the values of each point in the timevector is altered, or a (TIMESTAMPTZ, DOUBLE PRECISION), where both the times and values are changed. An example of the map() Lambda with a DOUBLE PRECISION return:

SELECT (
   toolkit_experimental.timevector(time, value)
   -> toolkit_experimental.map($$ $value + 1 $$)
   -> toolkit_experimental.unnest()).*
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

          time          | value
------------------------+-------
 2021-01-06 00:00:00+00 |     1
 2021-01-01 00:00:00+00 |    26
 2021-01-02 00:00:00+00 |   1.1
 2021-01-04 00:00:00+00 |    -9
 2021-01-05 00:00:00+00 |   4.3
(5 rows)

An example of the map() Lambda with a (TIMESTAMPTZ, DOUBLE PRECISION) return:

SELECT (
   toolkit_experimental.timevector(time, value)
   -> toolkit_experimental.map($$ ($time + '1day'i, $value * 2) $$)
   -> toolkit_experimental.unnest()).*
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

          time          | value
------------------------+-------
 2021-01-07 00:00:00+00 |     0
 2021-01-02 00:00:00+00 |    50
 2021-01-03 00:00:00+00 |   0.2
 2021-01-05 00:00:00+00 |   -20
 2021-01-06 00:00:00+00 |   6.6
(5 rows)

Filter Lambda

The filter() Lambda filters a timevector based on a Lambda expression that returns true for every point that should stay in the timevector timeseries, and false for every point that should be removed. For example:

SELECT (
   toolkit_experimental.timevector(time, value)
   -> toolkit_experimental.filter($$ $time != '2021-01-01't AND $value > 0 $$)
   -> toolkit_experimental.unnest()).*
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

          time          | value
------------------------+-------
 2021-01-02 00:00:00+00 |   0.1
 2021-01-05 00:00:00+00 |   3.3
(2 rows)

Finalizer elements

Finalizer elements complete the function pipeline, and output a value or an aggregate.

Output element

You can finalize a pipeline with a timevector output element. These are used at the end of a pipeline to return a timevector. This can be useful if you need to use them in another pipeline later on. The two types of output are:

• unnest(), which returns a set of (TimestampTZ, DOUBLE PRECISION) pairs.
• materialize(), which forces the pipeline to materialize a timevector. This blocks any optimizations that lazily materialize a timevector.

Aggregate output elements

These elements take a timevector and run the corresponding aggregate over it to produce a result.. The possible elements are:

• average()
• integral()
• counter_agg()
• hyperloglog()
• stats_agg()
• sum()
• num_vals()

An example of an aggregate output using num_vals():

SELECT toolkit_experimental.timevector(time, value) -> toolkit_experimental.num_vals()
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

 ?column?
----------
        5
(1 row)

An example of an aggregate output using stats_agg():

SELECT
    toolkit_experimental.timevector(time, value)
    -> toolkit_experimental.stats_agg()
    -> toolkit_experimental.stddev()
FROM (VALUES (TimestampTZ '2021-01-06 UTC',   0.0 ),
             (            '2021-01-01 UTC',  25.0 ),
             (            '2021-01-02 UTC',   0.10),
             (            '2021-01-04 UTC', -10.0 ),
             (            '2021-01-05 UTC',   3.3 )
     ) as v(time, value);

The output for this example:

      ?column?
--------------------
 12.924666339987272
(1 row)

Aggregate accessors and mutators

Aggregate accessors and mutators work in function pipelines in the same way as they do in other aggregates. You can use them to get a value from the aggregate part of a function pipeline. For example:

SELECT device_id,
timevector(ts, val) -> sort() -> delta() -> stats_agg() -> variance()
FROM measurements

When you use them in a pipeline instead of standard function accessors and mutators, they can make the syntax clearer by getting rid of nested functions. For example, the nested syntax looks like this:

SELECT approx_percentile(0.5, percentile_agg(val))
FROM measurements

Using a function pipeline with the -> operator instead looks like this:

SELECT percentile_agg(val) -> approx_percentile(0.5)
FROM measurements

Counter aggregates

Counter aggregates handle resetting counters. Counters are a common type of metric in application performance monitoring and metrics. All values have resets accounted for. These elements must have a CounterSummary to their left when used in a pipeline, from a counter_agg() aggregate or pipeline element. The available counter aggregate functions are:

Element	Description
counter_zero_time()	The time at which the counter value is predicted to have been zero based on the least squares fit of the points input to the CounterSummary(x intercept)
corr()	The correlation coefficient of the least squares fit line of the adjusted counter value
delta()	Computes the last - first value of the counter
extrapolated_delta(method)	Computes the delta extrapolated using the provided method to bounds of range. Bounds must have been provided in the aggregate or a with_bounds call.
idelta_left()/idelta_right()	Computes the instantaneous difference between the second and first points (left) or last and next-to-last points (right)
intercept()	The y-intercept of the least squares fit line of the adjusted counter value
irate_left()/irate_right()	Computes the instantaneous rate of change between the second and first points (left) or last and next-to-last points (right)
num_changes()	Number of times the counter changed values
num_elements()	Number of items - any with the exact same time have been counted only once
num_changes()	Number of times the counter reset
slope()	The slope of the least squares fit line of the adjusted counter value
with_bounds(range)	Applies bounds using the range (a TSTZRANGE) to the CounterSummary if they weren't provided in the aggregation step

Percentile approximation

Percentile approximation aggregate accessors are used to approximate percentiles. Currently, only accessors are implemented for percentile_agg and uddsketch based aggregates. We have not yet implemented the pipeline aggregate for percentile approximation with tdigest.

Element	Description
approx_percentile(p)	The approximate value at percentile p
approx_percentile_rank(v)	The approximate percentile a value v would fall in
error()	The maximum relative error guaranteed by the approximation
mean()	The exact average of the input values.
num_vals()	The number of input values

Statistical aggregates

Statistical aggregate accessors add support for common statistical aggregates. These allow you to compute and rollup() common statistical aggregates like average and stddev, more advanced aggregates like skewness, and two-dimensional aggregates like slope and covariance. Because there are both single-dimensional and two-dimensional versions of these, the accessors can have multiple forms. For example, average() calculates the average on a single-dimension aggregate, while average_y() and average_x() calculate the average on each of two dimensions. The available statistical aggregates are:

Element	Description
average()/average_y()/average_x()	The average of the values
corr()	The correlation coefficient of the least squares fit line
covariance(method)	The covariance of the values using either population or sample method
determination_coeff()	The determination coefficient (or R squared) of the values
kurtosis(method)/kurtosis_y(method)/kurtosis_x(method)	The kurtosis (fourth moment) of the values using either the population or sample method
intercept()	The intercept of the least squares fit line
num_vals()	The number of values seen
skewness(method)/skewness_y(method)/skewness_x(method)	The skewness (third moment) of the values using either the population or sample method
slope()	The slope of the least squares fit line
stddev(method)/stddev_y(method)/stddev_x(method)	The standard deviation of the values using either the population or sample method
sum()	The sum of the values
variance(method)/variance_y(method)/variance_x(method)	The variance of the values using either the population or sample method
x_intercept()	The x intercept of the least squares fit line

Time-weighted averages aggregates

The average() accessor can be called on the output of a time_weight(). For example:

SELECT time_weight('Linear', ts, val) -> average()  FROM measurements;

Approximate count distinct aggregates

This is an approximation for distinct counts. The distinct_count() accessor can be called on the output of a hyperloglog(). For example:

SELECT hyperloglog(device_id) -> distinct_count() FROM measurements;

Formatting timevectors

You can turn a timevector into a formatted text representation. There are two functions for turning a timevector to text:

• to_text, which allows you to specify the template
• to_plotly, which outputs a format suitable for use with the Plotly JSON chart schema

to_text

toolkit_experimental.to_text(
    timevector(time, value),
    format_string
)

This function produces a text representation, formatted according to the format_string. The format string can use any valid Tera template syntax, and it can include any of the built-in variables:

• TIMES: All the times in the timevector, as an array
• VALUES: All the values in the timevector, as an array
• TIMEVALS: All the time-value pairs in the timevector, formatted as {"time": $TIME, "val": $VAL}, as an array

For example, given this table of data:

CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION);

INSERT INTO data VALUES
    ('2020-1-1', 30.0),
    ('2020-1-2', 45.0),
    ('2020-1-3', NULL),
    ('2020-1-4', 55.5),
    ('2020-1-5', 10.0);

You can use a format string with TIMEVALS to produce the following text:

SELECT toolkit_experimental.to_text(
    timevector(time, value),
    '{{TIMEVALS}}'
) FROM data;

[{\"time\": \"2020-01-01 00:00:00+00\", \"val\": 30}, {\"time\": \"2020-01-02 00:00:00+00\", \"val\": 45}, {\"time\": \"2020-01-03 00:00:00+00\", \"val\": null}, {\"time\": \"2020-01-04 00:00:00+00\", \"val\": 55.5}, {\"time\": \"2020-01-05 00:00:00+00\", \"val\": 10} ]

Or you can use a format string with TIMES and VALUES to produce the following text:

SELECT toolkit_experimental.to_text(
    timevector(time,value),
    '{\"times\": {{ TIMES }}, \"vals\": {{ VALUES }}}'
) FROM data

{\"times\": [\"2020-01-01 00:00:00+00\",\"2020-01-02 00:00:00+00\",\"2020-01-03 00:00:00+00\",\"2020-01-04 00:00:00+00\",\"2020-01-05 00:00:00+00\"], \"vals\": [\"30\",\"45\",\"null\",\"55.5\",\"10\"]}

to_plotly

This function produces a text representation, formatted for use with Plotly.

For example, given this table of data:

CREATE TABLE data(time TIMESTAMPTZ, value DOUBLE PRECISION);

INSERT INTO data VALUES
    ('2020-1-1', 30.0),
    ('2020-1-2', 45.0),
    ('2020-1-3', NULL),
    ('2020-1-4', 55.5),
    ('2020-1-5', 10.0);

You can produce the following Plotly-compatible text:

SELECT toolkit_experimental.to_plotly(
    timevector(time, value)
) FROM data;

{\"times\": [\"2020-01-01 00:00:00+00\",\"2020-01-02 00:00:00+00\",\"2020-01-03 00:00:00+00\",\"2020-01-04 00:00:00+00\",\"2020-01-05 00:00:00+00\"], \"vals\": [\"30\",\"45\",\"null\",\"55.5\",\"10\"]}

All function pipeline elements

This table lists all function pipeline elements in alphabetical order:

Element	Category	Output
abs()	Unary Mathematical	timevector pipeline
add(val DOUBLE PRECISION)	Binary Mathematical	timevector pipeline
average()	Aggregate Finalizer	DOUBLE PRECISION
cbrt()	Unary Mathematical	timevector pipeline
ceil()	Unary Mathematical	timevector pipeline
counter_agg()	Aggregate Finalizer	CounterAgg
delta()	Compound	timevector pipeline
div	Binary Mathematical	timevector pipeline
fill_to	Compound	timevector pipeline
filter	Lambda	timevector pipeline
floor	Unary Mathematical	timevector pipeline
hyperloglog	Aggregate Finalizer	HyperLogLog
ln	Unary Mathematical	timevector pipeline
log10	Unary Mathematical	timevector pipeline
logn	Binary Mathematical	timevector pipeline
lttb	Compound	timevector pipeline
map	Lambda	timevector pipeline
materialize	Output	timevector pipeline
mod	Binary Mathematical	timevector pipeline
mul	Binary Mathematical	timevector pipeline
num_vals	Aggregate Finalizer	BIGINT
power	Binary Mathematical	timevector pipeline
round	Unary Mathematical	timevector pipeline
sign	Unary Mathematical	timevector pipeline
sort	Compound	timevector pipeline
sqrt	Unary Mathematical	timevector pipeline
stats_agg	Aggregate Finalizer	StatsSummary1D
sub	Binary Mathematical	timevector pipeline
sum	Aggregate Finalizer	timevector pipeline
trunc	Unary Mathematical	timevector pipeline
unnest	Output	TABLE (time TIMESTAMPTZ, value DOUBLE PRECISION)

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/time-weighted-averages/ =====

Time-weighted averages and integrals

Time weighted averages and integrals are used in cases where a time series is not evenly sampled. Time series data points are often evenly spaced, for example every 30 seconds, or every hour. But sometimes data points are recorded irregularly, for example if a value has a large change, or changes quickly. Computing an average using data that is not evenly sampled is not always useful.

For example, if you have a lot of ice cream in freezers, you need to make sure the ice cream stays within a 0-10℉ (-20 to -12℃) temperature range. The temperature in the freezer can vary if folks are opening and closing the door, but the ice cream only has a problem if the temperature is out of range for a long time. You can set your sensors in the freezer to sample every five minutes while the temperature is in range, and every 30 seconds while the temperature is out of range. If the results are generally stable, but with some quick moving transients, an average of all the data points weights the transient values too highly. A time weighted average weights each value by the duration over which it occurred based on the points around it, producing much more accurate results.

Time weighted integrals are useful when you need a time-weighted sum of irregularly sampled data. For example, if you bill your users based on irregularly sampled CPU usage, you need to find the total area under the graph of their CPU usage. You can use a time-weighted integral to find the total CPU-hours used by a user over a given time period.

• For more information about how time-weighted averages work, read our time-weighted averages blog.
• For more information about time-weighted average API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/about-hyperfunctions/ =====

About TimescaleDB hyperfunctions

TimescaleDB hyperfunctions are a specialized set of functions that power real-time analytics on time series and events. IoT devices, IT systems, marketing analytics, user behavior, financial metrics, cryptocurrency - these are only a few examples of domains where hyperfunctions can make a huge difference. Hyperfunctions provide you with meaningful, actionable insights in real time.

Tiger Cloud includes all hyperfunctions by default, while self-hosted TimescaleDB includes a subset of them. For additional hyperfunctions, install the TimescaleDB Toolkit Postgres extension.

Available hyperfunctions

Here is a list of all the hyperfunctions provided by TimescaleDB. Hyperfunctions with a tick in the Toolkit column require an installation of TimescaleDB Toolkit for self-hosted deployments. Hyperfunctions with a tick in the Experimental column are still under development.

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

When you upgrade the timescaledb extension, the experimental schema is removed by default. To use experimental features after an upgrade, you need to add the experimental schema again.

For more information about each of the API calls listed in this table, see the hyperfunction API documentation.

Function pipelines

Function pipelines are an experimental feature, designed to radically improve the developer ergonomics of analyzing data in Postgres and SQL, by applying principles from functional programming and popular tools like Python's Pandas, and PromQL.

SQL is the best language for data analysis, but it is not perfect, and at times can get quite unwieldy. For example, this query gets data from the last day from the measurements table, sorts the data by the time column, calculates the delta between the values, takes the absolute value of the delta, and then takes the sum of the result of the previous steps:

SELECT device id,
sum(abs_delta) as volatility
FROM (
 SELECT device_id,
abs(val - lag(val) OVER last_day) as abs_delta
FROM measurements
WHERE ts >= now()-'1 day'::interval) calc_delta
GROUP BY device_id;

You can express the same query with a function pipeline like this:

SELECT device_id,
 timevector(ts, val) -> sort() -> delta() -> abs() -> sum() as volatility
FROM measurements
WHERE ts >= now()-'1 day'::interval
GROUP BY device_id;

Function pipelines are completely SQL compliant, meaning that any tool that speaks SQL is able to support data analysis using function pipelines.

For more information about how function pipelines work, read our blog post.

Toolkit feature development

TimescaleDB Toolkit features are developed in the open. As features are developed they are categorized as experimental, beta, stable, or deprecated. This documentation covers the stable features, but more information on our experimental features in development can be found in the Toolkit repository.

Contribute to TimescaleDB Toolkit

We want and need your feedback! What are the frustrating parts of analyzing time-series data? What takes far more code than you feel it should? What runs slowly, or only runs quickly after many rewrites? We want to solve community-wide problems and incorporate as much feedback as possible.

• Join the discussion.
• Check out the proposed features.
• Explore the current feature requests.
• Add your own feature request.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/approx-count-distincts/ =====

Approximate count distincts

Approximate count distincts are typically used to find the number of unique values, or cardinality, in a large dataset. When you calculate cardinality in a dataset, the time it takes to process the query is proportional to how large the dataset is. So if you wanted to find the cardinality of a dataset that contained only 20 entries, the calculation would be very fast. Finding the cardinality of a dataset that contains 20 million entries, however, can take a significant amount of time and compute resources. Approximate count distincts do not calculate the exact cardinality of a dataset, but rather estimate the number of unique values, to reduce memory consumption and improve compute time by avoiding spilling the intermediate results to the secondary storage.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/gapfilling-interpolation/ =====

Gapfilling and interpolation

Most time-series data analysis techniques aggregate data into fixed time intervals, which smooths the data and makes it easier to interpret and analyze. When you write queries for data in this form, you need an efficient way to aggregate raw observations, which are often noisy and irregular, in to fixed time intervals. TimescaleDB does this using time bucketing, which gives a clear picture of the important data trends using a concise, declarative SQL query.

Sorting data into time buckets works well in most cases, but problems can arise if there are gaps in the data. This can happen if you have irregular sampling intervals, or you have experienced an outage of some sort. You can use a gapfilling function to create additional rows of data in any gaps, ensuring that the returned rows are in chronological order, and contiguous.

• For more information about how gapfilling works, read our gapfilling blog.
• For more information about gapfilling and interpolation API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/approximate-percentile/ =====

Approximate percentiles

TimescaleDB uses approximation algorithms to calculate a percentile without requiring all of the data. This also makes them more compatible with continuous aggregates.

By default, TimescaleDB Toolkit uses uddsketch, but you can also choose to use tdigest. For more information about these algorithms, see the advanced aggregation methods documentation.

Run an approximate percentage query

In this procedure, we use an example table called response_times that contains information about how long a server takes to respond to API calls.

Running an approximate percentage query

1. At the psql prompt, create a continuous aggregate that computes the daily aggregates:

   CREATE MATERIALIZED VIEW response_times_daily
   WITH (timescaledb.continuous)
   AS SELECT
     time_bucket('1 day'::interval, ts) as bucket,
     percentile_agg(response_time_ms)
   FROM response_times
   GROUP BY 1;
   

2. Re-aggregate the aggregate to get the last 30 days, and look for the ninety-fifth percentile:

   SELECT approx_percentile(0.95, percentile_agg) as threshold
   FROM response_times_daily
   WHERE bucket >= time_bucket('1 day'::interval, now() - '30 days'::interval);
   

3. You can also create an alert:

   WITH t as (SELECT approx_percentile(0.95, percentile_agg(percentile_agg)) as threshold
   FROM response_times_daily
   WHERE bucket >= time_bucket('1 day'::interval, now() - '30 days'::interval))
   
   SELECT count(*)
   FROM response_times
   WHERE ts > now()- '1 minute'::interval
   AND response_time_ms > (SELECT threshold FROM t);
   

For more information about percentile approximation API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/index/ =====

Hyperfunctions

Real-time analytics demands more than basic SQL functions, efficient computation becomes essential as datasets grow in size and complexity. That’s where TimescaleDB hyperfunctions come in: high-performance, SQL-native functions purpose-built for time-series analysis. They are designed to process, aggregate, and analyze large volumes of data with maximum efficiency while maintaining consistently high performance. With hyperfunctions, you can run sophisticated analytical queries and extract meaningful insights in real time.

Hyperfunctions introduce partial aggregation, letting TimescaleDB store intermediate states instead of raw data or final results. These partials can be merged later for rollups (consolidation), eliminating costly reprocessing and slashing compute overhead, especially when paired with continuous aggregates.

Take tracking p95 latency across thousands of app instances as an example:

• With standard SQL, every rollup requires rescanning and resorting massive datasets.
• With TimescaleDB, the percentile_agg hyperfunction stores a compact state per minute, which you simply merge to get hourly or daily percentiles—no full reprocess needed.

The result? Scalable, real-time percentile analytics that deliver fast, accurate insights across high-ingest, high-resolution data, while keeping resource use lean.

Tiger Cloud includes all hyperfunctions by default, while self-hosted TimescaleDB includes a subset of them. To include all hyperfunctions with TimescaleDB, install the TimescaleDB Toolkit Postgres extension on your self-hosted Postgres deployment.

For more information, read the hyperfunctions blog post.

Learn hyperfunction basics and install TimescaleDB Toolkit

• Learn about hyperfunctions to understand how they work before using them.
• Install the TimescaleDB Toolkit extension to access more hyperfunctions on self-hosted TimescaleDB.

Browse hyperfunctions and TimescaleDB Toolkit features by category

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/hyperloglog/ =====

Hyperloglog

Hyperloglog is typically used to find the cardinality of very large datasets. If you want to find the number of unique values, or cardinality, in a dataset, the time it takes to process this query is proportional to how large the dataset is. So if you wanted to find the cardinality of a dataset that contained only 20 entries, the calculation would be very fast. Finding the cardinality of a dataset that contains 20 million entries, however, can take a significant amount of time and compute resources.

Hyperloglog does not calculate the exact cardinality of a dataset, but rather estimates the number of unique values. It does this by converting the original data into a hash of random numbers that represents the cardinality of the dataset. This is not a perfect calculation of the cardinality, but it is usually within a margin of error of 2%.

The benefit of hyperloglog on time-series data is that it can continue to calculate the approximate cardinality of a dataset as it changes over time. It does this by adding an entry to the hyperloglog hash as new data is retrieved, rather than recalculating the result for the entire dataset every time it is needed. This makes it an ideal candidate for using with continuous aggregates.

For more information about approximate count distinct API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/time-bucket-gapfill/ =====

Time bucket gapfill

Sometimes data sorted into time buckets can have gaps. This can happen if you have irregular sampling intervals, or you have experienced an outage of some sort. If you have a time bucket that has no data at all, the average returned from the time bucket is NULL, which could cause problems. You can use a gapfilling function to create additional rows of data in any gaps, ensuring that the returned rows are in chronological order, and contiguous. The time bucket gapfill function creates a contiguous set of time buckets but does not fill the rows with data. You can create data for the new rows using another function, such as last observation carried forward (LOCF), or interpolation.

For more information about gapfilling and interpolation API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/percentile-approx/ =====

Percentile approximation

In general, percentiles are useful for understanding the distribution of data. The fiftieth percentile is the point at which half of your data is greater and half is lesser. The tenth percentile is the point at which 90% of the data is greater, and 10% is lesser. The ninety-ninth percentile is the point at which 1% is greater, and 99% is lesser.

The fiftieth percentile, or median, is often a more useful measure than the average, especially when your data contains outliers. Outliers can dramatically change the average, but do not affect the median as much. For example, if you have three rooms in your house and two of them are 40℉ (4℃) and one is 130℉ (54℃), the average room temperature is 70℉ (21℃), which doesn't tell you much. However, the fiftieth percentile temperature is 40℉ (4℃), which tells you that at least half your rooms are at refrigerator temperatures (also, you should probably get your heating checked!)

Percentiles are sometimes avoided because calculating them requires more CPU and memory than an average or other aggregate measures. This is because an exact computation of the percentile needs the full dataset as an ordered list. TimescaleDB uses approximation algorithms to calculate a percentile without requiring all of the data. This also makes them more compatible with continuous aggregates. By default, TimescaleDB uses uddsketch, but you can also choose to use tdigest. For more information about these algorithms, see the advanced aggregation methods documentation.

Technically, a percentile divides a group into 100 equally sized pieces, while a quantile divides a group into an arbitrary number of pieces. Because we don't always use exactly 100 buckets, "quantile" is the more technically correct term in this case. However, we use the word "percentile" because it's a more common word for this type of function.

• For more information about how percentile approximation works, read our percentile approximation blog.
• For more information about percentile approximation API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/advanced-agg/ =====

Percentile approximation advanced aggregation methods

TimescaleDB uses approximation algorithms to calculate a percentile without requiring all of the data. This also makes them more compatible with continuous aggregates. By default, TimescaleDB uses uddsketch, but you can also choose to use tdigest. This section describes the different methods, and helps you to decide which one you should use.

uddsketch is the default algorithm. It uses exponentially sized buckets to guarantee the approximation falls within a known error range, relative to the true discrete percentile. This algorithm offers the ability to tune the size and maximum error target of the sketch.

tdigest buckets data more aggressively toward the center of the quantile range, giving it greater accuracy at the tails of the range, around 0.001 or 0.995.

Choose the right algorithm

Each algorithm has different features, which can make one better than another depending on your use case. Here are some of the differences to consider when choosing an algorithm:

Before you begin, it is important to understand that the formal definition for a percentile is imprecise, and there are different methods for determining what the true percentile actually is. In Postgres, given a target percentile p, percentile_disc returns the smallest element of a set, so that p percent of the set is less than that element. However, percentile_cont returns an interpolated value between the two nearest matches for p. In practice, the difference between these methods is very small but, if it matters to your use case, keep in mind that tdigest approximates the continuous percentile, while uddsketch provides an estimate of the discrete value.

Think about the types of percentiles you're most interested in. tdigest is optimized for more accurate estimates at the extremes, and less accurate estimates near the median. If your workflow involves estimating ninety-ninth percentiles, then choose tdigest. If you're more concerned about getting highly accurate median estimates, choose uddsketch.

The algorithms differ in the way they estimate data. uddsketch has a stable bucketing function, so it always returns the same percentile estimate for the same underlying data, regardless of how it is ordered or re-aggregated. On the other hand, tdigest builds up incremental buckets based on the average of nearby points, which can result in some subtle differences in estimates based on the same data unless the order and batching of the aggregation is strictly controlled, which is sometimes difficult to do in Postgres. If stable estimates are important to you, choose uddsketch.

Calculating precise error bars for tdigest can be difficult, especially when merging multiple sub-digests into a larger one. This can occur through summary aggregation, or parallelization of the normal point aggregate. If you need to tightly characterize your errors, choose uddsketch. However, because uddsketch uses exponential bucketing to provide a guaranteed relative error, it can cause some wildly varying absolute errors if the dataset covers a large range. For example, if the data is evenly distributed over the range [1,100], estimates at the high end of the percentile range have about 100 times the absolute error of those at the low end of the range. This gets much more extreme if the data range is [0,100]. If having a stable absolute error is important to your use case, choose tdigest.

While both algorithms are likely to get smaller and faster with future optimizations, uddsketch generally requires a smaller memory footprint than tdigest, and a correspondingly smaller disk footprint for any continuous aggregates. Regardless of the algorithm you choose, the best way to improve the accuracy of your percentile estimates is to increase the number of buckets, which is simpler to do with uddsketch. If your use case does not get a clear benefit from using tdigest, the default uddsketch is your best choice.

For some more technical details and usage examples of the different algorithms, see the developer documentation for uddsketch and tdigest.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/locf/ =====

Last observation carried forward

Last observation carried forward (LOCF) is a form of linear interpolation used to fill gaps in your data. It takes the last known value and uses it as a replacement for the missing data.

For more information about gapfilling and interpolation API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/stats-aggs/ =====

Statistical aggregation

To make common statistical aggregates easier to work with in window functions and continuous aggregates, TimescaleDB provides common statistical aggregates in a slightly different form than otherwise available in Postgres.

This example calculates the average, standard deviation, and kurtosis of a value in the measurements table:

SELECT
    time_bucket('10 min'::interval, ts),
    average(stats_agg(val)),
    stddev(stats_agg(val), 'pop'),
    kurtosis(stats_agg(val), 'pop')
FROM measurements
GROUP BY 1;

This uses a two-step aggregation process. The first step is an aggregation step (stats_agg(val)), which creates a machine-readable form of the aggregate. The second step is an accessor. The available accessors are average, stddev, and kurtosis. The accessors run final calculations and output the calculated value in a human-readable way. This makes it easier to construct your queries, because it distinguishes the parameters, and makes it clear which aggregates are being re-aggregated or rolled up. Additionally, because this query syntax is used in all TimescaleDB Toolkit queries, when you are used to it, you can use it to construct more and more complicated queries.

A more complex example uses window functions to calculate tumbling window statistical aggregates. The statistical aggregate is first calculated over each minute in the subquery and then the rolling aggregate is used to re-aggregate it over each 15 minute period preceding. The accessors remain the same as the previous example:

SELECT
    bucket,
    average(rolling(stats_agg) OVER fifteen_min),
    stddev(rolling(stats_agg) OVER fifteen_min, 'pop'),
    kurtosis(rolling(stats_agg) OVER fifteen_min, 'pop')
FROM (SELECT
        time_bucket('1 min'::interval, ts) AS bucket,
        stats_agg(val)
     FROM measurements
     GROUP BY 1) AS stats
WINDOW fifteen_min as (ORDER BY bucket ASC RANGE '15 minutes' PRECEDING);

For some more technical details and usage examples of the two-step aggregation method, see the blog post on aggregates or the developer documentation.

The stats_agg aggregate is available in two forms, a one-dimensional aggregate shown earlier in this section, and a two-dimensional aggregate. The two-dimensional aggregate takes in two variables (Y, X), which are dependent and independent variables respectively. The two-dimensional aggregate performs all the same calculations on each individual variable as performing separate one-dimensional aggregates would, and additionally performs linear regression on the two variables. Accessors for one-dimensional values append a _y or _x to the name. For example:

SELECT
    average_y(stats_agg(val2, val1)), -- equivalent to average(stats_agg(val2))
    stddev_x(stats_agg(val2, val1)), -- equivalent to stddev(stats_agg(val1))
    slope(stats_agg(val2, val1)) -- the slope of the least squares fit line of the values in val2 & val1
FROM measurements_multival;

For more information about statistical aggregation API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/counter-aggregation/ =====

Counter aggregation

When you are monitoring application performance, there are two main types of metrics that you can collect: gauges, and counters. Gauges fluctuate up and down, like temperature or speed, while counters always increase, like the total number of miles travelled in a vehicle.

When you process counter data, it is usually assumed that if the value of the counter goes down, the counter has been reset. For example, if you wanted to count the total number of miles travelled in a vehicle, you would expect the values to continuously increase: 1, 2, 3, 4, and so on. If the counter reset to 0, you would expect that this was a new trip, or an entirely new vehicle. This can become a problem if you want to continue counting from where you left off, rather than resetting to 0. A reset could occur if you have had a short server outage, or any number of other reasons. To get around this, you can analyze counter data by looking at the change over time, which accounts for resets.

Accounting for resets can be difficult to do in SQL, so TimescaleDB has developed aggregate and accessor functions that handle calculations for counters in a more practical way.

Counter aggregates can be used in continuous aggregates, even though they are not parallelizable in Postgres. For more information, see the section on parallelism and ordering.

For more information about counter aggregation API calls, see the hyperfunction API documentation.

Run a counter aggregate query using a delta function

In this procedure, we are using an example table called example that contains counter data.

Running a counter aggregate query using a delta function

1. Create a table called example:

   CREATE TABLE example (
       measure_id      BIGINT,
       ts              TIMESTAMPTZ ,
       val             DOUBLE PRECISION,
       PRIMARY KEY (measure_id, ts)
   );
   

2. Create a counter aggregate and the delta accessor function. This gives you the change in the counter's value over the time period, accounting for any resets. This allows you to search for fifteen minute periods where the counter increased by a larger or smaller amount:

   SELECT measure_id,
       delta(
           counter_agg(ts, val)
       )
   FROM example
   GROUP BY measure_id;
   

3. You can also use the time_bucket function to produce a series of deltas over fifteen minute increments:

   SELECT measure_id,
       time_bucket('15 min'::interval, ts) as bucket,
       delta(
           counter_agg(ts, val)
       )
   FROM example
   GROUP BY measure_id, time_bucket('15 min'::interval, ts);
   

Run a counter aggregate query using an extrapolated delta function

If your series is less regular, the deltas are affected by the number of samples in each fifteen minute period. You can improve this by using the extrapolated_delta function. To do this, you need to provide bounds that define where to extrapolate to. In this example, we use the time_bucket_range function, which works in the same way as time_bucket but produces an open ended range of all the times in the bucket. This example also uses a CTE to do the counter aggregation, which makes it a little easier to understand what's going on in each part.

Running a counter aggregate query using an extrapolated delta function

1. Create a hypertable called example:

   CREATE TABLE example (
       measure_id      BIGINT,
       ts              TIMESTAMPTZ ,
       val             DOUBLE PRECISION,
       PRIMARY KEY (measure_id, ts)
   ) WITH (
     tsdb.hypertable,
     tsdb.partition_column='ts',
     tsdb.chunk_interval='15 days'
   );
   

If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

1. Create a counter aggregate and the extrapolated delta function:

   with t as (
       SELECT measure_id,
           time_bucket('15 min'::interval, ts) as bucket,
           counter_agg(ts, val, toolkit_experimental.time_bucket_range('15 min'::interval, ts))
       FROM example
       GROUP BY measure_id, time_bucket('15 min'::interval, ts))
   SELECT time_bucket,
       extrapolated_delta(counter_agg, method => 'prometheus')
   FROM t ;
   

In this procedure, Prometheus is used to do the extrapolation. TimescaleDB's current extrapolation function is built to mimic the Prometheus project's increase function, which measures the change of a counter extrapolated to the edges of the queried region.

Run a counter aggregate query with a continuous aggregate

Your counter aggregate might be more useful if you make a continuous aggregate out of it.

1. Create the continuous aggregate:

   CREATE MATERIALIZED VIEW example_15
   WITH (timescaledb.continuous)
   AS SELECT measure_id,
       time_bucket('15 min'::interval, ts) as bucket,
       counter_agg(ts, val, time_bucket_range('15 min'::interval, ts))
   FROM example
   GROUP BY measure_id, time_bucket('15 min'::interval, ts);
   

2. You can also re-aggregate from the continuous aggregate into a larger bucket size:

   SELECT
       measure_id,
       time_bucket('1 day'::interval, bucket),
       delta(
           rollup(counter_agg)
       )
   FROM example_15
   GROUP BY measure_id, time_bucket('1 day'::interval, bucket);
   

Parallelism and ordering

The counter reset calculations require a strict ordering of inputs, which means they are not parallelizable in Postgres. This is because Postgres handles parallelism by issuing rows randomly to workers. However, if your parallelism can guarantee sets of rows that are disjointed in time, the algorithm can be parallelized, as long as it is within a time range, and all rows go to the same worker. This is the case for both continuous aggregates and for distributed hypertables, as long as the partitioning keys are in the group by, even though the aggregate itself doesn't really make sense otherwise.

For more information about parallelism and ordering, see our developer documentation

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/heartbeat-agg/ =====

Heartbeat aggregation

Given a series of timestamped health checks, it can be tricky to determine the overall health of a system over a given interval. Postgres provides window functions that you use to get a sense of where unhealthy gaps are, but they can be somewhat awkward to use efficiently.

This is one of the many cases where hyperfunctions provide an efficient, simple solution for a frequently occurring problem. Heartbeat aggregation helps analyze event-based time-series data with intermittent or irregular signals.

This example uses the SustData public dataset. This dataset tracks the power usage of a small number of apartments and houses over four different deployment intervals. The data is collected in one-minute samples from each unit.

When you have loaded the data into hypertables, you can create a materialized view containing weekly heartbeat aggregates for each of the units:

CREATE MATERIALIZED VIEW weekly_heartbeat AS
  SELECT
    time_bucket('1 week', tmstp) as week,
    iid as unit,
    deploy,
    heartbeat_agg(tmstp, time_bucket('1w', tmstp), '1w', '2m')
  FROM power_samples
  GROUP BY 1,2,3;

The heartbeat aggregate takes four parameters: the timestamp column, the start of the interval, the length of the interval, and how long the aggregate is considered live after each timestamp. This example uses 2 minutes as the heartbeat lifetime to give some tolerance for small gaps.

You can use this data to see when you're receiving data for a particular unit. This example rolls up the weekly aggregates into a single aggregate, and then views the live ranges:

SELECT live_ranges(rollup(heartbeat_agg)) FROM weekly_heartbeat WHERE unit = 17;

                     live_ranges
-----------------------------------------------------
 ("2010-09-18 00:00:00+00","2011-03-27 01:01:50+00")
 ("2011-03-27 03:00:52+00","2011-07-03 00:01:00+00")
 ("2011-07-05 00:00:00+00","2011-08-21 00:01:00+00")
 ("2011-08-22 00:00:00+00","2011-08-25 00:01:00+00")
 ("2011-08-27 00:00:00+00","2011-09-06 00:01:00+00")
 ("2011-09-08 00:00:00+00","2011-09-29 00:01:00+00")
 ("2011-09-30 00:00:00+00","2011-10-04 00:01:00+00")
 ("2011-10-05 00:00:00+00","2011-10-17 00:01:00+00")
 ("2011-10-19 00:00:00+00","2011-11-09 00:01:00+00")
 ("2011-11-10 00:00:00+00","2011-11-14 00:01:00+00")
 ("2011-11-15 00:00:00+00","2011-11-18 00:01:00+00")
 ("2011-11-20 00:00:00+00","2011-11-23 00:01:00+00")
 ("2011-11-24 00:00:00+00","2011-12-01 00:01:00+00")
 ("2011-12-02 00:00:00+00","2011-12-12 00:01:00+00")
 ("2011-12-13 00:00:00+00","2012-01-12 00:01:00+00")
 ("2012-01-13 00:00:00+00","2012-02-03 00:01:00+00")
 ("2012-02-04 00:00:00+00","2012-02-10 00:01:00+00")
 ("2012-02-11 00:00:00+00","2012-03-25 01:01:50+00")
 ("2012-03-25 03:00:51+00","2012-04-11 00:01:00+00")

You can construct more elaborate queries. For example, to return the 5 units with the lowest uptime during the third deployment:

SELECT unit, uptime(rollup(heartbeat_agg))
FROM weekly_heartbeat
WHERE deploy = 3
GROUP BY unit
ORDER BY uptime LIMIT 5;

 unit |      uptime
------+-------------------
   31 | 203 days 22:05:00
   34 | 222 days 22:05:00
   32 | 222 days 22:05:00
   35 | 222 days 22:05:00
   30 | 222 days 22:05:00

Combine aggregates from different units to get the combined coverage. This example queries the interval where any part of a deployment was active:

SELECT deploy, live_ranges(rollup(heartbeat_agg))
FROM weekly_heartbeat group by deploy order by deploy;

 deploy |                     live_ranges
--------+-----------------------------------------------------
      1 | ("2010-07-29 00:00:00+00","2010-11-26 00:01:00+00")
      2 | ("2010-11-25 00:00:00+00","2011-03-27 01:01:59+00")
      2 | ("2011-03-27 03:00:00+00","2012-03-25 01:01:59+00")
      2 | ("2012-03-25 03:00:26+00","2012-04-17 00:01:00+00")
      2 | ("2012-04-20 00:00:00+00","2012-04-21 00:01:00+00")
      2 | ("2012-05-11 00:00:00+00","2012-05-13 00:01:00+00")
      2 | ("2013-02-20 00:00:00+00","2013-02-21 00:01:00+00")
      3 | ("2012-08-01 00:00:01+00","2013-03-31 01:01:16+00")
      3 | ("2013-03-31 03:00:03+00","2013-05-22 00:01:00+00")
      4 | ("2013-07-31 00:00:00+00","2014-03-30 01:01:49+00")
      4 | ("2014-03-30 03:00:01+00","2014-04-25 00:01:00+00")

Then use this data to make observations and draw conclusions:

• The second deployment had a lot more problems than the other ones.
• There were some readings from February 2013 that were incorrectly categorized as a second deployment.
• The timestamps are given in a local time without time zone, resulting in some missing hours around springtime daylight savings time changes.

For more information about heartbeat aggregation API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/troubleshoot-hyperfunctions/ =====

Troubleshooting hyperfunctions and TimescaleDB Toolkit

This section contains some ideas for troubleshooting common problems experienced with hyperfunctions and Toolkit.

Updating the Toolkit extension fails with an error saying no update path

In some cases, when you create the extension, or use the ALTER EXTENSION timescaledb_toolkit UPDATE command to update the Toolkit extension, it might fail with an error like this:

ERROR:  extension "timescaledb_toolkit" has no update path from version "1.2" to version "1.3"

This occurs if the list of available extensions does not include the version you are trying to upgrade to, and it can occur if the package was not installed correctly in the first place. To correct the problem, install the upgrade package, restart Postgres, verify the version, and then attempt the update again.

Troubleshooting Toolkit setup

1. If you're installing Toolkit from a package, check your package manager's local repository list. Make sure the TimescaleDB repository is available and contains Toolkit. For instructions on adding the TimescaleDB repository, see the installation guides:

   • Debian/Ubuntu installation guide
   • RHEL/CentOS installation guide

2. Update your local repository list with apt update or yum update.

3. Restart your Postgres service.

4. Check that the right version of Toolkit is among your available extensions:

   SELECT * FROM pg_available_extensions
     WHERE name = 'timescaledb_toolkit';
   

   The result should look like this:

   -[ RECORD 1 ]-----+--------------------------------------------------------------------------------------
   name              | timescaledb_toolkit
   default_version   | 1.6.0
   installed_version | 1.6.0
   comment           | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities
   

5. Retry CREATE EXTENSION or ALTER EXTENSION.

===== PAGE: https://docs.tigerdata.com/use-timescale/hyperfunctions/time-weighted-average/ =====

Time-weighted average

Time weighted average in TimescaleDB is implemented as an aggregate that weights each value using last observation carried forward (LOCF), or linear interpolation. The aggregate is not parallelizable, but it is supported with continuous aggregation.

Run a time-weighted average query

In this procedure, we are using an example table called freezer_temps that contains data about internal freezer temperatures.

Running a time-weighted average query

1. At the psqlprompt, find the average and the time-weighted average of the data:

   SELECT freezer_id,
     avg(temperature),
    average(time_weight('Linear', ts, temperature)) as time_weighted_average
   FROM freezer_temps
   GROUP BY freezer_id;
   

2. To determine if the freezer has been out of temperature range for more than 15 minutes at a time, use a time-weighted average in a window function:

   SELECT *,
   average(
           time_weight('Linear', ts, temperature) OVER (PARTITION BY freezer_id ORDER BY ts RANGE  '15 minutes'::interval PRECEDING )
          ) as rolling_twa
   FROM freezer_temps
   ORDER BY freezer_id, ts;
   

For more information about time-weighted average API calls, see the hyperfunction API documentation.

===== PAGE: https://docs.tigerdata.com/use-timescale/services/service-management/ =====

Service management

In the Service management section of the Operations dashboard, you can fork your service, reset the password, pause, or delete the service.

Fork a service

When you a fork a service, you create its exact copy including the underlying database. This allows you to create a copy that you can use for testing purposes, or to prepare for a major version upgrade. The only difference between the original and the forked service is that the tsdbadmin user has a different password.

The fork is created by restoring from backup and applying the write-ahead log. The data is fetched from Amazon S3, so forking doesn't tax the running instance.

You can fork services that have a status of Running or Paused. You cannot fork services while they have a status of In progress. Wait for the service to complete the transition before you start forking.

Forks only have data up to the point when the original service was forked. Any data written to the original service after the time of forking does not appear in the fork. If you want the fork to assume operations from the original service, pause your main service before forking to avoid any data discrepancy between services.

1. In Tiger Cloud Console, from the Services list, ensure the service you want to form has a status of Running or Paused, then click the name of the service you want to fork.
2. Navigate to the Operations tab.
3. In the Service management section, click Fork service. In the dialog, confirm by clicking Fork service. The forked service takes a few minutes to start.
4. To change the configuration of your fork, click Advanced options. You can set different compute and storage options, separate from your original service.
5. Confirm by clicking Fork service. The forked service takes a few minutes to start.
6. The forked service shows in the Services dashboard with a label stating which service it has been forked from.

Create a service fork using the CLI

To manage development forks:

1. Install Tiger CLI

   Use the terminal to install the CLI:

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.deb.sh | sudo os=any dist=any bash
   sudo apt-get install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   curl -s https://packagecloud.io/install/repositories/timescale/tiger-cli/script.rpm.sh | sudo os=rpm_any dist=rpm_any bash
   sudo yum install tiger-cli
   

   brew install --cask timescale/tap/tiger-cli
   

   curl -fsSL https://cli.tigerdata.com | sh
   

2. Set up API credentials

   1. Log Tiger CLI into your Tiger Data account:

      tiger auth login
      

      Tiger CLI opens Console in your browser. Log in, then click Authorize.

      You can have a maximum of 10 active client credentials. If you get an error, open credentials and delete an unused credential.

   2. Select a Tiger Cloud project:

      Auth URL is: https://console.cloud.timescale.com/oauth/authorize?client_id=lotsOfURLstuff
      Opening browser for authentication...
      Select a project:
      
      > 1. Tiger Project (tgrproject)
      2. YourCompany (Company wide project) (cpnproject)
      3. YourCompany Department (dptproject)
      
      Use ↑/↓ arrows or number keys to navigate, enter to select, q to quit
      

      If only one project is associated with your account, this step is not shown.

      Where possible, Tiger CLI stores your authentication information in the system keychain/credential manager. If that fails, the credentials are stored in ~/.config/tiger/credentials with restricted file permissions (600). By default, Tiger CLI stores your configuration in ~/.config/tiger/config.yaml.

3. Test your authenticated connection to Tiger Cloud by listing services

   tiger service list
   

   This call returns something like:

   • No services:

     🏜️  No services found! Your project is looking a bit empty.
     🚀 Ready to get started? Create your first service with: tiger service create
     

   • One or more services:

     ┌────────────┬─────────────────────┬────────┬─────────────┬──────────────┬──────────────────┐
     │ SERVICE ID │        NAME         │ STATUS │    TYPE     │    REGION    │     CREATED      │
     ├────────────┼─────────────────────┼────────┼─────────────┼──────────────┼──────────────────┤
     │ tgrservice │ tiger-agent-service │ READY  │ TIMESCALEDB │ eu-central-1 │ 2025-09-25 16:09 │
     └────────────┴─────────────────────┴────────┴─────────────┴──────────────┴──────────────────┘
     

4. Fork the service

    tiger service fork tgrservice --now --no-wait --name bob
   

   By default a fork matches the resource of the parent Tiger Cloud services. For paid plans specify --cpu and/or --memory for dedicated resources.

   You see something like:

   🍴 Forking service 'tgrservice' to create 'bob' at current state...
   ✅ Fork request accepted!
   📋 New Service ID: <service_id>
   🔐 Password saved to system keyring for automatic authentication
   🎯 Set service '<service_id>' as default service.
   ⏳ Service is being forked. Use 'tiger service list' to check status.
   ┌───────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
   │     PROPERTY      │                                              VALUE                                               │
   ├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
   │ Service ID        │ <service_id>                                                                                       │
   │ Name              │ bob                                                                                              │
   │ Status            │                                                                                                  │
   │ Type              │ TIMESCALEDB                                                                                      │
   │ Region            │ eu-central-1                                                                                     │
   │ CPU               │ 0.5 cores (500m)                                                                                 │
   │ Memory            │ 2 GB                                                                                             │
   │ Direct Endpoint   │ <service-id>.<project-id>.tsdb.cloud.timescale.com:<port>                                             │
   │ Created           │ 2025-10-08 13:58:07 UTC                                                                          │
   │ Connection String │ postgresql://tsdbadmin@<service-id>.<project-id>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require │
   └───────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
   

5. When you are done, delete your forked service

   1. Use the CLI to request service delete:

      tiger service delete <service_id>
      

   2. Validate the service delete:

      Are you sure you want to delete service '<service_id>'? This operation cannot be undone.
      Type the service ID '<service_id>' to confirm:
      <service_id>
      

      You see something like:

      🗑️  Delete request accepted for service '<service_id>'.
      ✅ Service '<service_id>' has been successfully deleted.
      

Reset your service password

You can reset your service password from the Operations dashboard. This is the password you use to connect to your service, not the password for Tiger Cloud Console. To reset your Console password, navigate to the Account page.

When you reset your service password, you are prompted for your Console password. When you have authenticated, you can create a new service password, ask Console to auto-generate a password, or switch your authentication type between SCRAM and MD5.

SCRAM (salted challenge response authentication mechanism) and MD5 (message digest algorithm 5) are cryptographic authentication mechanisms. Tiger Cloud Console uses SCRAM by default. It is more secure and strongly recommended. The MD5 option is provided for compatibility with older clients.

Pause a service

You can pause a service if you want to stop it running temporarily. When you pause a service, you are no longer billed for compute resources. However, you do need to continue paying for any storage you are using. Pausing a service ensures that it is still available, and is ready to be restarted at any time.

Delete a service

You can delete a service to remove it completely. This removes the service and its underlying data from the server. You cannot recover a deleted service.

===== PAGE: https://docs.tigerdata.com/use-timescale/services/connection-pooling/ =====

Connection pooling

You can scale your Tiger Cloud service connections and improve its performance by using connection poolers. Tiger Cloud uses pgBouncer for connection pooling.

If your service needs a large number of short-lived connections, a connection pooler is a great way to improve performance. For example, web, serverless, and IoT applications often use an event-based architecture where data is read or written from the database for a very short amount of time.

Your application rapidly opens and closes connections while the pooler maintains a set of long-running connections to the service. This improves performance because the pooler opens the connections in advance, allowing the application to open many short-lived connections, while the service opens few, long-lived connections.

User authentication

By default, the poolers have authentication to the service, so you can use any custom users you already have set up without further configuration. You can continue using the tsdbadmin user if that is your preferred method. However, you might need to add custom configurations for some cases such as statement_timeout for a pooler user.

Creating a new user with custom settings

1. Connect to your service as the tsdbadmin user, and create a new role named <MY_APP> with the password as <PASSWORD>:

   CREATE ROLE <MY_APP> LOGIN PASSWORD '<PASSWORD>';
   

2. Change the statement_timeout settings to 2 seconds for this user:

   ALTER ROLE my_app SET statement_timeout TO '2s';
   

3. In a new terminal window, connect on the pooler with the new user <MY_APP>:

   ❯ PGPASSWORD=<NEW_PASSWORD> psql 'postgres://my_app@service.project.tsdb.cloud.timescale.com:30477/tsdb?sslmode=require'
   

   The output looks something like this:

   <CodeBlock canCopy={false} showLineNumbers={true} children={psql (15.3 (Homebrew), server 15.4 (Ubuntu 15.4-1.pgdg22.04+1)) SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off) Type "help" for help.} />

4. Check that the settings are correct by logging in as the <MY_APP> user:

   SELECT current_user;
   
   ┌──────────────┐
   │ current_user │
   ├──────────────┤
   │ my_app       │
   └──────────────┘
   (1 row)
   

   Check the statement_timeout setting is correct for the <MY_APP> user:

   tsdb=> show statement_timeout;
   ┌───────────────────┐
   │ statement_timeout │
   ├───────────────────┤
   │ 2s                │
   └───────────────────┘
   (1 row)
   

Pool types

When you create a connection pooler, there are two pool types to choose from: session or transaction. Each pool type uses a different mode to handle connections.

Session pools allocate a connection from the pool until they are closed by the application, similar to a regular Postgres connection. When the application closes the connection, it is sent back to the pool.

Transaction pool connections are allocated only for the duration of the transaction, releasing the connection back to the pool when the transaction ends. If your application opens and closes connections frequently, choose the transaction pool type.

By default, the pooler supports both modes simultaneously. However, the connection string you use to connect your application is different, depending on whether you want a session or transaction pool type. When you create a connection pool in the Tiger Cloud Console, you are given the correct connection string for the mode you choose.

For example, a connection string to connect directly to your service looks a bit like this:

<CodeBlock canCopy={false} showLineNumbers={false} children={postgres://<USERNAME>:<PASSWORD>@service.example.cloud.timescale.com:30133/tsdb?sslmode=require} />

A session pool connection string is the same, but uses a different port number, like this:

<CodeBlock canCopy={false} showLineNumbers={false} children={postgres://<USERNAME>:<PASSWORD>@service.example.cloud.timescale.com:29303/tsdb?sslmode=require} />

The transaction pool connection string uses the same port number as a session pool connection, but uses a different database name, like this:

<CodeBlock canCopy={false} showLineNumbers={false} children={postgres://<USERNAME>:<PASSWORD>@service.example.cloud.timescale.com:29303/tsdb_transaction?sslmode=require} />

Make sure you check the Tiger Cloud Console output for the correct connection string to use in your application.

Connection pool sizes

A connection pooler manages connections to both the service itself, and the client application. It keeps a fixed number of connections open with the service, while allowing clients to open and close connections. Clients can request a connection from the session pool or the transaction pool. The connection pooler will then allocate the connection if there is one free.

The number of client connections allowed to each pool is proportional to the max_connections parameter set for the service. The session pool can have a maximum of max_connections - 17 client connections, while the transaction pool can have a maximum of (max_connections - 17) * 20 client connections.

Of the 17 reserved connections that are not allocated to either pool, 12 are reserved for the database superuser by default, and another 5 for Tiger Cloud operations.

For example, if max_connections is set to 500, the maximum number of client connections for your session pool is 483 (500 - 17) and 9,660 (483 * 20) for your transaction pool. The default value of max_connections varies depending on your service's compute size.

Add a connection pooler

When you create a new service, you can also create a connection pooler. Alternatively, you can add a connection pooler to an existing service in Console.

Adding a connection pooler

1. Log in to Console and click the service you want to add a connection pooler to.

2. In Operations, click Connection pooling > Add pooler.

   Your pooler connection details are displayed in the Connection pooling tab. Use this information to connect to your transaction or session pooler. For more information about the different pool types, see the pool types section.

Remove a connection pooler

If you no longer need a connection pooler, you can remove it in Console. When you have removed your connection pooler, make sure that you also update your application to adjust the port it uses to connect to your service.

1. In Console, select the service you want to remove a connection pooler from.

2. Select Operations, then Connection pooling.

3. Click Remove connection pooler.

   Confirm that you want to remove the connection pooler.

After you have removed a pooler, if you add it back in the future, it uses the same connection string and port that was used before.

pgBouncer statistics commands

1. Connect to your service.
2. Switch to the pgbouncer database: \c pgbouncer
3. Run any read-only command for the pgBouncer cli (e.g., SHOW STATS;).
4. For full options, see the pgBouncer docs here.

VPC and connection pooling

VPCs are supported with connection pooling. It does not matter the order you add the pooler or connect to a VPC. Your connection strings will automatically be updated to use the VPC connection string.

===== PAGE: https://docs.tigerdata.com/use-timescale/services/service-explorer/ =====

Service explorer

Service explorer in Tiger Cloud Console provides a rich administrative dashboard for understanding the state of your database instance. The explorer gives you insight into the performance of your database, giving you greater confidence and control over your data.

The explorer works like an operations center as you develop and run your applications with Tiger Cloud. It gives you quick access to the key properties of your database, like table sizes, schema definitions, and foreign key references, as well as information specific to Tiger Cloud, like information on your hypertables and continuous aggregates.

To see the explorer, select your service in Console and click Explorer.

General information

In the General information section, you can see a high-level summary of your service, including all your hypertables and relational tables. It summarizes your overall compression ratios, and other policy and continuous aggregate data. And, if you aren't already using key features like continuous aggregates, columnstore compression, or other automation policies and actions, it provides pointers to tutorials and documentation to help you get started.

Tables

You can have a detailed look into all your tables, including information about table schemas, table indexes, and foreign keys. For your hypertables, it shows details about chunks, continuous aggregates, and policies such as data retention policies and data reordering. You can also inspect individual hypertables, including their sizes, dimension ranges, and columnstore compression status.

From this section, you can also set an automated policy to compress chunks into the columnstore. For more information, see the hypercore documentation.

For more information about hypertables, see the hypertables section.

Continuous aggregates

In the Continuous aggregate section, you can see all your continuous aggregates, including top-level information such as their size, whether they are configured for real-time aggregation, and their refresh periods.

For more information about continuous aggregates, see the continuous aggregates section.

===== PAGE: https://docs.tigerdata.com/use-timescale/services/service-overview/ =====

About services

You manage your Tiger Cloud services and interact with your data in Tiger Cloud Console using the following modes:

Ops mode	Data mode
You use the ops mode to: Ensure data security with high availability and read replicas Save money with columnstore compression and tiered storage Enable Postgres extensions to add extra functionality Increase security using VPCs Perform day-to-day administration	Powered by PopSQL, you use the data mode to: Write queries with autocomplete Visualize data with charts and dashboards Schedule queries and dashboards for alerts or recurring reports Share queries and dashboards Interact with your data on auto-pilot with SQL assistant This feature is not available under the Free pricing plan.

When you log into Tiger Cloud Console, you see the project overview. Click a service to view run-time data and connection information. Click Operations to configure your service.

Each service hosts a single database managed for you by Tiger Cloud. If you need more than one database, create a new service.

Service users

By default, when you create a new service, a new tsdbadmin user is created. This is the user that you use to connect to your new service.

The tsdbadmin user is the owner of the database, but is not a superuser. You cannot access the postgres user. There is no superuser access to Tiger Cloud databases.

In your service, the tsdbadmin user can create another user with any other role. For a complete list of roles available, see the Postgres role attributes documentation.

You cannot create multiple databases in a single service. If you need data isolation, use schemas or create additional services.

===== PAGE: https://docs.tigerdata.com/use-timescale/services/change-resources/ =====

Manually change compute resources

Tiger Cloud charges are based on the amount of storage you use. You don't pay for fixed storage size, and you don't need to worry about scaling disk size as your data grows—we handle it all for you. To reduce your data costs further, combine hypercore, a data retention policy, and tiered storage.

You use Tiger Cloud Console to resize the compute (CPU/RAM) resources available to your Tiger Cloud services at any time, with a short downtime.

Update compute resources for a service

You can change the CPU and memory allocation for your service at any time with minimal downtime, usually less than a minute. The new resources become available as soon as the service restarts. You can change the CPU and memory allocation up or down, as frequently as required.

Note that:

• For the 48 CPU / 192 GiB option, 6 CPU / 14 GiB is reserved for platform operations.
• For the 64 CPU / 256 GiB option, 6 CPU / 16 GiB is reserved for platform operations.

There is momentary downtime while the new compute settings are applied. In most cases, this is less than a minute. However, before making changes to your service, best practice is to enable HA replication on the service. When you resize a service with HA enabled, Tiger Cloud:

1. Resizes the replica.
2. Waits for the replica to catch up.
3. Performs a switchover to the resized replica.
4. Restarts the primary.

HA reduce downtime in the case of resizes or maintenance window restarts, from a minute or so to a couple of seconds.

When you change resource settings, the current and new charges are displayed immediately so that you can verify how the changes impact your costs.

Because compute changes require an interruption to your services, plan accordingly so that the settings are applied during an appropriate service window.

1. In Console, choose the service to modify.
2. Click Operations > Compute and storage.
3. Select the new CPU / Memory allocation. You see the allocation and costs in the comparison chart
4. Click Apply. Your service goes down briefly while the changes are applied.

Out of memory errors

If you run intensive queries on your services, you might encounter out of memory (OOM) errors. This occurs if your query consumes more memory than is available.

When this happens, an OOM killer process shuts down Postgres processes using SIGKILL commands until the memory usage falls below the upper limit. Because this kills the entire server process, it usually requires a restart.

To prevent service disruption caused by OOM errors, Tiger Cloud attempts to shut down only the query that caused the problem. This means that the problematic query does not run, but that your service continues to operate normally.

• If the normal OOM killer is triggered, the error log looks like this:

  2021-09-09 18:15:08 UTC [560567]:TimescaleDB: LOG: server process (PID 2351983) was terminated by signal 9: Killed
  

  Wait for the service to come back online before reconnecting.

• Tiger Cloud shuts the client connection only

  If Tiger Cloud successfully guards the service against the OOM killer, it shuts down only the client connection that was using too much memory. This prevents the entire service from shutting down, so you can reconnect immediately. The error log looks like this:

  2022-02-03 17:12:04 UTC [2253150]:TimescaleDB: tsdbadmin@tsdb,app=psql [53200] ERROR: out of memory
  

===== PAGE: https://docs.tigerdata.com/use-timescale/time-buckets/use-time-buckets/ =====

Aggregate time-series data with time bucket

The time_bucket function helps you group in a hypertable so you can perform aggregate calculations over arbitrary time intervals. It is usually used in combination with GROUP BY for this purpose.

This section shows examples of time_bucket use. To learn how time buckets work, see the about time buckets section.

Group data by time buckets and calculate a summary value

Group data into time buckets and calculate a summary value for a column. For example, calculate the average daily temperature in a table named weather_conditions. The table has a time column named time and a temperature column:

SELECT time_bucket('1 day', time) AS bucket,
  avg(temperature) AS avg_temp
FROM weather_conditions
GROUP BY bucket
ORDER BY bucket ASC;

The time_bucket function returns the start time of the bucket. In this example, the first bucket starts at midnight on November 15, 2016, and aggregates all the data from that day:

bucket                 |      avg_temp
-----------------------+---------------------
2016-11-15 00:00:00+00 | 68.3704391666665821
2016-11-16 00:00:00+00 | 67.0816684374999347

Group data by time buckets and show the end time of the bucket

By default, the time_bucket column shows the start time of the bucket. If you prefer to show the end time, you can shift the displayed time using a mathematical operation on time.

For example, you can calculate the minimum and maximum CPU usage for 5-minute intervals, and show the end of time of the interval. The example table is named metrics. It has a time column named time and a CPU usage column named cpu:

SELECT time_bucket('5 min', time) + '5 min' AS bucket,
  min(cpu),
  max(cpu)
FROM metrics
GROUP BY bucket
ORDER BY bucket DESC;

The addition of + '5 min' changes the displayed timestamp to the end of the bucket. It doesn't change the range of times spanned by the bucket.

Group data by time buckets and change the time range of the bucket

To change the time range spanned by the buckets, use the offset parameter, which takes an INTERVAL argument. A positive offset shifts the start and end time of the buckets later. A negative offset shifts the start and end time of the buckets earlier.

For example, you can calculate the average CPU usage for 5-hour intervals, and shift the start and end times of all buckets 1 hour later:

SELECT time_bucket('5 hours', time, '1 hour'::INTERVAL) AS bucket,
  avg(cpu)
FROM metrics
GROUP BY bucket
ORDER BY bucket DESC;

Calculate the time bucket of a single value

Time buckets are usually used together with GROUP BY to aggregate data. But you can also run time_bucket on a single time value. This is useful for testing and learning, because you can see what bucket a value falls into.

For example, to see the 1-week time bucket into which January 5, 2021 would fall, run:

SELECT time_bucket(INTERVAL '1 week', TIMESTAMP '2021-01-05');

The function returns 2021-01-04 00:00:00. The start time of the time bucket is the Monday of that week, at midnight.

===== PAGE: https://docs.tigerdata.com/use-timescale/time-buckets/about-time-buckets/ =====

About time buckets

Time bucketing is essential for real-time analytics. The time_bucket function enables you to aggregate data in a hypertable into buckets of time. For example, 5 minutes, 1 hour, or 3 days. It's similar to Postgres's date_bin function, but it gives you more flexibility in the bucket size and start time.

You can use it to roll up data for analysis or downsampling. For example, you can calculate 5-minute averages for a sensor reading over the last day. You can perform these rollups as needed, or pre-calculate them in continuous aggregates.

This section explains how time bucketing works. For examples of the time_bucket function, see the section on Aggregate time-series data with time_bucket.

How time bucketing works

Time bucketing groups data into time intervals. With time_bucket, the interval length can be any number of microseconds, milliseconds, seconds, minutes, hours, days, weeks, months, years, or centuries.

The time_bucket function is usually used in combination with GROUP BY to aggregate data. For example, you can calculate the average, maximum, minimum, or sum of values within a bucket.

Origin

The origin determines when time buckets start and end. By default, a time bucket doesn't start at the earliest timestamp in your data. There is often a more logical time. For example, you might collect your first data point at 00:37, but you probably want your daily buckets to start at midnight. Similarly, you might collect your first data point on a Wednesday, but you might want your weekly buckets calculated from Sunday or Monday.

Instead, time is divided into buckets based on intervals from the origin. The following diagram shows how, using the example of 2-week buckets. The first possible start date for a bucket is origin. The next possible start date for a bucket is origin + bucket interval. If your first timestamp does not fall exactly on a possible start date, the immediately preceding start date is used for the beginning of the bucket.

For example, say that your data's earliest timestamp is April 24, 2020. If you bucket by an interval of two weeks, the first bucket doesn't start on April 24, which is a Friday. It also doesn't start on April 20, which is the immediately preceding Monday. It starts on April 13, because you can get to April 13, 2020, by counting in two-week increments from January 3, 2000, which is the default origin in this case.

Default origins

For intervals that don't include months or years, the default origin is January 3, 2000. For month, year, or century intervals, the default origin is January 1, 2000. For integer time values, the default origin is 0.

These choices make the time ranges of time buckets more intuitive. Because January 3, 2000, is a Monday, weekly time buckets start on Monday. This is compliant with the ISO standard for calculating calendar weeks. Monthly and yearly time buckets use January 1, 2000, as an origin. This allows them to start on the first day of the calendar month or year.

If you prefer another origin, you can set it yourself using the origin parameter. For example, to start weeks on Sunday, set the origin to Sunday, January 2, 2000.

Timezones

The origin time depends on the data type of your time values.

If you use TIMESTAMP, by default, bucket start times are aligned with 00:00:00. Daily and weekly buckets start at 00:00:00. Shorter buckets start at a time that you can get to by counting in bucket increments from 00:00:00 on the origin date.

If you use TIMESTAMPTZ, by default, bucket start times are aligned with 00:00:00 UTC. To align time buckets to another timezone, set the timezone parameter.

===== PAGE: https://docs.tigerdata.com/mst/vpc-peering/vpc-peering-gcp/ =====

Set up Virtual Private Cloud (VPC) peering on GCP

You can configure VPC peering for your Managed Service for TimescaleDB project, using VPC provided by GCP.

Before you begin

• Set up a VPC peering for your project in MST.
• In your GCP console, click the project name and make a note of the Project ID.
• In your GCP console, go to VPC Networks, find the VPC that you want to connect, and make a note of the network name for that VPC.

Configuring a VPC peering on GCP

To set up VPC peering for your project:

1. In MST Console, click VPC and select the VPC connection that you created.

2. Type the project ID of your GCP project in GCP Project ID.

3. Type the network name of the VPC in GCP in GCP VPC network name.

4. Click Add peering connection.

   A new connection with a status of Pending Peer is listed in your GCP console. Make a note of the project name and the network name.

5. In the GCP console, go to VPC > VPC network peering and select Create Connection.

6. Type a name for the peering connection and type the project ID and network name that you made a note of.

7. Click Create.

After the peering is successful, it is active in both MST_CONSOLE_SHORT and your GCP console.

===== PAGE: https://docs.tigerdata.com/mst/vpc-peering/vpc-peering/ =====

Configure VPC peering

You can Configure VPC peering for your Managed Service for TimescaleDB project, using the VPC section of the dashboard for your project. VPC peering setup is a per project and per region setting. This means that all services created and running utilize the same VPC peering connection. If needed, you can have multiple projects that peer with different connections.

Configuring a VPC peering

You can configure VPC peering as a project and region-specific setting. This means that all services created and running use the same VPC peering connection. If necessary, you can use different connections for VPC peering across multiple projects. Only Admin and operator user roles can create a VPC.

To set up VPC peering for your project:

1. In MST Console, click VPC.

2. Click Create VPC.

3. Choose a cloud provider in the Cloud list.

4. In the IP range field, type the IP range that you want to use for the VPC connection. Use an IP range that does not overlap with any networks that you want to connect through VPC peering. For example, if your own networks use the range 10.0.0.0/8, you could set the range for your Managed Service for TimescaleDB project VPC to 192.168.0.0/24.

5. Click Create VPC.

The state of the VPC is listed in the table.

===== PAGE: https://docs.tigerdata.com/mst/vpc-peering/vpc-peering-aws-transit/ =====

Set up Transit Gateway on AWS

AWS Transit Gateway (TGW) enables transitive routing from on-premises networks through VPN and from other VPC. By creating a Transit Gateway VPC attachment, services in an MST Project VPC can route traffic to all other networks attached - directly or indirectly - to the Transit Gateway.

Before you begin

• Set up a VPC peering for your project in MST.
• In your AWS console, go to My Account and make a note of your account ID.
• In your AWS console, go to Transit Gateways, find the transit gateway that you want to attach, and make a note of the ID.

Attaching a VPC to an AWS Transit Gateway

To set up VPC peering for your project:

1. In MST Console, click VPC and select the VPC connection that you created.

2. In the VPC Peering connections page select Transit Gateway VPC Attachment.

3. Type the account ID of your AWS account in AWS Account ID.

4. Type the ID of the Transit Gateway of AWS in Transit Gateway ID.

5. Type the IP range in the Network cidrs field.

   Each Transit Gateway has a route table of its own, and by default routes traffic to each attached network directly to attached VPCs or indirectly through VPN attachments. The attached VPCs' route tables need to be updated to include the TGW as a target for any IP range (CIDR) that should be routed using the VPC attachment. These IP ranges must be configured when creating the attachment for an MST Project VPC.

6. Click Add peering connection.

   A new connection with a status of Pending Acceptance is listed in your AWS console. Verify that the account ID and transit gateway ID match those listed in MST Console.

7. In the AWS console, go to Actions and select Accept Request. Update your AWS route tables to match your Managed Service for TimescaleDB CIDR settings.

After you accept the request in AWS Console, the peering connection is active in the MST Console.

===== PAGE: https://docs.tigerdata.com/mst/vpc-peering/vpc-peering-aws/ =====

Set up Virtual Private Cloud (VPC) peering on AWS

You can configure VPC peering for your Managed Service for TimescaleDB project, using the VPC on AWS.

Before you begin

• Set up a VPC peering for your project in MST.
• In your AWS console, go to My Account and make a note of your account ID.
• In your AWS console, go to Peering connections, find the VPC that you want to connect, and make a note of the ID for that VPC.

Configuring a VPC peering

To set up VPC peering for your project:

1. In MST Console, click VPC and select the VPC connection that you created.

2. Type the account ID of your AWS account in AWS Account ID.

3. Type the ID of the VPC in AWS in AWS VPC ID.

4. Click Add peering connection.

   A new connection with a status of Pending Acceptance is listed in your AWS console. Verify that the account ID and VPC ID match those listed in MST Console.

5. In the AWS console, go to Actions and select Accept Request. Update your AWS route tables to match your Aiven CIDR settings.

After you accept the request in AWS Console, the peering connection is active in the MST portal.

===== PAGE: https://docs.tigerdata.com/mst/vpc-peering/vpc-peering-azure/ =====

Set up Virtual Private Cloud (VPC) peering on Azure

You can Configure VPC peering for your Managed Service for TimescaleDB project, using the VPC on Azure.

Before you begin

• Installed Aiven Client.
• Signed in to MST Console.
• Set up a VPC peering for your project in MST.

Configuring a VPC peering on Azure

1. Log in with an Azure administration account, using the Azure CLI:

   az account clear
   az login
   

   This should open a window in your browser prompting you to choose an Azure account to log in with. You need an account with at least the Application administrator role to create VPC peering. If you manage multiple Azure subscriptions, configure the Azure CLI to default to the correct subscription using the command:

   az account set --subscription <subscription name or id>
   

2. Create an application object in your AD tenant, using the Azure CLI:

   az ad app create --display-name "<NAME>" --sign-in-audience AzureADMultipleOrgs --key-type Password
   

   This creates an entity to your AD that can be used to log into multiple AD tenants (--sign-in-audience AzureADMultipleOrgs), but only the home tenant (the tenant the app was created in) has the credentials to authenticate the app. Save the appId field from the output - this is referred to as $user_app_id.

3. Create a service principal for your app object. Ensure that the service principal is created to the Azure subscription containing the VNet you wish to peer:

   az ad sp create --id $user_app_id
   

   This creates a service principal to your subscription that may have permissions to peer your VNet. Save the objectId field from the output - this is referred to as $user_sp_id.

4. Set a password for your app object:

      az ad app credential reset --id $user_app_id
   

   Save the password field from the output - this is referred to as $user_app_secret.

5. Find the ID properties of your virtual network:

   az network vnet list
   

   Make a note of these:

   • The id field, which is referred to as $user_vnet_id

   • The Azure Subscription ID, which is the part after /subscriptions/ in the resource ID. This is referred to as $user_subscription_id.

   • The resource group name or the resourceGroup field in the output. This is referred to as $user_resource_group.

   • The Vnet name or the name field from the output as $user_vnet_name The $user_vnet_id should have the format:

     /subscriptions/$user_subscription_id/resourceGroups/$user_resource_group/providers/Microsoft.Network/virtualNetworks/$user_vnet_name.

6. Grant your service principal permissions to peer. The service principal that you created needs to be assigned a role that has permission for the Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write action on the scope of your VNet. To limit the permissions granted to the app object and service principal, you can create a custom role with just that permission. The built-in Network Contributor role includes that permission, and can be found using az role definition list --name "Network Contributor" The id field from the output is used as $network_contributor_role_id to assign the service principal that role:

   az role assignment create --role $network_contributor_role_id --assignee-object-id $user_sp_id --scope $user_vnet_id
   

   This allows the application object to manage the network in the --scope. Because you control the application object, it may also be given permission for the scope of an entire resource group, or the whole subscription to allow create other peerings later without assigning the role again for each VNet separately.

7. Create a service principal for the Managed Service for TimescaleDB application object

   The Managed Service for TimescaleDB AD tenant contains an application object similar to the one you created, and Managed Service for TimescaleDB uses it to create a peering from the Project VPC VNet in Managed Service for TimescaleDB to the VNet in Azure. For this, the Managed Service for TimescaleDB app object needs a service principal in your subscription:

   az ad sp create --id <ID_OF_THE_TIMESCALE_APPLICATION_OBJECT>
   

   Save the objectId field from the output - it is referred to as $aiven_sp_id.

   If this fails with the error "When using this permission, the backing application of the service principal being created must in the local tenant" then your account does not have the correct permissions. Use an account with at least the Application administrator role assigned.

8. Create a custom role for the Managed Service for TimescaleDB application object

   The Managed Service for TimescaleDB application now has a service principal that can be given permissions. In order to target a network in your subscription with a peering and nothing else, you can create a custom role definition, with only a single action allowing to do that and only that:

   az role definition create --role-definition '{"Name": "<name of your choosing>",
   "Description": "Allows creating a peering to vnets in scope (but not from)",
   "Actions": ["Microsoft.Network/virtualNetworks/peer/action"],
   "AssignableScopes": ["/subscriptions/'$user_subscription_id'"]}'
   

   Creating a custom role must include your subscription's id in AssignableScopes . This in itself does not give permissions to your subscription - it merely restricts which scopes a role assignment can include. Save the id field from the output - this is referred to as $aiven_role_id.

9. Assign the custom role to the service principal to peer with your VNet. Assign the role that you created in the previous step to the Managed Service for TimescaleDB service principal with the scope of your VNet:

   az role assignment create --role $aiven_role_id --assignee-object-id $aiven_sp_id --scope $user_vnet_id
   

10. Get your Azure Active Directory (AD) tenant id:

az account list

Make note of the tenantId field from the output. It is referred to as $user_tenant_id.

1. Create a peering connection from the Managed Service for TimescaleDB Project VPC using Aiven CLI:

   avn vpc peering-connection create --project-vpc-id $aiven_project_vpc_id --peer-cloud-account $user_subscription_id --peer-resource-group $user_resource_group --peer-vpc $user_vnet_name --peer-azure-app-id $user_app_id --peer-azure-tenant-id $user_tenant_id
   

   $aiven_project_vpc_id is the ID of the Managed Service for TimescaleDB project VPC, and can be found using the avn vpc list command.

Managed Service for TimescaleDB creates a peering from the VNet in the Managed Service for TimescaleDB Project VPC to the VNet in your subscription. In addition, it creates a service principal for the application object in your tenant --peer-azure-app-id $user_app_id, giving it permission to target the Managed Service for TimescaleDB subscription VNet with a peering. Your AD tenant ID is also needed in order for the Managed Service for TimescaleDB application object to authenticate with your tenant to give it access to the service principal that you created --peer-azure-tenant-id $user_tenant_id.

Ensure that the arguments starting with `$user_` are in lower case. Azure
resource names are case-agnostic, but the Aiven API currently only accepts
names in lower case. If no error is shown, the peering connection is being set
up by Managed Service for TimescaleDB.

1. Run the following command until the state is no longer APPROVED , but PENDING_PEER:

   avn vpc peering-connection get -v --project-vpc-id $aiven_project_vpc_id --peer-cloud-account $user_subscription_id --peer-resource-group $user_resource_group --peer-vpc $user_vnet_name
   

   A state such as INVALID_SPECIFICATION or REJECTED_BY_PEER may be shown if the VNet specified did not exist, or the Managed Service for TimescaleDB app object wasn't given permissions to peer with it. If that occurs, check your configuration and then recreate the peering connection. If everything went as expected, the state changes to PENDING_PEER within a couple of minutes showing details to set up the peering connection from your VNet to the Project VPC's VNet in Managed Service for TimescaleDB.

   Save the to-tenant-id field in the output. It is referred to as the aiven_tenant_id. The to-network-id field from the output is referred to as the $aiven_vnet_id.

2. Log out the Azure user you logged in using:

   az account clear
   

3. Log in the application object you created to your AD tenant using:

   az login --service-principal -u $user_app_id -p $user_app_secret --tenant $user_tenant_id
   

4. Log in the same application object to the Managed Service for TimescaleDB AD tenant:

   az login --service-principal -u $user_app_id -p $user_app_secret --tenant
   $aiven_tenant_id
   

   Now your application object has a session with both AD tenants

5. Create a peering from your VNet to the VNet in the Managed Service for TimescaleDB subscription:

   az network vnet peering create --name <peering name of your choosing> --remote-vnet $aiven_vnet_id --vnet-name $user_vnet_name --resource-group $user_resource_group --subscription $user_subscription_id --allow-vnet-access
   

   If you do not specify --allow-vnet-access no traffic is allowed to flow from the peered VNet and services cannot be reached through the peering. After the peering has been created, the peering should be in the state connected.

   In case you get the following error, it's possible the role assignment hasn't taken effect yet. If that is the case, try logging in again and creating the peering again after waiting a bit by repeating the commands in this step. If the error message persists, check the role assignment was correct.

   The client `<random uuid>` with object id `<another random uuid>` does not have
   authorization to perform action
   `Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write` over scope
   '$user_vnet_id' If access was recently granted, refresh your credentials.
   

6. In the Aiven CLI, check if the peering connection is ACTIVE:

   avn vpc peering-connection get -v --project-vpc-id $aiven_project_vpc_id --peer-cl
   

Managed Service for TimescaleDB polls peering connections in state PENDING_PEER regularly to see if your subscription has created a peering connection to the Managed Service for TimescaleDB Project VPC's VNet. After this is detected, the state changes from PENDING_PEER to ACTIVE. After this services in the Project VPC can be reached through the peering.

===== PAGE: https://docs.tigerdata.com/mst/integrations/grafana-mst/ =====

Integrate Managed Service for TimescaleDB as a data source in Grafana

You can integrate Managed Service for TimescaleDB with Grafana to visualize your data. Grafana service in MST has built-in Prometheus, Postgres, Jaeger, and other data source plugins that allow you to query and visualize data from a compatible database.

Prerequisites

Before you begin, make sure you have:

• Created a service
• Created a Grafana service

Configure Managed Service for TimescaleDB as a data source

You can configure a service as a data source to a Grafana service to query and visualize the data from the database.

Configuring Managed Service for TimescaleDB as a data source

1. In MST Console, click the service that you want to add as a data source for the Grafana service.
2. In the Overview tab for the service go to the Service Integrations section.
3. Click the Set up integration button.
4. In the Available service integrations for TimescaleDB dialog, click the Use Integration button for Datasource.
5. In the dialog that appears, choose the Grafana service in the drop-down menu, and click the Enable button.
6. In the Services view, click the Grafana service to which you added the MST service as a data source.
7. In the Overview tab for the Grafana service, make a note of the User and Password fields.
8. In the Overview tab for the Grafana service, click the link in the Service URI field to open Grafana.
9. Log in to Grafana with your service credentials.
10. Navigate to Configuration → Data sources. The data sources page lists Managed Service for TimescaleDB as a configured data source for the Grafana instance.

When you have configured Managed Service for TimescaleDB as a data source in Grafana, you can create panels that are populated with data using SQL.

===== PAGE: https://docs.tigerdata.com/mst/integrations/google-data-studio-mst/ =====

Integrate Managed Service for TimescaleDB and Google Data Studio

You can create reports or perform some analysis on data you have in Managed Service for TimescaleDB using Google Data Studio. You can use Data Studio to integrate other data sources, such as YouTube Analytics, MySQL, BigQuery, AdWords, and others.

Before you begin

• You should also have a Google account.
• In the overview page of your service:
  • Download the CA certificate named ca.pem for your service.
  • Make a note of the Host, Port, Database name, User, and Password fields for the service.

Connecting to a Managed Service for TimescaleDB data source from Data Studio

1. Log in to Google and open Google Data Studio.
2. Click the Create + button and choose Data source.
3. Select PostgreSQL as the Google Connector.
4. In the Database Authentication tab, type details for the Host Name, Port, Database, Username, and Password fields.
5. Select Enable SSL and upload your server certificate file, ca.pem.
6. Click AUTHENTICATE.
7. Choose the table to be queried, or select CUSTOM QUERY to create an SQL query.
8. Click CONNECT.

===== PAGE: https://docs.tigerdata.com/mst/integrations/logging/ =====

Logging

There are a number of different ways to review logs and metrics for your services. You can use the native logging tool in MST Console, retrieve details logs using the Aiven CLI tool, or integrate a third-party service, such as SolarWinds Loggly.

Native logging

To see the most recent logged events for your service.

1. In MST Console, in the Services tab, find the service you want to review, and check it is marked as Running.

2. Navigate to the Logs tab to see a constantly updated list of logged events.

   

Dump logs to a text file with the Aiven CLI

If you want to dump your Managed Service for TimescaleDB logs to a text file or an archive for use later on, you can use the Aiven CLI.

Sign in to your Managed Service for TimescaleDB account from the Aiven CLI tool, and use this command to dump your logs to a text file called tslogs.txt:

avn service logs -S desc -f --project <project name> <service_name> > tslogs.txt

For more information about the Aiven CLI tool, see the Aiven CLI section.

Logging integrations

If you need to access logs for your services regularly, or if you need more detailed logging than Managed Service for TimescaleDB can provide in MST Console, you can connect your Managed Service for TimescaleDB to a logging service such as SolarWinds Loggly.

This section covers how to create a service integration to Loggly with Managed Service for TimescaleDB.

Creating a Loggly service integration

1. Navigate to SolarWinds Loggly and create or log in to your account.

2. From the Loggly Home screen, navigate to Logs→Source Setup. Click Customer Tokens from the top menu bar.

3. On the Customer Tokens page, click Add New to create a new token. Give your token a name, and click Save. Copy your new token to your clipboard.

4. Log in to your Managed Service for TimescaleDB account, and navigate to Service Integrations.

5. In the Service Integrations page, navigate to Syslog, and click Add new endpoint.

6. In the Create new syslog endpoint dialog, complete these fields:

   • In the Endpoint name field, type a name for your endpoint.
   • In the Server field, type logs-01.loggly.com.
   • In the Port field, type 514.
   • Uncheck the TLS checkbox.
   • In the Format field, select rfc5425.
   • In the Structured Data field, type <LOGGLY_TOKEN>@41058, using the Loggly token you copied earlier. You can also add a tag here, which you can use to more easily search for your logs in Loggly. For example, 8480330f5-aa09-46b0-b220-a0efa372b17b@41058 TAG="example-tag".

   Click Create to create the endpoint. When the endpoint has been created, it shows as an enabled service integration, with a green active indicator.

7. In the Loggly dashboard, navigate to Search to see your incoming logs. From here, you can create custom dashboards and view reports for your logs.

   

===== PAGE: https://docs.tigerdata.com/mst/integrations/metrics-datadog/ =====

Metrics and Datadog

Datadog is a popular cloud-based monitoring service. You can send metrics to Datadog using a metrics collection agent for graphing, service dashboards, alerting, and logging. Managed Service for TimescaleDB (MST) can send data directly to Datadog for monitoring. Datadog integrations are provided free of charge on Managed Service for TimescaleDB.

You need to create a Datadog API key, and use the key to enable metrics for your service.

Datadog logging is not currently supported on MST.

Prerequisites

Before you begin, make sure you have:

• Created a service.
• Signed up for Datadog, and can log in to your Datadog dashboard.
• Created an API key in your Datadog account. For more information about creating a Datadog API key, see Datadog API and Application Keys.

Upload a Datadog API key

To integrate MST with Datadog you need to upload the API key that you generated in your Datadog account to MST.

Uploading a Datadog API key to MST

1. In MST Console, choose the project you want to connect to Datadog, and click Integration Endpoints.
2. Select Datadog, then choose Create new.
3. In Add new Datadog service integration. complete these details:
   • In the Endpoint integration section, give your endpoint a name, and paste the API key from your Datadog dashboard. Ensure you choose the site location that matches where your Datadog service is hosted.
   • Optional: In the Endpoint tags section, you can add custom tags to help you manage your integrations.
4. Click Add endpoint to save the integration.

Activate Datadog integration for a service

When you have successfully added the endpoint, you can set up one of your service to send data to Datadog.

Activating Datadog integration for a service

1. Sign in to MST Console, navigate to Services, and select the service you want to monitor.

2. In the Integrations tab, go to External integrations section and select Datadog Metrics.

3. In the Datadog integration dialog, select the Datadog endpoint that you created.

4. Click Enable.

   The Datadog endpoint is listed under Enabled integrations for the service.

Datadog dashboards

When you have your Datadog integration set up successfully, you can use the Datadog dashboard editor to configure your visualizations. For more information, see the Datadog Dashboard documentation.

===== PAGE: https://docs.tigerdata.com/mst/integrations/prometheus-mst/ =====

Prometheus endpoint for Managed Service for TimescaleDB

You can get more insights into the performance of your service by monitoring it using Prometheus, a popular open source metrics-based systems monitoring solution.

Prerequisites

Before you begin, make sure you have:

• Created a service.
• Made a note of the Port and Host for your service.

Enabling Prometheus service integration

1. In MST Console, choose a project and navigate to Integration Endpoints.

2. In the Integration endpoints page, navigate to Prometheus, and click Create new.

3. In the Create new Prometheus endpoint dialog, complete these fields:

   • In the Endpoint name field, type a name for your endpoint.
   • In the Username field, type your username.
   • In the Password field, type your password.
   • Click Create to create the endpoint.

   These details are used when setting up your Prometheus installation, in the prometheus.yml configuration file. This allows you to make this Managed Service for TimescaleDB endpoint a target for Prometheus to scrape.

4. Use this sample configuration file to set up your Prometheus installation, by substituting <PORT>, <HOST>, <USER>, and <PASSWORD> with those of your service:

 global:
  scrape_interval:     10s
  evaluation_interval: 10s
 scrape_configs:
  - job_name: prometheus
    scheme: https
    static_configs:
      - targets: ['<HOST>:<PORT>']
    tls_config:
      insecure_skip_verify: true
    basic_auth:
      username: <USER>
      password: <PASSWORD>
 remote_write:
  - url: "http://<HOST>:9201/write"
 remote_read:
  - url: "http://<HOST>:9201/read"

1. In the MST Console, navigate to Services and select the service you want to monitor.

2. In the Integrations tab, go to External integrations section and select Prometheus.

3. In the Prometheus integrations dialog, select the Prometheus endpoint that you created.

4. Click Enable.

   The Prometheus endpoint is listed under Enabled integrations for the service.

===== PAGE: https://docs.tigerdata.com/mst/aiven-client/replicas-cli/ =====

Create a read-only replica using Aiven client

Read-only replicas enable you to perform read-only queries against the replica and reduce the load on the primary server. It is also a good way to optimize query response times across different geographical locations, because the replica can be placed in different regions or even different cloud providers.

Prerequisites

Before you begin, make sure you have:

• Created a service.
• Installed Aiven Client.

Creating a read-only replica of your service

1. In the Aiven client, connect to your service.

2. Switch to the project that contains the service you want to create a read-only replica for:

   avn project switch <PROJECT>
   

3. List the MST_SERVICE_SHORTs in the project, and make a note of the service that you want to create a read-only replica for. It is listed under theSERVICE_NAME column in the output:

   avn service list
   

4. Get the details of the service that you want to fork:

   avn service get <SERVICE_NAME>
   

5. Create a read-only replica:

   avn service create <NAME_OF_REPLICA> --project <PROJECT_ID>\
   -t pg --plan <PLAN_TYPE> --cloud timescale-aws-us-east-1\
   -c pg_read_replica=true\
   -c service_to_fork_from=<NAME_OF_SERVICE_TO_FORK>\
   -c pg_version=11 -c variant=timescale
   

Example

To create a fork named replica-fork for a service named timescaledb with these parameters:

• PROJECT_ID: fork-project
• CLOUD_NAME: timescale-aws-us-east-1
• PLAN_TYPE: timescale-basic-100-compute-optimized

avn service create replica-fork --project fork-project\
-t pg --plan timescale-basic-100-compute-optimized\
--cloud timescale-aws-us-east-1 -c pg_read_replica=true\
-c service_to_fork_from=timescaledb -c\
pg_version=11 -c variant=timescale

You can switch to project-fork and view the newly created replica-fork using:

avn service list

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-docker-based/ =====

1. Run the TimescaleDB Docker image

   The TimescaleDB HA Docker image offers the most complete TimescaleDB experience. It uses Ubuntu, includes TimescaleDB Toolkit, and support for PostGIS and Patroni.

   To install the latest release based on Postgres 17:

   docker pull timescale/timescaledb-ha:pg17
   

   TimescaleDB is pre-created in the default Postgres database and is added by default to any new database you create in this image.

2. Run the container

   Replace </a/local/data/folder> with the path to the folder you want to keep your data in the following command.

   docker run -d --name timescaledb -p 5432:5432  -v </a/local/data/folder>:/pgdata -e PGDATA=/pgdata -e POSTGRES_PASSWORD=password timescale/timescaledb-ha:pg17
   

   If you are running multiple container instances, change the port each Docker instance runs on.

   On UNIX-based systems, Docker modifies Linux IP tables to bind the container. If your system uses Linux Uncomplicated Firewall (UFW), Docker may override your UFW port binding settings. To prevent this, add DOCKER_OPTS="--iptables=false" to /etc/default/docker.

3. Connect to a database on your Postgres instance

   The default user and database are both postgres. You set the password in POSTGRES_PASSWORD in the previous step. The default command to connect to Postgres is:

   psql -d "postgres://postgres:password@localhost/postgres"
   

4. Check that TimescaleDB is installed

   \dx
   

   You see the list of installed extensions:

   Name         | Version |   Schema   |                                      Description
   ---------------------+---------+------------+---------------------------------------------------------------------------------------
   plpgsql             | 1.0     | pg_catalog | PL/pgSQL procedural language
   timescaledb         | 2.20.3  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
   timescaledb_toolkit | 1.21.0  | public     | Library of analytical hyperfunctions, time-series pipelining, and other SQL utilities
   (3 rows)
   

   Press q to exit the list of extensions.

More Docker options

If you want to access the container from the host but avoid exposing it to the outside world, you can bind to 127.0.0.1 instead of the public interface, using this command:

docker run -d --name timescaledb -p 127.0.0.1:5432:5432 \
-v </a/local/data/folder>:/pgdata -e PGDATA=/pgdata -e POSTGRES_PASSWORD=password timescale/timescaledb-ha:pg17

If you don't want to install psql and other Postgres client tools locally, or if you are using a Microsoft Windows host system, you can connect using the version of psql that is bundled within the container with this command:

docker exec -it timescaledb psql -U postgres

When you install TimescaleDB using a Docker container, the Postgres settings are inherited from the container. In most cases, you do not need to adjust them. However, if you need to change a setting, you can add -c setting=value to your Docker run command. For more information, see the Docker documentation.

The link provided in these instructions is for the latest version of TimescaleDB on Postgres 17. To find other Docker tags you can use, see the Dockerhub repository.

View logs in Docker

If you have TimescaleDB installed in a Docker container, you can view your logs using Docker, instead of looking in /var/lib/logs or /var/logs. For more information, see the Docker documentation on logs.

1. Run the TimescaleDB Docker image

   The light-weight TimescaleDB Docker image uses Alpine and does not contain TimescaleDB Toolkit or support for PostGIS and Patroni.

   To install the latest release based on Postgres 17:

   docker pull timescale/timescaledb:latest-pg17
   

   TimescaleDB is pre-created in the default Postgres database and added by default to any new database you create in this image.

2. Run the container

   docker run -v </a/local/data/folder>:/pgdata -e PGDATA=/pgdata \
       -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg17
   

   If you are running multiple container instances, change the port each Docker instance runs on.

   On UNIX-based systems, Docker modifies Linux IP tables to bind the container. If your system uses Linux Uncomplicated Firewall (UFW), Docker may override your UFW port binding settings. To prevent this, add DOCKER_OPTS="--iptables=false" to /etc/default/docker.

3. Connect to a database on your Postgres instance

   The default user and database are both postgres. You set the password in POSTGRES_PASSWORD in the previous step. The default command to connect to Postgres in this image is:

   psql -d "postgres://postgres:password@localhost/postgres"
   

4. Check that TimescaleDB is installed

   \dx
   

You see the list of installed extensions:

```sql
Name         | Version |   Schema   |                                      Description
---------------------+---------+------------+---------------------------------------------------------------------------------------
plpgsql             | 1.0     | pg_catalog | PL/pgSQL procedural language
timescaledb         | 2.20.3  | public     | Enables scalable inserts and complex queries for time-series data (Community Edition)
```

Press `q` to exit the list of extensions.

More Docker options

If you want to access the container from the host but avoid exposing it to the outside world, you can bind to 127.0.0.1 instead of the public interface, using this command:

docker run -v </a/local/data/folder>:/pgdata -e PGDATA=/pgdata \
  -d --name timescaledb -p 127.0.0.1:5432:5432 \
  -e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg17

If you don't want to install psql and other Postgres client tools locally, or if you are using a Microsoft Windows host system, you can connect using the version of psql that is bundled within the container with this command:

docker exec -it timescaledb psql -U postgres

Existing containers can be stopped using docker stop and started again with docker start while retaining their volumes and data. When you create a new container using the docker run command, by default you also create a new data volume. When you remove a Docker container with docker rm, the data volume persists on disk until you explicitly delete it. You can use the docker volume ls command to list existing docker volumes. If you want to store the data from your Docker container in a host directory, or you want to run the Docker image on top of an existing data directory, you can specify the directory to mount a data volume using the -v flag:

docker run -d --name timescaledb -p 5432:5432 \
-v </your/data/dir>:/pgdata -e PGDATA=/pgdata \
-e POSTGRES_PASSWORD=password timescale/timescaledb:latest-pg17

When you install TimescaleDB using a Docker container, the Postgres settings are inherited from the container. In most cases, you do not need to adjust them. However, if you need to change a setting, you can add -c setting=value to your Docker run command. For more information, see the Docker documentation.

The link provided in these instructions is for the latest version of TimescaleDB on Postgres 16. To find other Docker tags you can use, see the Dockerhub repository.

View logs in Docker

If you have TimescaleDB installed in a Docker container, you can view your logs using Docker, instead of looking in /var/log. For more information, see the Docker documentation on logs.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-source-based/ =====

1. Install the latest Postgres source

   1. At the command prompt, clone the TimescaleDB GitHub repository:

      git clone https://github.com/timescale/timescaledb
      

   2. Change into the cloned directory:

      cd timescaledb
      

   3. Checkout the latest release. You can find the latest release tag on our Releases page:

      git checkout 2.17.2
      

      This command produces an error that you are now in detached head state. It is expected behavior, and it occurs because you have checked out a tag, and not a branch. Continue with the steps in this procedure as normal.

2. Build the source

   1. Bootstrap the build system:

      ./bootstrap
      

      bootstrap.bat
      

      For installation on Microsoft Windows, you might need to add the pg_config and cmake file locations to your path. In the Windows Search tool, search for system environment variables. The path for pg_config should be C:\Program Files\PostgreSQL\<version>\bin. The path for cmake is within the Visual Studio directory.

   2. Build the extension:

      cd build && make
      

      cmake --build ./build --config Release
      

3. Install TimescaleDB

   make install
   

   cmake --build ./build --config Release --target install
   

4. Configure Postgres

   If you have more than one version of Postgres installed, TimescaleDB can only be associated with one of them. The TimescaleDB build scripts use pg_config to find out where Postgres stores its extension files, so you can use pg_config to find out which Postgres installation TimescaleDB is using.

   1. Locate the postgresql.conf configuration file:

      psql -d postgres -c "SHOW config_file;"
      

   2. Open the postgresql.conf file and update shared_preload_libraries to:

      shared_preload_libraries = 'timescaledb'
      

      If you use other preloaded libraries, make sure they are comma separated.

   3. Tune your Postgres instance for TimescaleDB

      sudo timescaledb-tune
      

      This script is included with the timescaledb-tools package when you install TimescaleDB. For more information, see configuration.

   4. Restart the Postgres instance:

      service postgresql restart
      

      pg_ctl restart
      

5. Set the user password

   1. Log in to Postgres as postgres

      sudo -u postgres psql
      

      You are in the psql shell.

   2. Set the password for postgres

      \password postgres
      

      When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-homebrew-based/ =====

1. Install Homebrew, if you don't already have it:

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   

   For more information about Homebrew, including installation instructions, see the Homebrew documentation.

2. At the command prompt, add the TimescaleDB Homebrew tap:

   brew tap timescale/tap
   

3. Install TimescaleDB and psql:

   brew install timescaledb libpq
   

4. Update your path to include psql.

   brew link --force libpq
   

   On Intel chips, the symbolic link is added to /usr/local/bin. On Apple Silicon, the symbolic link is added to /opt/homebrew/bin.

5. Run the timescaledb-tune script to configure your database:

timescaledb-tune --quiet --yes

1. Change to the directory where the setup script is located. It is typically, located at /opt/homebrew/Cellar/timescaledb/<VERSION>/bin/, where <VERSION> is the version of timescaledb that you installed:

cd /opt/homebrew/Cellar/timescaledb/<VERSION>/bin/

1. Run the setup script to complete installation.

   ./timescaledb_move.sh
   

2. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

3. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-macports-based/ =====

1. Install MacPorts by downloading and running the package installer.

   For more information about MacPorts, including installation instructions, see the MacPorts documentation.

2. Install TimescaleDB and psql:

   sudo port install timescaledb libpqxx
   

   To view the files installed, run:

   port contents timescaledb libpqxx
   

   MacPorts does not install the timescaledb-tools package or run the timescaledb-tune script. For more information about tuning your database, see the TimescaleDB tuning tool.

3. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

4. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== PAGE: https://docs.tigerdata.com/_partials/_install-self-hosted-windows-based/ =====

1. Install the latest version of Postgres and psql

   1. Download Postgres, then run the installer.

      1. In the Select Components dialog, check Command Line Tools, along with any other components you want to install, and click Next.

      2. Complete the installation wizard.

   2. Check that you can run pg_config. If you cannot run pg_config from the command line, in the Windows Search tool, enter system environment variables. The path should be C:\Program Files\PostgreSQL\<version>\bin.

2. Install TimescaleDB

   1. Unzip the TimescaleDB installer to <install_dir>, that is, your selected directory.

      Best practice is to use the latest version.

   2. In <install_dir>\timescaledb, right-click setup.exe, then choose Run as Administrator.

   3. Complete the installation wizard.

      If you see an error like could not load library "C:/Program Files/PostgreSQL/17/lib/timescaledb-2.17.2.dll": The specified module could not be found., use Dependencies to ensure that your system can find the compatible DLLs for this release of TimescaleDB.

3. Tune your Postgres instance for TimescaleDB

   Run the timescaledb-tune script included in the timescaledb-tools package with TimescaleDB. For more information, see configuration.

4. Log in to Postgres as postgres

   sudo -u postgres psql
   

   You are in the psql shell.

5. Set the password for postgres

   \password postgres
   

   When you have set the password, type \q to exit psql.

===== LINK REFERENCES =====