diff --git a/documentation/cookbook/demo-data-schema.md b/documentation/cookbook/demo-data-schema.md index c179eaa16..ff47c426d 100644 --- a/documentation/cookbook/demo-data-schema.md +++ b/documentation/cookbook/demo-data-schema.md @@ -1,5 +1,5 @@ --- -title: Demo Data Schema +title: Demo data schema sidebar_label: Demo data schema description: Schema and structure of the FX market data and cryptocurrency trades available on demo.questdb.io --- @@ -19,11 +19,11 @@ The demo instance provides two independent datasets: --- -## FX Market Data (Simulated) +## FX market data (simulated) The FX dataset contains simulated foreign exchange market data for 30 currency pairs. We fetch real reference prices from Yahoo Finance every few seconds, but all order book levels and price updates are generated algorithmically based on these reference prices. -### core_price Table +### core_price table The `core_price` table contains individual FX price updates from various liquidity providers. Each row represents a bid/ask quote update for a specific currency pair from a specific ECN. @@ -32,16 +32,16 @@ The `core_price` table contains individual FX price updates from various liquidi ```sql title="core_price table structure" CREATE TABLE 'core_price' ( timestamp TIMESTAMP, - symbol SYMBOL CAPACITY 16384 CACHE, - ecn SYMBOL CAPACITY 256 CACHE, + symbol SYMBOL, + ecn SYMBOL, bid_price DOUBLE, bid_volume LONG, ask_price DOUBLE, ask_volume LONG, - reason SYMBOL CAPACITY 256 CACHE, + reason SYMBOL, indicator1 DOUBLE, indicator2 DOUBLE -) timestamp(timestamp) PARTITION BY HOUR TTL 3 DAYS WAL; +) timestamp(timestamp) PARTITION BY HOUR TTL 3 DAYS; ``` #### Columns @@ -58,7 +58,7 @@ CREATE TABLE 'core_price' ( The table tracks **30 currency pairs**: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, USDCAD, NZDUSD, EURJPY, GBPJPY, EURGBP, AUDJPY, CADJPY, NZDJPY, EURAUD, EURNZD, AUDNZD, GBPAUD, GBPNZD, AUDCAD, NZDCAD, EURCAD, EURCHF, GBPCHF, USDNOK, USDSEK, USDZAR, USDMXN, USDSGD, USDHKD, USDTRY. -#### Sample Data +#### Sample data ```questdb-sql demo title="Recent core_price updates" SELECT * FROM core_price @@ -81,7 +81,7 @@ LIMIT -10; | 2025-12-18T11:46:13.066700Z | CADJPY | Currenex | 113.63 | 20300827 | 114.11 | 19720915 | normal | 0.55 | | | 2025-12-18T11:46:13.071607Z | NZDJPY | Currenex | 89.95 | 35284228 | 90.46 | 30552528 | liquidity_event | 0.69 | | -### market_data Table +### market_data table The `market_data` table contains order book snapshots for currency pairs. Each row represents a complete view of the order book at a specific timestamp, with bid and ask prices and volumes stored as 2D arrays. @@ -109,7 +109,7 @@ The arrays are structured so that: - `asks[1]` contains ask prices (ascending order - lowest first) - `asks[2]` contains corresponding ask volumes -#### Sample Query +#### Sample query ```questdb-sql demo title="Recent order book snapshots" SELECT timestamp, symbol, @@ -132,7 +132,7 @@ LIMIT -5; Each order book snapshot contains 40 bid levels and 40 ask levels. -### fx_trades Table +### fx_trades table The `fx_trades` table contains simulated FX trade executions. Each row represents a trade that executed against the order book, with realistic partial fills and level walking. @@ -150,7 +150,7 @@ CREATE TABLE 'fx_trades' ( quantity DOUBLE, counterparty SYMBOL, order_id UUID -) timestamp(timestamp) PARTITION BY HOUR TTL 1 MONTH WAL; +) timestamp(timestamp) PARTITION BY HOUR TTL 1 MONTH; ``` #### Columns @@ -166,7 +166,7 @@ CREATE TABLE 'fx_trades' ( - **`counterparty`** - 20-character LEI (Legal Entity Identifier) of the counterparty - **`order_id`** - Parent order identifier (multiple trades can share the same `order_id` for partial fills) -#### Sample Data +#### Sample data ```questdb-sql demo title="Recent FX trades" SELECT * FROM fx_trades @@ -189,36 +189,36 @@ LIMIT -10; | 2026-01-12T12:18:57.509773474Z | GBPUSD | Currenex | ae6b771b-5abd-44c7-9e0e-3527ce6fb5b4 | sell | false | 1.3404 | 62305.0 | 006728CF215E44412D18 | 54ff8191-1891-4a5c-8b67-d5cd961ec5e8 | | 2026-01-12T12:18:57.334732460Z | USDTRY | EBS | 469637a5-6553-4aad-aad9-f7114c8a442d | sell | true | 43.1 | 101177.0 | 002CAC92E93AB4B3D30C | 2ce77a03-0f21-4241-8ca7-903080848dc0 | -### FX Materialized Views +### FX materialized views The FX dataset includes several materialized views providing pre-aggregated data at different time intervals: -#### Best Bid/Offer (BBO) Views +#### Best bid/offer (BBO) views - **`bbo_1s`** - Best bid and offer aggregated every 1 second - **`bbo_1m`** - Best bid and offer aggregated every 1 minute - **`bbo_1h`** - Best bid and offer aggregated every 1 hour - **`bbo_1d`** - Best bid and offer aggregated every 1 day -#### Core Price Aggregations +#### Core price aggregations - **`core_price_1s`** - Core prices aggregated every 1 second - **`core_price_1d`** - Core prices aggregated every 1 day -#### Market Data OHLC +#### Market data OHLC - **`market_data_ohlc_1m`** - Open, High, Low, Close candlesticks at 1-minute intervals - **`market_data_ohlc_15m`** - OHLC candlesticks at 15-minute intervals - **`market_data_ohlc_1d`** - OHLC candlesticks at 1-day intervals -#### FX Trades OHLC +#### FX trades OHLC - **`fx_trades_ohlc_1m`** - OHLC candlesticks from trade executions at 1-minute intervals - **`fx_trades_ohlc_1h`** - OHLC candlesticks from trade executions at 1-hour intervals These views are continuously updated and optimized for dashboard and analytics queries on FX data. -### FX Data Volume +### FX data volume - **`market_data`**: Approximately **160 million rows** per day (order book snapshots) - **`core_price`**: Approximately **73 million rows** per day (price updates across all ECNs and symbols) @@ -226,11 +226,11 @@ These views are continuously updated and optimized for dashboard and analytics q --- -## Cryptocurrency Trades (Real) +## Cryptocurrency trades (real) The cryptocurrency dataset contains **real market data** streamed live from the OKX exchange using FeedHandler. These are actual executed trades, not simulated data. -### trades Table +### trades table The `trades` table contains real cryptocurrency trade data. Each row represents an actual executed trade for a cryptocurrency pair. @@ -243,7 +243,7 @@ CREATE TABLE 'trades' ( price DOUBLE, amount DOUBLE, timestamp TIMESTAMP -) timestamp(timestamp) PARTITION BY DAY WAL; +) timestamp(timestamp) PARTITION BY DAY; ``` #### Columns @@ -254,9 +254,9 @@ CREATE TABLE 'trades' ( - **`price`** - Execution price of the trade - **`amount`** - Trade size (volume in base currency) -The table tracks **12 cryptocurrency pairs**: ADA-USDT, AVAX-USD, BTC-USDT, DAI-USD, DOT-USD, ETH-BTC, ETH-USDT, LTC-USD, SOL-BTC, SOL-USD, UNI-USD, XLM-USD. +The table tracks **12 cryptocurrency pairs**: ADA-USDT, AVAX-USDT, BTC-USDT, DAI-USDT, DOT-USDT, ETH-BTC, ETH-USDT, LTC-USDT, SOL-BTC, SOL-USDT, UNI-USDT, XLM-USDT. -#### Sample Data +#### Sample data ```questdb-sql demo title="Recent cryptocurrency trades" SELECT * FROM trades @@ -268,40 +268,40 @@ LIMIT -10; | symbol | side | price | amount | timestamp | | -------- | ---- | ------- | ---------- | --------------------------- | | BTC-USDT | buy | 85721.6 | 0.00045714 | 2025-12-18T19:31:11.203000Z | -| BTC-USD | buy | 85721.6 | 0.00045714 | 2025-12-18T19:31:11.203000Z | +| BTC-USDT | buy | 85721.6 | 0.00045714 | 2025-12-18T19:31:11.203000Z | | BTC-USDT | buy | 85726.6 | 0.00001501 | 2025-12-18T19:31:11.206000Z | -| BTC-USD | buy | 85726.6 | 0.00001501 | 2025-12-18T19:31:11.206000Z | +| BTC-USDT | buy | 85726.6 | 0.00001501 | 2025-12-18T19:31:11.206000Z | +| BTC-USDT | buy | 85726.9 | 0.000887 | 2025-12-18T19:31:11.206000Z | | BTC-USDT | buy | 85726.9 | 0.000887 | 2025-12-18T19:31:11.206000Z | -| BTC-USD | buy | 85726.9 | 0.000887 | 2025-12-18T19:31:11.206000Z | | BTC-USDT | buy | 85731.3 | 0.00004393 | 2025-12-18T19:31:11.206000Z | -| BTC-USD | buy | 85731.3 | 0.00004393 | 2025-12-18T19:31:11.206000Z | +| BTC-USDT | buy | 85731.3 | 0.00004393 | 2025-12-18T19:31:11.206000Z | +| ETH-USDT | sell | 2827.54 | 0.006929 | 2025-12-18T19:31:11.595000Z | | ETH-USDT | sell | 2827.54 | 0.006929 | 2025-12-18T19:31:11.595000Z | -| ETH-USD | sell | 2827.54 | 0.006929 | 2025-12-18T19:31:11.595000Z | -### Cryptocurrency Materialized Views +### Cryptocurrency materialized views The cryptocurrency dataset includes materialized views for aggregated trade data: -#### Trades Aggregations +#### Trades aggregations - **`trades_latest_1d`** - Latest trade data aggregated daily - **`trades_OHLC_15m`** - OHLC candlesticks for cryptocurrency trades at 15-minute intervals These views are continuously updated and provide faster query performance for cryptocurrency trade analysis. -### Cryptocurrency Data Volume +### Cryptocurrency data volume - **`trades`**: Approximately **3.7 million rows** per day (real cryptocurrency trades) --- -## Data Retention +## Data retention **FX tables** (`core_price` and `market_data`) use a **3-day TTL (Time To Live)**, meaning data older than 3 days is automatically removed. This keeps the demo instance responsive while providing sufficient recent data. **Cryptocurrency trades table** has **no retention policy** and contains historical data dating back to **March 8, 2022**. This provides over 3 years of real cryptocurrency trade history for long-term analysis and backtesting. -## Using the Demo Data +## Using the demo data You can run queries against both datasets directly on [demo.questdb.com](https://demo.questdb.io). Throughout the Cookbook, recipes using demo data will include a direct link to execute the query. diff --git a/documentation/cookbook/index.md b/documentation/cookbook/index.md index f71ff71b4..4089f536b 100644 --- a/documentation/cookbook/index.md +++ b/documentation/cookbook/index.md @@ -1,12 +1,12 @@ --- -title: Cookbook Overview +title: Cookbook overview sidebar_label: Overview description: Quick recipes and practical examples for common QuestDB tasks and queries --- The Cookbook is a collection of **short, actionable recipes** that demonstrate how to accomplish specific tasks with QuestDB. Each recipe follows a problem-solution-result format, making it easy to find and apply solutions quickly. -## What is the Cookbook? +## What is the cookbook? Unlike comprehensive reference documentation, the Cookbook focuses on practical examples for: @@ -24,7 +24,7 @@ The Cookbook is organized into three main sections: - **Programmatic** - Language-specific client examples and integration patterns - **Operations** - Deployment, configuration, and operational tasks -## Running the Examples +## Running the examples **Most recipes run directly on our [live demo instance at demo.questdb.com](https://demo.questdb.com)** without any local setup. Queries that can be executed on the demo site are marked with a direct link to run them. @@ -32,7 +32,7 @@ For recipes that require write operations or specific configuration, the recipe The demo instance contains live FX market data with tables for core prices and order book snapshots. See the [Demo Data Schema](/docs/cookbook/demo-data-schema/) page for details about available tables and their structure. -## Using the Cookbook +## Using the cookbook Each recipe follows a consistent format: diff --git a/documentation/cookbook/integrations/grafana/dynamic-table-queries.md b/documentation/cookbook/integrations/grafana/dynamic-table-queries.md index b9422b5e4..eb2e3b781 100644 --- a/documentation/cookbook/integrations/grafana/dynamic-table-queries.md +++ b/documentation/cookbook/integrations/grafana/dynamic-table-queries.md @@ -1,23 +1,23 @@ --- -title: Query Multiple Tables Dynamically in Grafana +title: Query multiple tables dynamically in Grafana sidebar_label: Dynamic table queries description: Use Grafana variables to dynamically query multiple tables with the same schema for time-series visualization --- Query multiple QuestDB tables dynamically in Grafana using dashboard variables. This is useful when you have many tables with identical schemas (e.g., sensor data, metrics from different sources) and want to visualize them together without hardcoding table names in your queries. -## Problem: Visualize Many Similar Tables +## Problem: Visualize many similar tables You have 100+ tables with the same structure (e.g., `sensor_1`, `sensor_2`, ..., `sensor_n`) and want to: 1. Display data from all tables on a single Grafana chart 2. Avoid manually updating queries when tables are added or removed 3. Allow users to select which tables to visualize via dashboard controls -## Solution: Use Grafana Variables with Dynamic SQL +## Solution: Use Grafana variables with dynamic SQL Create Grafana dashboard variables that query QuestDB for table names, then use string aggregation functions to build the SQL query dynamically. -### Step 1: Get Table Names +### Step 1: Get table names First, query QuestDB to get all relevant table names: @@ -28,7 +28,7 @@ WHERE table_name LIKE 'sensor_%'; This returns a list of all tables matching the pattern. -### Step 2: Create Grafana Variables +### Step 2: Create Grafana variables Create two dashboard variables to construct the dynamic query: @@ -59,7 +59,7 @@ WHERE table_name LIKE 'sensor_%'; This creates the column selection list with aggregation functions. -### Step 3: Use Variables in Dashboard Query +### Step 3: Use variables in dashboard query Now reference these variables in your Grafana chart query: @@ -77,7 +77,7 @@ FROM sensor_1 ASOF JOIN sensor_2 ASOF JOIN sensor_3 ASOF JOIN sensor_4 SAMPLE BY 1s FROM cast(1571176800000000 as timestamp) TO cast(1571349600000000 as timestamp) FILL(PREV); ``` -## How It Works +## How it works The solution uses three key QuestDB features: @@ -99,7 +99,7 @@ Combined with Grafana's variable interpolation: This ensures that even if tables update at different rates, you get a complete dataset with the most recent known value from each table. -## Adapting the Pattern +## Adapting the pattern **Filter by different patterns:** ```sql @@ -114,7 +114,7 @@ WHERE table_name LIKE 'sensor_%' AND table_name NOT IN ('sensor_test', 'sensor_backup') ``` -## Programmatic Alternative +## Programmatic alternative If you're not using Grafana, you can achieve the same result programmatically: @@ -142,7 +142,7 @@ If you're not using Grafana, you can achieve the same result programmatically: """ ``` -## Handling Different Sampling Intervals +## Handling different sampling intervals When tables have different update frequencies, use FILL to handle gaps: diff --git a/documentation/cookbook/integrations/grafana/overlay-timeshift.md b/documentation/cookbook/integrations/grafana/overlay-timeshift.md index d45865e19..aac4ca768 100644 --- a/documentation/cookbook/integrations/grafana/overlay-timeshift.md +++ b/documentation/cookbook/integrations/grafana/overlay-timeshift.md @@ -1,5 +1,5 @@ --- -title: Overlay Two Time Series with Time Shift +title: Overlay two time series with time shift sidebar_label: Overlay with timeshift description: Overlay yesterday's and today's data on the same Grafana chart using time shift --- diff --git a/documentation/cookbook/integrations/grafana/read-only-user.md b/documentation/cookbook/integrations/grafana/read-only-user.md index 27be21221..064b64af0 100644 --- a/documentation/cookbook/integrations/grafana/read-only-user.md +++ b/documentation/cookbook/integrations/grafana/read-only-user.md @@ -1,5 +1,5 @@ --- -title: Configure Read-Only User for Grafana +title: Configure read-only user for Grafana sidebar_label: Read-only user description: Set up a read-only PostgreSQL user for Grafana dashboards while maintaining admin access for DDL operations --- @@ -10,7 +10,7 @@ Configure a dedicated read-only user for Grafana to improve security by preventi For QuestDB Enterprise, use the comprehensive [Role-Based Access Control (RBAC)](/docs/security/rbac/) system to create granular user permissions and roles. The configuration below applies to QuestDB Open Source. ::: -## Problem: Separate Read and Write Access +## Problem: Separate read and write access You want to: 1. Connect Grafana with read-only credentials @@ -19,7 +19,7 @@ You want to: However, QuestDB's PostgreSQL wire protocol doesn't support standard PostgreSQL user management commands like `CREATE USER` or `GRANT`. -## Solution: Enable the Read-Only User +## Solution: Enable the read-only user QuestDB Open Source supports a built-in read-only user that can be enabled via configuration. This gives you two users: - **Admin user** (default: `admin`): Full access for DDL and DML operations @@ -58,7 +58,7 @@ docker run \ questdb/questdb:latest ``` -### Using the Read-Only User +### Using the read-only user After enabling, you have two separate users: diff --git a/documentation/cookbook/integrations/grafana/variable-dropdown.md b/documentation/cookbook/integrations/grafana/variable-dropdown.md index 700dc04e3..337caaccb 100644 --- a/documentation/cookbook/integrations/grafana/variable-dropdown.md +++ b/documentation/cookbook/integrations/grafana/variable-dropdown.md @@ -1,12 +1,12 @@ --- -title: Grafana Variable Dropdown with Name and Value +title: Grafana variable dropdown with name and value sidebar_label: Variable dropdown description: Create Grafana variable dropdowns that display one value but use another in queries using regex filters --- Create Grafana variable dropdowns where the displayed label differs from the value used in queries. This is useful when you want to show user-friendly names in the dropdown while using different values (like IDs, prices, or technical identifiers) in your actual SQL queries. -## Problem: Separate Display and Query Values +## Problem: Separate display and query values You want a Grafana variable dropdown that: - **Displays:** Readable labels like `"BTC-USDT"`, `"ETH-USDT"`, `"SOL-USDT"` @@ -22,11 +22,11 @@ For example, with this query result: You want the dropdown to show `"BTC-USDT"` but use `37779.62` in your queries. -## Solution: Use Regex Variable Filters +## Solution: Use regex variable filters When using the QuestDB data source plugin, you can use Grafana's regex variable filters to parse a concatenated string into separate `text` and `value` fields. -### Step 1: Concatenate Columns in Query +### Step 1: Concatenate columns in query First, combine both columns into a single string with a separator that doesn't appear in your data: @@ -51,7 +51,7 @@ BTC-USDC#60511.1 Each row is now a single string with symbol and price separated by `#`. -### Step 2: Apply Regex Filter in Grafana Variable +### Step 2: Apply regex filter in Grafana variable In your Grafana variable configuration: @@ -75,7 +75,7 @@ This regex pattern: - `#`: Matches the separator - `(?.*)`: Captures everything after `#` into the `value` group (the query value) -### Step 3: Use Variable in Queries +### Step 3: Use variable in queries Now you can reference the variable in your dashboard queries: @@ -89,7 +89,7 @@ WHERE price = $your_variable_name When a user selects "BTC-USDT" from the dropdown, Grafana will substitute the corresponding price value (`37779.62`) into the query. -## How It Works +## How it works Grafana's regex filter with named capture groups enables the separation: @@ -98,7 +98,7 @@ Grafana's regex filter with named capture groups enables the separation: 3. **`value` group**: Becomes the interpolated value in queries 4. **Pattern matching**: The regex must match the entire string returned by your query -### Regex Pattern Breakdown +### Regex pattern breakdown ```regex /(?[^#]+)#(?.*)/ @@ -111,7 +111,7 @@ Grafana's regex filter with named capture groups enables the separation: - `(?.*)`: Named capture group called "value" - `.*`: Zero or more characters of any type (captures rest of string) -## Choosing a Separator +## Choosing a separator Pick a separator that **never** appears in your data: @@ -128,9 +128,9 @@ Pick a separator that **never** appears in your data: - `,` - Common in CSV-like data - Space - Can cause parsing issues -## Alternative Patterns +## Alternative patterns -### Multiple Data Fields +### Multiple data fields If you need more than two fields, use additional separators: @@ -144,7 +144,7 @@ SELECT concat(symbol, '#', price, '#', volume) FROM trades; Now you have three captured groups, though Grafana's variable system typically only uses `text` and `value`. -### Numeric IDs with Descriptions +### Numeric IDs with descriptions Common pattern for entity selection: @@ -158,7 +158,7 @@ SELECT concat(name, '#', id) FROM users; Output in dropdown: User sees "John Doe", query uses `42`. -### Escaping Special Characters +### Escaping special characters If your data contains regex special characters, escape them in the pattern: @@ -172,7 +172,7 @@ SELECT concat(name, ' (', id, ')', '#', id) FROM users; /(?.*?)#(?\d+)/ ``` -## PostgreSQL Data Source Alternative +## PostgreSQL data source alternative If using the PostgreSQL data source (instead of the QuestDB plugin), you can use special column aliases: @@ -188,7 +188,7 @@ The PostgreSQL data source recognizes `__text` and `__value` as special column n **Note:** This works with the PostgreSQL data source plugin pointing to QuestDB, but NOT with the native QuestDB data source plugin. -## Adapting the Pattern +## Adapting the pattern **Different filter conditions:** ```sql diff --git a/documentation/cookbook/integrations/opcua-dense-format.md b/documentation/cookbook/integrations/opcua-dense-format.md index 74dc24c92..b5a0826bd 100644 --- a/documentation/cookbook/integrations/opcua-dense-format.md +++ b/documentation/cookbook/integrations/opcua-dense-format.md @@ -1,12 +1,12 @@ --- -title: Collect OPC-UA Data with Telegraf in Dense Format +title: Collect OPC-UA data with Telegraf in dense format sidebar_label: OPC-UA dense format description: Configure Telegraf to merge sparse OPC-UA metrics into dense rows for efficient storage and querying in QuestDB --- Configure Telegraf to collect OPC-UA industrial automation data and insert it into QuestDB in a dense format. By default, Telegraf creates one row per metric with sparse columns, but for QuestDB it's more efficient to merge all metrics from the same timestamp into a single dense row. -## Problem: Sparse Data Format +## Problem: Sparse data format When using Telegraf's OPC-UA input plugin with the default configuration, each metric value generates a separate row. Even when multiple metrics are collected at the same timestamp, they arrive as individual sparse rows: @@ -26,14 +26,14 @@ This wastes storage space and makes queries more complex. |------------------------------|------------|-----------|----------| | 2024-01-15T10:00:00.000000Z | 45.2 | 8192.0 | 1250.5 | -## Solution: Use Telegraf's Merge Aggregator +## Solution: Use Telegraf's merge aggregator Configure Telegraf to merge metrics with matching timestamps and tags before sending to QuestDB. This requires two key changes: 1. Add a common tag to all metrics 2. Use the `merge` aggregator to combine rows -### Complete Configuration +### Complete configuration ```toml [agent] @@ -82,9 +82,9 @@ Configure Telegraf to merge metrics with matching timestamps and tags before sen content_encoding = "identity" ``` -### Key Configuration Elements +### Key configuration elements -**1. Common Tag** +**1. Common tag** ```toml default_tags = { source="opcua_merge" } @@ -92,7 +92,7 @@ default_tags = { source="opcua_merge" } Adds the same tag value (`source="opcua_merge"`) to all metrics. The merge aggregator uses this to identify which metrics should be combined. -**2. Merge Aggregator** +**2. Merge aggregator** ```toml [[aggregators.merge]] @@ -103,7 +103,7 @@ Adds the same tag value (`source="opcua_merge"`) to all metrics. The merge aggre - `drop_original = true`: Discards the original sparse rows after merging - `tags = ["source"]`: Merges metrics with matching `source` tag values and the same timestamp -**3. QuestDB Output** +**3. QuestDB output** ```toml [[outputs.influxdb_v2]] @@ -114,7 +114,7 @@ Adds the same tag value (`source="opcua_merge"`) to all metrics. The merge aggre - Uses the InfluxDB Line Protocol (ILP) over HTTP - `content_encoding = "identity"`: Disables gzip compression (QuestDB doesn't require it) -## How It Works +## How it works The data flow is: @@ -123,7 +123,7 @@ The data flow is: 3. **Merge aggregator** → Combines rows with matching timestamp + `source` tag 4. **QuestDB output** → Sends merged dense rows via ILP -### Merging Logic +### Merging logic The merge aggregator combines metrics when: - **Timestamps match**: Metrics collected at the same moment @@ -131,11 +131,11 @@ The merge aggregator combines metrics when: If metrics have different timestamps or tag values, they won't be merged. -## Handling Tag Conflicts +## Handling tag conflicts If your OPC-UA nodes have additional tags with **different** values, those tags will prevent merging. Solutions: -### Remove Conflicting Tags +### Remove conflicting tags Use the `override` processor to remove unwanted tags: @@ -146,7 +146,7 @@ Use the `override` processor to remove unwanted tags: namespace = "" # Removes the 'namespace' tag ``` -### Convert Tags to Fields +### Convert tags to fields Use the `converter` processor to convert tags to fields (fields don't affect merging): @@ -158,7 +158,7 @@ Use the `converter` processor to convert tags to fields (fields don't affect mer This converts the tags to string fields, which won't interfere with the merge aggregator. -### Remove the Common Tag After Merging +### Remove the common tag after merging If you don't want the `source` tag in your final QuestDB table: @@ -169,7 +169,7 @@ If you don't want the `source` tag in your final QuestDB table: source = "" # Removes the 'source' tag ``` -## Environment Variables +## Environment variables Use environment variables for sensitive configuration: @@ -225,7 +225,7 @@ If you see sparse rows, check: - The merge aggregator is configured correctly - Timestamps are identical (not just close) -## Alternative: TCP Output +## Alternative: TCP output For higher throughput, use TCP instead of HTTP: @@ -248,7 +248,7 @@ Choose HTTP when: - You need error feedback - You're sending over the internet -## Multiple OPC-UA Sources +## Multiple OPC-UA sources To collect from multiple OPC-UA servers into separate tables: diff --git a/documentation/cookbook/operations/check-transaction-applied.md b/documentation/cookbook/operations/check-transaction-applied.md index 8efc11d9f..75a6c29c1 100644 --- a/documentation/cookbook/operations/check-transaction-applied.md +++ b/documentation/cookbook/operations/check-transaction-applied.md @@ -1,5 +1,5 @@ --- -title: Check Transaction Applied After Ingestion +title: Check transaction applied after ingestion sidebar_label: Check transaction applied description: Verify that all ingested rows to a WAL table are visible for queries in QuestDB --- diff --git a/documentation/cookbook/operations/copy-data-between-instances.md b/documentation/cookbook/operations/copy-data-between-instances.md index 17007bb17..f26f82be3 100644 --- a/documentation/cookbook/operations/copy-data-between-instances.md +++ b/documentation/cookbook/operations/copy-data-between-instances.md @@ -1,5 +1,5 @@ --- -title: Copy Data Between QuestDB Instances +title: Copy data between QuestDB instances sidebar_label: Copy data between instances description: Copy a subset of data from production to development QuestDB instances --- @@ -10,7 +10,7 @@ Copy a subset of data from one QuestDB instance to another for testing or develo You want to copy data between QuestDB instances. This method allows you to copy any arbitrary query result, but if you want a full database copy please check the [backup and restore documentation](/docs/operations/backup/). -## Solution: Table2Ilp Utility +## Solution: Table2Ilp utility QuestDB ships with a `utils` folder that includes a tool to read from one instance (using the PostgreSQL protocol) and write into another (using ILP). @@ -28,7 +28,7 @@ java -cp utils.jar io.questdb.cliutil.Table2Ilp \ This reads from the source instance using PostgreSQL wire protocol and writes to the destination using ILP. -## Alternative: Export Endpoint +## Alternative: Export endpoint You can also use [the export endpoint](/docs/query/rest-api/#exp---export-data) to export data to CSV or other formats. diff --git a/documentation/cookbook/operations/csv-import-milliseconds.md b/documentation/cookbook/operations/csv-import-milliseconds.md index 59178549d..330476439 100644 --- a/documentation/cookbook/operations/csv-import-milliseconds.md +++ b/documentation/cookbook/operations/csv-import-milliseconds.md @@ -1,24 +1,24 @@ --- -title: Import CSV with Millisecond Timestamps +title: Import CSV with millisecond timestamps sidebar_label: CSV import with milliseconds description: Import CSV files with epoch millisecond timestamps into QuestDB --- -Import CSV files containing epoch timestamps in milliseconds into QuestDB, which expects microseconds. +Import CSV files containing epoch timestamps in milliseconds into QuestDB. ## Problem -QuestDB does not support flags for timestamp conversion during CSV import. +QuestDB expects either date/timestamp literals, or epochs in microseconds or nanoseconds. -## Solution Options +## Solution options Here are the options available: -### Option 1: Pre-process the Dataset +### Option 1: Pre-process the dataset Convert timestamps from milliseconds to microseconds before import. If importing lots of data, create Parquet files, copy them to the QuestDB import folder, and read them with `read_parquet('file.parquet')`. Then use `INSERT INTO SELECT` to copy to another table. -### Option 2: Staging Table +### Option 2: Staging table Import into a non-partitioned table as DATE, then `INSERT INTO` a partitioned table as TIMESTAMP: @@ -56,7 +56,7 @@ DROP TABLE trades_staging; You would be using twice the storage temporarily, but then you can drop the initial staging table. -### Option 3: ILP Client +### Option 3: ILP client Read the CSV line-by-line and convert, then send via the ILP client. diff --git a/documentation/cookbook/operations/docker-compose-config.md b/documentation/cookbook/operations/docker-compose-config.md index b10be4ead..96e77ee7b 100644 --- a/documentation/cookbook/operations/docker-compose-config.md +++ b/documentation/cookbook/operations/docker-compose-config.md @@ -6,7 +6,7 @@ description: Override QuestDB configuration parameters using environment variabl You can override any QuestDB configuration parameter using environment variables in Docker Compose. This is useful for setting custom ports, authentication credentials, memory limits, and other operational settings without modifying configuration files. -## Environment Variable Format +## Environment variable format To override configuration parameters via environment variables: @@ -34,7 +34,7 @@ QUESTDB_PASSWORD=your_secure_password ::: -## Example: Custom PostgreSQL Credentials +## Example: Custom PostgreSQL credentials This Docker Compose file overrides the default PostgreSQL wire protocol credentials: @@ -73,7 +73,7 @@ This configuration: If you encounter permission errors with mounted volumes, ensure the QuestDB container user has write access to the host directory. You may need to set ownership with `chown -R 1000:1000 ./questdb_root` or run the container with a specific user ID. ::: -## Custom Data Directory Permissions +## Custom data directory permissions ```yaml title="Run with specific user/group for volume permissions" services: @@ -86,7 +86,7 @@ services: - ./questdb_data:/var/lib/questdb ``` -## Complete Configuration Reference +## Complete configuration reference For a full list of available configuration parameters, see: - [Server Configuration Reference](/docs/configuration/overview/) - All configurable parameters with descriptions diff --git a/documentation/cookbook/operations/optimize-many-tables.md b/documentation/cookbook/operations/optimize-many-tables.md index 2d5d4c47c..1cf58d221 100644 --- a/documentation/cookbook/operations/optimize-many-tables.md +++ b/documentation/cookbook/operations/optimize-many-tables.md @@ -1,5 +1,5 @@ --- -title: Optimize Disk and Memory Usage with Many Tables +title: Optimize disk and memory usage with many tables sidebar_label: Optimize for many tables description: Reduce memory and disk usage when running QuestDB with many tables by adjusting memory allocation and disk chunk sizes --- diff --git a/documentation/cookbook/operations/query-times-histogram.md b/documentation/cookbook/operations/query-times-histogram.md index 5f88ee633..0ecffa0a4 100644 --- a/documentation/cookbook/operations/query-times-histogram.md +++ b/documentation/cookbook/operations/query-times-histogram.md @@ -1,5 +1,5 @@ --- -title: Query Performance Histogram +title: Query performance histogram sidebar_label: Query times histogram description: Create histogram of query execution times using _query_trace table --- @@ -11,7 +11,7 @@ Create a histogram of query execution times using the `_query_trace` system tabl [Query tracing](/docs/concepts/deep-dive/query-tracing/) needs to be enabled for the `_query_trace` table to be populated. ::: -## Solution: Percentile-Based Histogram +## Solution: Percentile-based histogram We can create a subquery that first calculates the percentiles for each bucket, in this case at 10% intervals. Then on a second query we can do a `UNION` of 10 subqueries where each is doing a `CROSS JOIN` against the calculated percentiles and finding how many queries are below the threshold for the bucket. diff --git a/documentation/cookbook/operations/show-non-default-params.md b/documentation/cookbook/operations/show-non-default-params.md index 19ad614e7..46357fe71 100644 --- a/documentation/cookbook/operations/show-non-default-params.md +++ b/documentation/cookbook/operations/show-non-default-params.md @@ -1,5 +1,5 @@ --- -title: Show Parameters with Non-Default Values +title: Show parameters with non-default values sidebar_label: Show non-default params description: List all QuestDB configuration parameters that have been modified from their default values --- diff --git a/documentation/cookbook/operations/store-questdb-metrics.md b/documentation/cookbook/operations/store-questdb-metrics.md index 06cbfacaf..ffb596e5e 100644 --- a/documentation/cookbook/operations/store-questdb-metrics.md +++ b/documentation/cookbook/operations/store-questdb-metrics.md @@ -1,12 +1,12 @@ --- -title: Store QuestDB Metrics in QuestDB +title: Store QuestDB metrics in QuestDB sidebar_label: Store QuestDB metrics description: Scrape QuestDB Prometheus metrics using Telegraf and store them in QuestDB --- Store QuestDB's operational metrics in QuestDB itself by scraping Prometheus metrics using Telegraf. -## Solution: Telegraf Configuration +## Solution: Telegraf configuration You could use Prometheus to scrape those metrics, but you can also use any server agent that understands the Prometheus format. It turns out Telegraf has input plugins for Prometheus and output plugins for QuestDB, so you can use it to get the metrics from the endpoint and insert them into a QuestDB table. diff --git a/documentation/cookbook/operations/tls-pgbouncer.md b/documentation/cookbook/operations/tls-pgbouncer.md index 8f23cfa35..6db184d79 100644 --- a/documentation/cookbook/operations/tls-pgbouncer.md +++ b/documentation/cookbook/operations/tls-pgbouncer.md @@ -10,7 +10,7 @@ Configure PgBouncer to provide TLS termination for QuestDB Open Source PostgreSQ For QuestDB Enterprise, there is native TLS support, so you can connect directly with TLS or use PgBouncer with full TLS end-to-end encryption. ::: -## Solution: TLS Termination at PgBouncer +## Solution: TLS termination at PgBouncer QuestDB Open Source does not implement TLS on the PostgreSQL wire protocol, so TLS termination needs to be done at the PgBouncer level. diff --git a/documentation/cookbook/programmatic/cpp/missing-columns.md b/documentation/cookbook/programmatic/cpp/missing-columns.md index 204f28cdc..2c31b5698 100644 --- a/documentation/cookbook/programmatic/cpp/missing-columns.md +++ b/documentation/cookbook/programmatic/cpp/missing-columns.md @@ -1,5 +1,5 @@ --- -title: Handle Missing Columns in C++ Client +title: Handle missing columns in C++ client sidebar_label: Missing columns description: Send rows with optional columns using the QuestDB C++ client by conditionally calling column methods --- diff --git a/documentation/cookbook/programmatic/php/inserting-ilp.md b/documentation/cookbook/programmatic/php/inserting-ilp.md index f1d89cffc..2695b88cc 100644 --- a/documentation/cookbook/programmatic/php/inserting-ilp.md +++ b/documentation/cookbook/programmatic/php/inserting-ilp.md @@ -1,12 +1,12 @@ --- -title: Insert Data from PHP Using ILP +title: Insert data from PHP using ILP sidebar_label: Inserting via ILP description: Send time-series data from PHP to QuestDB using the InfluxDB Line Protocol --- QuestDB doesn't maintain an official PHP library, but since the ILP (InfluxDB Line Protocol) is text-based, you can easily send your data using PHP's built-in HTTP or socket functions, or use the official InfluxDB PHP client library. -## Available Approaches +## Available approaches This guide covers three methods for sending ILP data to QuestDB from PHP: @@ -26,7 +26,7 @@ This guide covers three methods for sending ILP data to QuestDB from PHP: - No acknowledgments - data loss possible - Manual implementation required -## ILP Protocol Overview +## ILP protocol overview The ILP protocol allows you to send data to QuestDB using a simple line-based text format: @@ -49,7 +49,7 @@ The format consists of: For complete ILP specification, see the [ILP reference documentation](/docs/ingestion/ilp/overview/). -## ILP Over HTTP +## ILP over HTTP QuestDB supports ILP data via HTTP or TCP. **HTTP is the recommended approach** for most use cases as it provides better reliability and easier debugging. @@ -59,7 +59,7 @@ To send data via HTTP: 3. Include ILP-formatted rows in the request body 4. For higher throughput, batch multiple rows in a single request -### HTTP Buffering Example +### HTTP buffering example The following PHP class provides buffered insertion with automatic flushing based on either row count or elapsed time: @@ -168,7 +168,7 @@ This class: For production use, consider adding error handling to check the HTTP response status and implement retry logic for failed requests. ::: -## Using the InfluxDB v2 PHP Client +## Using the InfluxDB v2 PHP client Another approach is to use the official [InfluxDB PHP client library](https://github.com/influxdata/influxdb-client-php), which supports the InfluxDB v2 write API. QuestDB is compatible with this API, making the client library a convenient option. @@ -201,7 +201,7 @@ When using the InfluxDB client with QuestDB: QuestDB only supports the **InfluxDB v2 write API** when using this client. Query operations are not supported through the InfluxDB client - use QuestDB's PostgreSQL wire protocol or REST API for queries instead. ::: -### Example Code +### Example code ```php title="Using InfluxDB v2 PHP client with QuestDB" close(); ?> ``` -### Benefits and Limitations +### Benefits and limitations The Point builder provides several advantages: - **Automatic ILP formatting and escaping** - No need to manually construct ILP strings @@ -270,7 +270,7 @@ The InfluxDB PHP client **cannot be used with custom timestamps** when writing t **If you need client-side timestamps:** Use the raw HTTP cURL approach (documented above) where you manually format the ILP string with full control over timestamp formatting. ::: -## ILP Over TCP Socket +## ILP over TCP socket TCP over socket provides higher throughput but is less reliable than HTTP. The message format is identical - only the transport changes. @@ -279,7 +279,7 @@ Use TCP when: - Your application can handle potential data loss on connection failures - You're willing to implement your own connection management and error handling -### TCP Socket Example +### TCP socket example Here's a basic example using PHP's socket functions: @@ -337,7 +337,7 @@ For production use with TCP, you should: TCP ILP does not provide acknowledgments for successful writes. If the connection drops, you may lose data without notification. For critical data, use HTTP ILP instead. ::: -## Choosing the Right Approach +## Choosing the right approach | Feature | HTTP (cURL) | HTTP (InfluxDB Client) | TCP Socket | |---------|-------------|------------------------|------------| diff --git a/documentation/cookbook/programmatic/ruby/inserting-ilp.md b/documentation/cookbook/programmatic/ruby/inserting-ilp.md index 32753da66..41ccd10c4 100644 --- a/documentation/cookbook/programmatic/ruby/inserting-ilp.md +++ b/documentation/cookbook/programmatic/ruby/inserting-ilp.md @@ -1,12 +1,12 @@ --- -title: Insert Data from Ruby Using ILP +title: Insert data from Ruby using ILP sidebar_label: Inserting via ILP description: Send time-series data from Ruby to QuestDB using the InfluxDB Line Protocol over HTTP --- Send time-series data from Ruby to QuestDB using the InfluxDB Line Protocol (ILP). While QuestDB doesn't maintain an official Ruby client, you can easily use the official InfluxDB Ruby gem to send data via ILP over HTTP, which QuestDB fully supports. -## Available Approaches +## Available approaches Two methods for sending ILP data from Ruby: @@ -22,7 +22,7 @@ Two methods for sending ILP data from Ruby: - Higher throughput, no dependencies - Requires: Built-in Ruby socket library -## Using the InfluxDB v2 Ruby Client +## Using the InfluxDB v2 Ruby client The InfluxDB v2 client provides a convenient Point builder API that works with QuestDB. @@ -38,7 +38,7 @@ Or add to your `Gemfile`: gem 'influxdb-client', '~> 3.1' ``` -### Example Code +### Example code ```ruby require 'influxdb-client' @@ -85,7 +85,7 @@ write_api.write(data: points) client.close! ``` -### Configuration Notes +### Configuration notes When using the InfluxDB client with QuestDB: @@ -95,7 +95,7 @@ When using the InfluxDB client with QuestDB: - **`precision`**: Use `NANOSECOND` for compatibility (QuestDB's native precision) - **`use_ssl`**: Set to `false` for local development, `true` for production with TLS -### Data Types +### Data types The InfluxDB client automatically handles type conversions: @@ -109,11 +109,11 @@ point = InfluxDB2::Point.new(name: 'measurements') .add_field('online', true) # BOOLEAN ``` -## TCP Socket Approach +## TCP socket approach For maximum control and performance, send ILP messages directly via TCP sockets. -### Basic TCP Example +### Basic TCP example ```ruby require 'socket' @@ -147,7 +147,7 @@ ensure end ``` -### ILP Message Format +### ILP message format The ILP format is: @@ -168,7 +168,7 @@ Breaking it down: readings,city=London,make=Omron temperature=23.5,humidity=0.343 1465839830100400000\n ``` -### Escaping Special Characters +### Escaping special characters ILP requires escaping for certain characters: @@ -188,7 +188,7 @@ escaped = escape_ilp(tag_value) # "London\\, UK" s.puts "readings,city=#{escaped} temperature=23.5\n" ``` -### Batching for Performance +### Batching for performance Send multiple rows in a single TCP write: @@ -224,7 +224,7 @@ ensure end ``` -## Comparison: InfluxDB Client vs TCP Socket +## Comparison: InfluxDB client vs TCP socket | Feature | InfluxDB Client | TCP Socket | |---------|----------------|------------| @@ -238,9 +238,9 @@ end | **Escaping** | Automatic | Manual implementation required | | **Recommended for** | Most applications | High-throughput scenarios, custom needs | -## Best Practices +## Best practices -### Connection Management +### Connection management **InfluxDB Client:** ```ruby @@ -265,7 +265,7 @@ ensure end ``` -### Error Handling +### Error handling **InfluxDB Client:** ```ruby @@ -290,7 +290,7 @@ rescue StandardError => e end ``` -### Timestamp Generation +### Timestamp generation Use nanosecond precision for maximum compatibility: @@ -312,7 +312,7 @@ timestamp = current_nanos timestamp = time_to_nanos(Time.parse("2024-09-05 14:30:00 UTC")) ``` -### Batching Strategy +### Batching strategy For high-throughput scenarios: diff --git a/documentation/cookbook/programmatic/tls-ca-configuration.md b/documentation/cookbook/programmatic/tls-ca-configuration.md index c0e80ad9d..dc8847476 100644 --- a/documentation/cookbook/programmatic/tls-ca-configuration.md +++ b/documentation/cookbook/programmatic/tls-ca-configuration.md @@ -1,5 +1,5 @@ --- -title: Configure TLS Certificate Authorities +title: Configure TLS certificate authorities sidebar_label: TLS CA configuration description: Configure TLS certificate authority validation for QuestDB clients --- @@ -16,7 +16,7 @@ When using the PostgreSQL wire interface, you can insert data passing `sslmode=r QuestDB clients support the `tls_ca` parameter, which has multiple values to configure certificate authority validation: -### Option 1: Use WebPKI and OS Certificate Roots (Recommended for Production) +### Option 1: Use WebPKI and OS certificate roots (recommended for production) If you want to accept both the webpki-root certificates plus whatever you have on the OS, pass `tls_ca=webpki_and_os_roots`: @@ -26,7 +26,7 @@ https::addr=localhost:9000;username=admin;password=quest;tls_ca=webpki_and_os_ro This will work with certificates signed by standard certificate authorities. -### Option 2: Use a Custom PEM File +### Option 2: Use a custom PEM file Point to a PEM-encoded certificate file for self-signed or custom CA certificates: @@ -36,7 +36,7 @@ https::addr=localhost:9000;username=admin;password=quest;tls_ca=pem_file;tls_roo This is useful for self-signed certificates or internal CAs. -### Option 3: Skip Verification (Development Only) +### Option 3: Skip verification (development only) For development environments with self-signed certificates, you might be tempted to disable verification by passing `tls_verify=unsafe_off`: @@ -50,7 +50,7 @@ This is a very bad idea for production and should only be used for testing on a **Note:** Some clients require enabling an optional feature (like `insecure-skip-verify` in Rust) before the `tls_verify=unsafe_off` parameter will work. Check your client's documentation for details. -## Available tls_ca Values +## Available tls_ca values | Value | Description | |-------|-------------| @@ -59,7 +59,7 @@ This is a very bad idea for production and should only be used for testing on a | `webpki_and_os_roots` | Both WebPKI and OS roots (recommended) | | `pem_file` | Load from a PEM file (requires `tls_roots` parameter) | -## Example: Rust Client +## Example: Rust client ```rust use questdb::ingress::{Sender, SenderBuilder}; diff --git a/documentation/cookbook/sql/advanced/array-from-string.md b/documentation/cookbook/sql/advanced/array-from-string.md index 4f598574e..d255dd3b4 100644 --- a/documentation/cookbook/sql/advanced/array-from-string.md +++ b/documentation/cookbook/sql/advanced/array-from-string.md @@ -1,5 +1,5 @@ --- -title: Create Arrays from String Literals +title: Create arrays from string literals sidebar_label: Array from string literal description: Cast string literals to array types in QuestDB --- diff --git a/documentation/cookbook/sql/advanced/conditional-aggregates.md b/documentation/cookbook/sql/advanced/conditional-aggregates.md index d68fb679b..c59472f81 100644 --- a/documentation/cookbook/sql/advanced/conditional-aggregates.md +++ b/documentation/cookbook/sql/advanced/conditional-aggregates.md @@ -1,5 +1,5 @@ --- -title: Multiple Conditional Aggregates +title: Multiple conditional aggregates sidebar_label: Conditional aggregates description: Calculate multiple conditional aggregates in a single query using CASE expressions --- @@ -18,7 +18,7 @@ You need to calculate various metrics from the same dataset with different condi Running separate queries is inefficient. -## Solution: CASE Within Aggregate Functions +## Solution: CASE within aggregate functions Use CASE expressions inside aggregates to calculate all metrics in one query: @@ -38,9 +38,18 @@ WHERE timestamp >= dateadd('d', -1, now()) GROUP BY symbol; ``` -## How It Works +Which returns: -### CASE Returns NULL for Non-Matching Rows + +| symbol | buy_count | sell_count | avg_buy_price | avg_sell_price | large_trade_volume | small_trade_volume | total_volume | +| -------- | --------- | ---------- | ----------------- | ------------------ | ------------------ | ------------------ | ------------------ | +| ETH-USDT | 262870 | 212163 | 3275.286678129868 | 3273.6747631773655 | 152042.02150799974 | 51934.917160999976 | 203976.93866900489 | +| BTC-USDT | 789959 | 712152 | 94286.52121793582 | 94304.92124321847 | 1713.1241887299993 | 8803.505760999722 | 10516.629949730019 | + + +## How it works + +### CASE returns NULL for non-matching rows ```sql count(CASE WHEN side = 'buy' THEN 1 END) @@ -51,7 +60,7 @@ count(CASE WHEN side = 'buy' THEN 1 END) - `count()` only counts non-NULL values - Result: counts only rows where side is 'buy' -### Aggregate Functions Ignore NULL +### Aggregate functions ignore NULL ```sql avg(CASE WHEN side = 'buy' THEN price END) diff --git a/documentation/cookbook/sql/advanced/consistent-histogram-buckets.md b/documentation/cookbook/sql/advanced/consistent-histogram-buckets.md index 67831ab5c..5734868f4 100644 --- a/documentation/cookbook/sql/advanced/consistent-histogram-buckets.md +++ b/documentation/cookbook/sql/advanced/consistent-histogram-buckets.md @@ -1,5 +1,5 @@ --- -title: Consistent Histogram Buckets +title: Consistent histogram buckets sidebar_label: Histogram buckets description: Generate histogram data with fixed bucket boundaries for consistent distribution analysis --- @@ -10,7 +10,7 @@ Create histograms with consistent bucket boundaries for distribution analysis. D A fixed bucket size works well for some data but poorly for others. For example, a bucket size of 0.5 produces a nice histogram for BTC trade amounts, but may produce just one or two buckets for assets with smaller typical values. -## Solution 1: Fixed Bucket Size +## Solution 1: Fixed bucket size When you know your data range, use a fixed bucket size: @@ -25,7 +25,7 @@ GROUP BY bucket ORDER BY bucket; ``` -### How It Works +### How it works ```sql floor(amount / 0.5) * 0.5 @@ -44,7 +44,7 @@ Examples: You must tune `@bucket_size` for your data range. A size that works for one symbol may not work for another. ::: -## Solution 2: Fixed Bucket Count (Dynamic Size) +## Solution 2: Fixed bucket count (dynamic size) To always get approximately N buckets regardless of the data range, calculate the bucket size dynamically: @@ -72,7 +72,7 @@ This calculates `(max - min) / 49` to create 50 evenly distributed buckets. The If there are fewer distinct values than requested buckets, or if some buckets have no data, you'll get fewer than 50 results. ::: -## Solution 3: Logarithmic Buckets +## Solution 3: Logarithmic buckets For data spanning multiple orders of magnitude: @@ -90,7 +90,7 @@ ORDER BY bucket; Each bucket covers one order of magnitude (0.001-0.01, 0.01-0.1, 0.1-1.0, etc.). -## Solution 4: Manual Buckets +## Solution 4: Manual buckets For simple categorical grouping: @@ -108,7 +108,7 @@ WHERE symbol = 'BTC-USDT' AND timestamp IN today() GROUP BY bucket; ``` -## Time-Series Histogram +## Time-series histogram Track distribution changes over time by combining with `SAMPLE BY`: diff --git a/documentation/cookbook/sql/advanced/general-and-sampled-aggregates.md b/documentation/cookbook/sql/advanced/general-and-sampled-aggregates.md index 3c5e04bd7..510fda63a 100644 --- a/documentation/cookbook/sql/advanced/general-and-sampled-aggregates.md +++ b/documentation/cookbook/sql/advanced/general-and-sampled-aggregates.md @@ -1,5 +1,5 @@ --- -title: General and Sampled Aggregates +title: General and sampled aggregates sidebar_label: General + sampled aggregates description: Combine overall statistics with time-bucketed aggregates using CROSS JOIN --- diff --git a/documentation/cookbook/sql/advanced/local-min-max.md b/documentation/cookbook/sql/advanced/local-min-max.md index 81f20ded2..47e731648 100644 --- a/documentation/cookbook/sql/advanced/local-min-max.md +++ b/documentation/cookbook/sql/advanced/local-min-max.md @@ -1,5 +1,5 @@ --- -title: Find Local Minimum and Maximum +title: Find local minimum and maximum sidebar_label: Local min and max description: Find the minimum and maximum values within a time range around each row --- @@ -10,7 +10,7 @@ Find the minimum and maximum values within a time window around each row to dete You want to find the local minimum and maximum bid price within a time range of each row - for example, the min/max within 1 second before and after each data point. -## Solution 1: Window Function (Past Only) +## Solution 1: Window function (past only) If you only need to look at **past data**, use a window function with `RANGE`: @@ -24,7 +24,7 @@ WHERE timestamp >= dateadd('m', -1, now()) AND symbol = 'EURUSD'; This returns the minimum and maximum bid price from the 1 second preceding each row. -## Solution 2: WINDOW JOIN (Past and Future) +## Solution 2: WINDOW JOIN (past and future) If you need to look at **both past and future data**, use a `WINDOW JOIN`. QuestDB window functions don't support `FOLLOWING`, but WINDOW JOIN allows bidirectional lookback: @@ -40,7 +40,7 @@ WHERE p.timestamp >= dateadd('m', -1, now()) AND p.symbol = 'EURUSD'; This returns the minimum and maximum bid price from 1 second before to 1 second after each row. -## When to Use Each Approach +## When to use each approach | Approach | Use When | |----------|----------| diff --git a/documentation/cookbook/sql/advanced/pivot-with-others.md b/documentation/cookbook/sql/advanced/pivot-with-others.md index 7c70a56ad..7d3b99f88 100644 --- a/documentation/cookbook/sql/advanced/pivot-with-others.md +++ b/documentation/cookbook/sql/advanced/pivot-with-others.md @@ -1,5 +1,5 @@ --- -title: Pivot with "Others" Column +title: Pivot with "Others" column sidebar_label: Pivot with Others description: Pivot specific values into columns while aggregating remaining values into an "Others" column using CASE statements --- @@ -72,7 +72,7 @@ LIMIT 5; Each timestamp now has a single row with specific symbols as columns, plus an "Others" column aggregating all remaining symbols. -## When to Use This Pattern +## When to use this pattern Use `CASE` statements instead of `PIVOT` when you need: - An "Others" or "Else" column to catch unspecified values diff --git a/documentation/cookbook/sql/advanced/rows-before-after-value-match.md b/documentation/cookbook/sql/advanced/rows-before-after-value-match.md index ca4e1f233..bf36be392 100644 --- a/documentation/cookbook/sql/advanced/rows-before-after-value-match.md +++ b/documentation/cookbook/sql/advanced/rows-before-after-value-match.md @@ -1,5 +1,5 @@ --- -title: Access Rows Before and After Current Row +title: Access rows before and after current row sidebar_label: Rows before/after description: Use LAG and LEAD window functions to access values from surrounding rows --- @@ -30,7 +30,7 @@ FROM core_price WHERE timestamp >= dateadd('m', -1, now()) AND symbol = 'EURUSD'; ``` -## How It Works +## How it works - **`LAG(column, N)`** - Gets the value from N rows **before** the current row (earlier in time) - **`LEAD(column, N)`** - Gets the value from N rows **after** the current row (later in time) diff --git a/documentation/cookbook/sql/advanced/sankey-funnel.md b/documentation/cookbook/sql/advanced/sankey-funnel.md index 653f9e2ce..c2bfa908a 100644 --- a/documentation/cookbook/sql/advanced/sankey-funnel.md +++ b/documentation/cookbook/sql/advanced/sankey-funnel.md @@ -1,5 +1,5 @@ --- -title: Sankey and Funnel Diagrams +title: Sankey and funnel diagrams sidebar_label: Sankey/funnel diagrams description: Create session-based analytics for Sankey diagrams and conversion funnels --- @@ -23,7 +23,7 @@ CREATE TABLE events ( ) TIMESTAMP(timestamp) PARTITION BY MONTH WAL; ``` -## Solution: Session Window Functions +## Solution: Session window functions By combining window functions and `CASE` statements: diff --git a/documentation/cookbook/sql/advanced/top-n-plus-others.md b/documentation/cookbook/sql/advanced/top-n-plus-others.md index 0cf6426ad..9ac7b0aeb 100644 --- a/documentation/cookbook/sql/advanced/top-n-plus-others.md +++ b/documentation/cookbook/sql/advanced/top-n-plus-others.md @@ -1,12 +1,12 @@ --- -title: Top N Plus Others Row +title: Top N plus others row sidebar_label: Top N + Others description: Group query results into top N rows plus an aggregated "Others" row using rank() and CASE expressions --- Create aggregated results showing the top N items individually, with all remaining items combined into a single "Others" row. This pattern is useful for dashboards and reports where you want to highlight the most important items while still showing the total. -## Problem: Show Top Items Plus Remainder +## Problem: Show top items plus remainder You want to display results like: @@ -21,7 +21,7 @@ You want to display results like: Instead of listing all symbols (which might be thousands), show the top 5 individually and aggregate the rest. -## Solution: Use rank() with CASE Statement +## Solution: Use rank() with CASE statement Use `rank()` to identify top N rows, then use `CASE` to group remaining rows: @@ -61,7 +61,7 @@ ORDER BY total_trades DESC; | AVAX-USDT | 5891 | | -Others- | 23456 | ← Sum of all other symbols -## How It Works +## How it works The query uses a three-step approach: @@ -118,7 +118,7 @@ ORDER BY total_trades DESC; - `rank()`: May include more than N if there are ties at position N - `row_number()`: Always exactly N in top tier (breaks ties arbitrarily) -## Adapting the Pattern +## Adapting the pattern **Different top N:** ```sql @@ -185,7 +185,7 @@ ORDER BY total_trades DESC; ``` -## Multiple Grouping Columns +## Multiple grouping columns Show top N for multiple dimensions: @@ -215,7 +215,7 @@ ORDER BY side, total_trades DESC; This shows top 3 symbols separately for buy and sell sides. -## Visualization Considerations +## Visualization considerations This pattern is particularly useful for charts: diff --git a/documentation/cookbook/sql/advanced/unpivot-table.md b/documentation/cookbook/sql/advanced/unpivot-table.md index 47a071931..d047a3ffa 100644 --- a/documentation/cookbook/sql/advanced/unpivot-table.md +++ b/documentation/cookbook/sql/advanced/unpivot-table.md @@ -1,12 +1,12 @@ --- -title: Unpivoting Query Results +title: Unpivoting query results sidebar_label: Unpivoting results description: Convert wide-format data to long format using UNION ALL --- Transform wide-format data (multiple columns) into long format (rows) using UNION ALL. -## Problem: Wide Format to Long Format +## Problem: Wide format to long format You have query results with multiple columns where only one column has a value per row: @@ -30,7 +30,7 @@ You want to convert this to a format where side and price are explicit: | 08:10:00 | ETH-USDT | buy | 3678.01 | | 08:10:00 | ETH-USDT | sell | 3678.00 | -## Solution: UNION ALL with Literal Values +## Solution: UNION ALL with literal values Use UNION ALL to stack columns as rows, then filter NULL values: @@ -68,9 +68,9 @@ ORDER BY timestamp; | 08:10:00 | ETH-USDT | buy | 3678.01 | | 08:10:00 | ETH-USDT | sell | 3678.00 | -## How It Works +## How it works -### Step 1: Create Wide Format (if needed) +### Step 1: Create wide format (if needed) If your data is already in narrow format, you may need to pivot first: @@ -101,7 +101,7 @@ WHERE price IS NOT NULL Removes rows where the price column is NULL (the opposite side). -## Unpivoting Multiple Columns +## Unpivoting multiple columns Transform multiple numeric columns to name-value pairs: @@ -140,7 +140,7 @@ ORDER BY timestamp, sensor_id, metric; | 10:00:00 | S001 | pressure | 1013.2| | 10:00:00 | S001 | temperature | 22.5 | -## Performance Considerations +## Performance considerations **UNION ALL vs UNION:** ```sql diff --git a/documentation/cookbook/sql/finance/aggressor-volume-imbalance.md b/documentation/cookbook/sql/finance/aggressor-volume-imbalance.md new file mode 100644 index 000000000..05cd8fd7f --- /dev/null +++ b/documentation/cookbook/sql/finance/aggressor-volume-imbalance.md @@ -0,0 +1,38 @@ +--- +title: Aggressor volume imbalance +sidebar_label: Aggressor imbalance +description: Calculate buy vs sell aggressor volume imbalance for order flow analysis +--- + +Calculate the imbalance between buy and sell aggressor volume to analyze order flow. The aggressor is the party that initiated the trade by crossing the spread. + +## Problem: Measure order flow imbalance + +You have trade data with a `side` column indicating the aggressor (buyer or seller), and want to measure the imbalance between buying and selling pressure. + +## Solution: Aggregate by side and calculate ratios + +```questdb-sql demo title="Aggressor volume imbalance per symbol" +WITH volumes AS ( + SELECT + symbol, + SUM(CASE WHEN side = 'buy' THEN amount ELSE 0 END) AS buy_volume, + SUM(CASE WHEN side = 'sell' THEN amount ELSE 0 END) AS sell_volume + FROM trades + WHERE timestamp IN yesterday() + AND symbol IN ('ETH-USDT', 'BTC-USDT', 'ETH-BTC') +) +SELECT + symbol, + buy_volume, + sell_volume, + ((buy_volume - sell_volume)::double / (buy_volume + sell_volume)) * 100 AS imbalance +FROM volumes; +``` + +The imbalance ranges from -100% (all sell) to +100% (all buy), with 0% indicating balanced flow. + +:::info Related documentation +- [CASE expressions](/docs/query/sql/case/) +- [Aggregation functions](/docs/query/functions/aggregation/) +::: diff --git a/documentation/cookbook/sql/finance/bollinger-bands.md b/documentation/cookbook/sql/finance/bollinger-bands.md index af6b71f52..a95fd7601 100644 --- a/documentation/cookbook/sql/finance/bollinger-bands.md +++ b/documentation/cookbook/sql/finance/bollinger-bands.md @@ -1,18 +1,20 @@ --- -title: Bollinger Bands +title: Bollinger bands sidebar_label: Bollinger Bands description: Calculate Bollinger Bands using window functions for volatility analysis and mean reversion trading strategies --- Calculate Bollinger Bands for volatility analysis and mean reversion trading. Bollinger Bands consist of a moving average with upper and lower bands set at a specified number of standard deviations above and below it. They help identify overbought/oversold conditions and measure market volatility. -## Problem: Calculate Rolling Bands with Standard Deviation - -You want to calculate Bollinger Bands with a 20-period simple moving average (SMA) and bands at ±2 standard deviations. The challenge is that QuestDB doesn't support `STDDEV` as a window function, so you need a workaround using the mathematical relationship between variance and standard deviation. +:::note +Bollinger Bands can be calculated using either population standard deviation (stddev) or sample standard deviation (stddev_samp), producing slightly different results. This recipe uses stddev. +::: -## Solution: Calculate Variance Using Window Functions +## Solution: Calculate variance using window functions -Since standard deviation is the square root of variance, and variance is the average of squared differences from the mean, we can calculate it using window functions: +Since standard deviation is the square root of variance, and variance is the average of squared differences from the mean, +we can calculate everything in SQL using window functions. This query will compute Bollinger Bands with a 20-period +simple moving average (SMA) and bands at ±2 standard deviations: ```questdb-sql demo title="Calculate Bollinger Bands with 20-period SMA" WITH OHLC AS ( @@ -32,11 +34,11 @@ WITH OHLC AS ( close, AVG(close) OVER ( ORDER BY timestamp - ROWS BETWEEN 19 PRECEDING AND CURRENT ROW + ROWS 19 PRECEDING ) AS sma20, AVG(close * close) OVER ( ORDER BY timestamp - ROWS BETWEEN 19 PRECEDING AND CURRENT ROW + ROWS 19 PRECEDING ) AS avg_close_sq FROM OHLC ) @@ -58,37 +60,19 @@ This query: 4. Computes standard deviation using the mathematical identity: `σ = √(E[X²] - E[X]²)` 5. Adds/subtracts 2× standard deviation to create upper and lower bands -## How It Works - -The mathematical relationship used here is: - -``` -Variance(X) = E[X²] - (E[X])² -StdDev(X) = √(E[X²] - (E[X])²) -``` - -Where: -- `E[X]` is the average (SMA) of closing prices -- `E[X²]` is the average of squared closing prices -- `√` is the square root function - -Breaking down the calculation: -1. **`AVG(close)`**: Simple moving average over 20 periods -2. **`AVG(close * close)`**: Average of squared prices over 20 periods -3. **`sqrt(avg_close_sq - (sma20 * sma20))`**: Standard deviation derived from variance -4. **Upper/Lower bands**: SMA ± (multiplier × standard deviation) +## How it works -### Window Frame Clause +The core of the Bollinger Bands calculation is the rolling standard deviation. Please check our +[rolling standard deviation recipe](../rolling-stddev/) in the cookbook for an explanation about the mathematical formula. -`ROWS BETWEEN 19 PRECEDING AND CURRENT ROW` creates a sliding window of exactly 20 rows (19 previous + current), which gives us the 20-period moving calculations required for standard Bollinger Bands. -## Adapting the Parameters +## Adapting the parameters **Different period lengths:** ```sql -- 10-period Bollinger Bands (change 19 to 9) -AVG(close) OVER (ORDER BY timestamp ROWS BETWEEN 9 PRECEDING AND CURRENT ROW) AS sma10, -AVG(close * close) OVER (ORDER BY timestamp ROWS BETWEEN 9 PRECEDING AND CURRENT ROW) AS avg_close_sq +AVG(close) OVER (ORDER BY timestamp ROWS 9 PRECEDING) AS sma10, +AVG(close * close) OVER (ORDER BY timestamp ROWS 9 PRECEDING) AS avg_close_sq ``` **Different band multipliers:** @@ -131,12 +115,12 @@ WITH OHLC AS ( AVG(close) OVER ( PARTITION BY symbol ORDER BY timestamp - ROWS BETWEEN 19 PRECEDING AND CURRENT ROW + ROWS 19 PRECEDING ) AS sma20, AVG(close * close) OVER ( PARTITION BY symbol ORDER BY timestamp - ROWS BETWEEN 19 PRECEDING AND CURRENT ROW + ROWS 19 PRECEDING ) AS avg_close_sq FROM OHLC ) @@ -153,20 +137,6 @@ ORDER BY symbol, timestamp; Note the addition of `PARTITION BY symbol` to calculate separate Bollinger Bands for each symbol. -:::tip Trading Signals -- **Bollinger Squeeze**: When bands narrow, it indicates low volatility and often precedes significant price moves -- **Band Walk**: Price consistently touching the upper band suggests strong uptrend; lower band suggests downtrend -- **Mean Reversion**: Price touching or exceeding bands often signals potential reversals back to the mean -- **Volatility Measure**: Width between bands indicates market volatility - wider bands mean higher volatility -::: - -:::tip Parameter Selection -- **Standard settings**: 20-period SMA with 2σ bands (captures ~95% of price action) -- **Day trading**: Use shorter periods (10 or 15) for more responsive bands -- **Swing trading**: Use standard 20-period or longer (50-period) for smoother signals -- **Volatility adjustment**: Use 2.5σ or 3σ bands in highly volatile markets -::: - :::info Related Documentation - [Window functions](/docs/query/functions/window-functions/syntax/) - [AVG window function](/docs/query/functions/window-functions/reference/#avg) diff --git a/documentation/cookbook/sql/finance/compound-interest.md b/documentation/cookbook/sql/finance/compound-interest.md index 7e8d9634a..b6a826f07 100644 --- a/documentation/cookbook/sql/finance/compound-interest.md +++ b/documentation/cookbook/sql/finance/compound-interest.md @@ -1,5 +1,5 @@ --- -title: Calculate Compound Interest +title: Calculate compound interest sidebar_label: Compound interest description: Calculate compound interest over time using POWER and window functions --- @@ -10,13 +10,13 @@ Calculate compound interest over multiple periods using SQL, where each period's This query uses generated data from `long_sequence()` to create a time series of years, so it can run directly on the demo instance without requiring any existing tables. ::: -## Problem: Need Year-by-Year Growth +## Problem: Need year-by-year growth You want to calculate compound interest over 5 years, starting with an initial principal of 1000, with an annual interest rate of 0.1 (10%). Each year's interest should be calculated on the previous year's ending balance. -## Solution: Use POWER Function with Window Functions +## Solution: Use POWER function with window functions -Combine the `POWER()` function with `FIRST_VALUE()` window function to calculate compound interest: +The compound interest formula is `principal * (1 + rate)^periods`. Use `POWER()` to calculate the exponential part: ```questdb-sql demo title="Calculate compound interest over 5 years" WITH @@ -45,11 +45,7 @@ SELECT timestamp, initial_principal, interest_rate, - FIRST_VALUE(cv.compounding) - OVER ( - ORDER BY timestamp - ROWS between 1 preceding and 1 preceding - ) AS year_principal, + LAG(cv.compounding) OVER (ORDER BY timestamp) AS year_principal, cv.compounding as compounding_amount FROM compounded_values cv @@ -74,17 +70,15 @@ from compounding_year_before; Each row shows how the principal grows year over year, with interest compounding on the previous year's ending balance. -## How It Works +## How it works The query uses a multi-step CTE approach: 1. **Generate year series**: Use `long_sequence(5)` to create 5 rows representing years 2000-2004 2. **Calculate compound amount**: Use `POWER(1 + interest_rate, years)` to compute the ending balance for each year -3. **Get previous year's balance**: Use `FIRST_VALUE()` with window frame `ROWS between 1 preceding and 1 preceding` to access the previous row's compounding amount +3. **Get previous year's balance**: Use `LAG()` to access the previous row's compounding amount 4. **Handle first year**: Use `COALESCE()` to show the initial principal for the first year -The `POWER()` function calculates the compound interest formula: `principal * (1 + rate)^periods` - :::tip For more complex scenarios like monthly or quarterly compounding, adjust the time period generation and the exponent in the POWER function accordingly. @@ -93,6 +87,6 @@ For more complex scenarios like monthly or quarterly compounding, adjust the tim :::info Related Documentation - [POWER function](/docs/query/functions/numeric/#power) - [Window functions](/docs/query/functions/window-functions/syntax/) -- [FIRST_VALUE window function](/docs/query/functions/window-functions/reference/#first_value) +- [LAG window function](/docs/query/functions/window-functions/reference/#lag) - [long_sequence](/docs/query/functions/row-generator/#long_sequence) ::: diff --git a/documentation/cookbook/sql/finance/cumulative-product.md b/documentation/cookbook/sql/finance/cumulative-product.md index 37be84e6c..2ce9af558 100644 --- a/documentation/cookbook/sql/finance/cumulative-product.md +++ b/documentation/cookbook/sql/finance/cumulative-product.md @@ -1,12 +1,12 @@ --- -title: Cumulative Product for Random Walk +title: Cumulative product for random walk sidebar_label: Cumulative product description: Calculate cumulative product to simulate stock price paths from daily returns --- Calculate the cumulative product of daily returns to simulate a stock's price path (random walk). This is useful for financial modeling, backtesting trading strategies, and portfolio analysis where you need to compound returns over time. -## Problem: Compound Daily Returns +## Problem: Compound daily returns You have a table with daily returns for a stock and want to calculate the cumulative price starting from an initial value (e.g., $100). Each day's price is calculated by multiplying the previous price by `(1 + return)`. @@ -28,9 +28,11 @@ You want to calculate: | 2024-09-07 | 1.50 | 102.49 | | 2024-09-08 | -3.00 | 99.42 | -## Solution: Use Logarithm Mathematics Trick +## Solution: Use logarithms to convert multiplication to addition -Since QuestDB doesn't allow functions on top of window function results, we use a mathematical trick: **the exponential of the sum of logarithms equals the product**. +Use the mathematical identity: **exp(sum(ln(x))) = product(x)** + +This converts the cumulative product into a cumulative sum, which window functions handle naturally: ```questdb-sql title="Calculate cumulative product via logarithms" WITH ln_values AS ( @@ -52,7 +54,7 @@ This query: 2. Uses a cumulative `SUM` window function to add up the logarithms 3. Applies `exp()` to convert back to the product -## How It Works +## How it works The mathematical identity used here is: @@ -66,13 +68,7 @@ Breaking it down: - `exp(ln_value)` converts the cumulative sum back to a cumulative product - Multiply by 100 to apply the starting price of $100 -### Why This Works - -QuestDB doesn't support direct window functions like `PRODUCT() OVER()`, and attempting `exp(SUM(ln(1 + return)) OVER ())` fails with a "dangling literal" error because you can't nest functions around window functions. - -The workaround is to use a CTE to compute the cumulative sum first, then apply `exp()` in the outer query where it's operating on a regular column, not a window function result. - -## Adapting to Your Data +## Adapting to your data You can easily modify this pattern: diff --git a/documentation/cookbook/sql/finance/rolling-stddev.md b/documentation/cookbook/sql/finance/rolling-stddev.md index 14b0c8ea0..1248f01e5 100644 --- a/documentation/cookbook/sql/finance/rolling-stddev.md +++ b/documentation/cookbook/sql/finance/rolling-stddev.md @@ -1,73 +1,60 @@ --- -title: Rolling Standard Deviation +title: Rolling standard deviation sidebar_label: Rolling std dev -description: Calculate rolling standard deviation using window functions and CTEs +description: Calculate rolling standard deviation using window functions --- Calculate rolling standard deviation to measure price volatility over time. ## Problem -You want to calculate the standard deviation in a time window. QuestDB supports stddev as an aggregate function, but not as a window function. +You want to calculate rolling standard deviation. ## Solution -The standard deviation can be calculated from the variance, which is the average of the square differences from the mean. +Use the mathematical identity: `σ = √(E[X²] - E[X]²)` -In general we could write it in SQL like this: - -```sql -SELECT - symbol, - price, - AVG(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS rolling_mean, - SQRT(AVG(POWER(price - AVG(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), 2)) - OVER (PARTITION BY symbol ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)) AS rolling_stddev -FROM - fx_trades -WHERE timestamp IN yesterday() -``` - -But in QuestDB we cannot do any operations on the return value of a window function, so we need to do this using CTEs: +Compute both `AVG(price)` and `AVG(price * price)` as window functions, then derive the standard deviation: ```questdb-sql demo title="Calculate rolling standard deviation" -WITH rolling_avg_cte AS ( - SELECT - timestamp, - symbol, - price, - AVG(price) OVER (PARTITION BY symbol ORDER BY timestamp) AS rolling_avg - FROM - fx_trades - WHERE - timestamp IN yesterday() AND symbol = 'EURUSD' -), -variance_cte AS ( +WITH stats AS ( SELECT timestamp, symbol, price, - rolling_avg, - AVG(POWER(price - rolling_avg, 2)) OVER (PARTITION BY symbol ORDER BY timestamp) AS rolling_variance - FROM - rolling_avg_cte + AVG(price) OVER (PARTITION BY symbol ORDER BY timestamp) AS rolling_avg, + AVG(price * price) OVER (PARTITION BY symbol ORDER BY timestamp) AS rolling_avg_sq + FROM fx_trades + WHERE timestamp IN yesterday() AND symbol = 'EURUSD' ) SELECT timestamp, symbol, price, rolling_avg, - rolling_variance, - SQRT(rolling_variance) AS rolling_stddev -FROM - variance_cte; + SQRT(rolling_avg_sq - rolling_avg * rolling_avg) AS rolling_stddev +FROM stats +LIMIT 10; +``` + +## How it works + +The mathematical relationship used here is: + ``` +Variance(X) = E[X²] - (E[X])² +StdDev(X) = √(E[X²] - (E[X])²) +``` + +Where: +- `E[X]` is the average (SMA) of prices +- `E[X²]` is the average of squared prices +- `√` is the square root function -I first get the rolling average/mean, then from that I get the variance, and then I can do the `sqrt` to get the standard deviation as requested. +This query calculates an expanding standard deviation from the beginning of the period to the current row. For a fixed rolling window, add a [frame clause](/docs/query/functions/window-functions/syntax/#frame-types-and-behavior) to both window functions using `ROWS` (fixed number of rows) or `RANGE` (time-based window). -:::info Related Documentation +:::info Related documentation - [Window functions](/docs/query/functions/window-functions/syntax/) - [AVG window function](/docs/query/functions/window-functions/reference/#avg) -- [POWER function](/docs/query/functions/numeric/#power) - [SQRT function](/docs/query/functions/numeric/#sqrt) ::: diff --git a/documentation/cookbook/sql/finance/tick-trin.md b/documentation/cookbook/sql/finance/tick-trin.md index cd7090145..437ccb54d 100644 --- a/documentation/cookbook/sql/finance/tick-trin.md +++ b/documentation/cookbook/sql/finance/tick-trin.md @@ -1,185 +1,138 @@ --- -title: Cumulative Tick and Trin Indicators -sidebar_label: Tick & Trin -description: Calculate cumulative Tick and Trin (ARMS Index) for market sentiment analysis and breadth indicators +title: TICK and TRIN indicators +sidebar_label: TICK & TRIN +description: Calculate TICK and TRIN (ARMS Index) for market breadth analysis --- -Calculate cumulative Tick and Trin (also known as the ARMS Index) to measure market sentiment and breadth. These indicators compare advancing versus declining trades in terms of both count and volume, helping identify overbought/oversold conditions and potential market reversals. - -## Problem: Calculate Running Market Breadth - -You have a table with trade data including `side` (buy/sell) and `quantity`, and want to calculate cumulative Tick and Trin values throughout the trading day. Tick measures the ratio of upticks to downticks, while Trin (Trading Index) adjusts this ratio by volume to identify divergences between price action and volume. - -## Solution: Use Window Functions with CASE Statements - -Use `SUM` as a window function combined with `CASE` statements to compute running totals of upticks, downticks, and their respective volumes: - -```questdb-sql demo title="Calculate cumulative Tick and Trin indicators" -WITH tick_vol AS ( - SELECT - timestamp, - side, - quantity, - SUM(CASE WHEN side = 'sell' THEN 1.0 END) OVER (ORDER BY timestamp) as downtick, - SUM(CASE WHEN side = 'buy' THEN 1.0 END) OVER (ORDER BY timestamp) as uptick, - SUM(CASE WHEN side = 'sell' THEN quantity END) OVER (ORDER BY timestamp) as downvol, - SUM(CASE WHEN side = 'buy' THEN quantity END) OVER (ORDER BY timestamp) as upvol - FROM fx_trades - WHERE timestamp IN yesterday() AND symbol = 'EURUSD' +Calculate TICK and TRIN (Trading Index, also known as the ARMS Index) to measure market breadth. These indicators count how many symbols are advancing versus declining across a market. + +## TICK + +**TICK** measures market direction by counting symbols: +- For each symbol, check if its last trade price > previous trade price (uptick) or < (downtick) +- TICK = number of symbols on uptick - number of symbols on downtick +- Positive TICK = more symbols rising, negative = more falling + +### TICK snapshot + +Calculate a single market-wide TICK value. For each symbol, compare the last trade price to the previous trade price to determine if it's on an uptick or downtick. The final result is one row showing how many symbols are rising versus falling. + +```questdb-sql demo title="TICK - market breadth snapshot" +WITH with_previous AS ( + SELECT timestamp, symbol, price, + LAG(price) OVER (PARTITION BY symbol ORDER BY timestamp) AS prev_price + FROM fx_trades + WHERE timestamp IN today() +), +classified AS ( + SELECT symbol, + CASE WHEN price > prev_price THEN 1 ELSE 0 END AS is_uptick, + CASE WHEN price < prev_price THEN 1 ELSE 0 END AS is_downtick + FROM with_previous + WHERE prev_price IS NOT NULL + LATEST ON timestamp PARTITION BY symbol -- use only the latest entry per symbol, together with the previous price ) SELECT - timestamp, - side, - quantity, - uptick, - downtick, - upvol, - downvol, - uptick / downtick as tick, - (uptick / downtick) / (upvol / downvol) as trin -FROM tick_vol -LIMIT -8; -``` - -**Results:** - -| timestamp | side | quantity | uptick | downtick | upvol | downvol | tick | trin | -| ------------------------------ | ---- | -------- | -------- | -------- | ------------- | ------------- | ------------------ | ------------------ | -| 2026-01-11T23:59:58.997072039Z | sell | 98659.0 | 342395.0 | 343426.0 | 45996256659.0 | 46085483999.0 | 0.9969978976548077 | 0.9989319565729681 | -| 2026-01-11T23:59:59.084976043Z | buy | 99311.0 | 342396.0 | 343426.0 | 45996355970.0 | 46085483999.0 | 0.997000809490254 | 0.9989327172509304 | -| 2026-01-11T23:59:59.085326995Z | buy | 57591.0 | 342397.0 | 343426.0 | 45996413561.0 | 46085483999.0 | 0.9970037213257005 | 0.9989343839854824 | -| 2026-01-11T23:59:59.085700555Z | buy | 119667.0 | 342398.0 | 343426.0 | 45996533228.0 | 46085483999.0 | 0.9970066331611468 | 0.9989347025717739 | -| 2026-01-11T23:59:59.642850139Z | sell | 57695.0 | 342398.0 | 343427.0 | 45996533228.0 | 46085541694.0 | 0.9970037300503455 | 0.9989330444221086 | -| 2026-01-11T23:59:59.643380840Z | sell | 130834.0 | 342398.0 | 343428.0 | 45996533228.0 | 46085672528.0 | 0.9970008269564509 | 0.9989329716112184 | -| 2026-01-11T23:59:59.643482764Z | sell | 119573.0 | 342398.0 | 343429.0 | 45996533228.0 | 46085792101.0 | 0.9969979238794627 | 0.9989326547129301 | -| 2026-01-11T23:59:59.643517597Z | sell | 33928.0 | 342398.0 | 343430.0 | 45996533228.0 | 46085826029.0 | 0.996995020819381 | 0.9989304814235689 | - -:::warning Handling NULL Values -The first rows will have NULL values for tick and trin until there's at least one trade on each side (buy and sell). You can filter these out with `WHERE uptick IS NOT NULL AND downtick IS NOT NULL` if needed. -::: - -Each row shows the cumulative values from the start of the day, with Tick and Trin calculated at every trade. - -## How It Works - -The indicators are calculated using these formulas: - -``` -Tick = Upticks / Downticks - -Trin = (Upticks / Downticks) / (Upvol / Downvol) - = Tick / Volume Ratio + SUM(is_uptick) AS uptick_symbols, + SUM(is_downtick) AS downtick_symbols, + SUM(is_uptick) - SUM(is_downtick) AS tick +FROM classified; ``` -Where: -- **Upticks**: Cumulative count of buy transactions -- **Downticks**: Cumulative count of sell transactions -- **Upvol**: Cumulative volume of buy transactions -- **Downvol**: Cumulative volume of sell transactions - -The query uses: -1. **Window functions**: `SUM(...) OVER (ORDER BY timestamp)` creates running totals from the start of the period -2. **CASE statements**: Conditionally sum only trades matching the specified side -3. **Type casting**: Using `1.0` instead of `1` ensures results are doubles, avoiding explicit casting - -### Interpreting the Indicators - -**Tick Indicator:** -- **Tick > 1.0**: More buying pressure (bullish sentiment) -- **Tick < 1.0**: More selling pressure (bearish sentiment) -- **Tick = 1.0**: Neutral market (equal buying and selling) - -**Trin (ARMS Index):** -- **Trin < 1.0**: Strong market (volume flowing into advancing trades) -- **Trin > 1.0**: Weak market (volume flowing into declining trades) -- **Trin = 1.0**: Balanced market -- **Extreme readings**: Trin > 2.0 suggests oversold conditions; Trin < 0.5 suggests overbought - -**Divergences:** -When Tick and Trin move in opposite directions, it can signal important market conditions: -- High Tick + High Trin: Advances lack volume confirmation (bearish divergence) -- Low Tick + Low Trin: Declines lack volume confirmation (bullish divergence) - -## Adapting the Query - -**Multiple symbols:** -```questdb-sql demo title="Tick and Trin for multiple symbols" -WITH tick_vol AS ( - SELECT - timestamp, - symbol, - side, - quantity, - SUM(CASE WHEN side = 'sell' THEN 1.0 END) - OVER (PARTITION BY symbol ORDER BY timestamp) as downtick, - SUM(CASE WHEN side = 'buy' THEN 1.0 END) - OVER (PARTITION BY symbol ORDER BY timestamp) as uptick, - SUM(CASE WHEN side = 'sell' THEN quantity END) - OVER (PARTITION BY symbol ORDER BY timestamp) as downvol, - SUM(CASE WHEN side = 'buy' THEN quantity END) - OVER (PARTITION BY symbol ORDER BY timestamp) as upvol - FROM fx_trades - WHERE timestamp IN yesterday() +### Interpreting TICK + +- **Positive**: More symbols on uptick (bullish) +- **Negative**: More symbols on downtick (bearish) +- **Near zero**: Balanced market + +## TRIN + +**TRIN** (ARMS Index) adds volume weighting: +- TRIN = (advancing symbols / declining symbols) / (advancing volume / declining volume) +- TRIN < 1.0 = volume favoring advances (bullish) +- TRIN > 1.0 = volume favoring declines (bearish) +- TRIN = 1.0 = neutral + +### TRIN snapshot + +Calculate a single market-wide TRIN value. For each symbol, aggregate intraday volume and classify it as advancing or declining based on whether the current price is above or below the day's open. The final result is one row showing overall market breadth. + +```questdb-sql demo title="TRIN - daily breadth snapshot" +WITH daily_stats AS ( + SELECT symbol, + first(price) AS open_price, + last(price) AS current_price, + sum(quantity) AS total_volume + FROM fx_trades + WHERE timestamp IN today() + SAMPLE BY 1d +), +classified AS ( + SELECT *, + CASE WHEN current_price > open_price THEN 1 ELSE 0 END AS is_advancing, + CASE WHEN current_price < open_price THEN 1 ELSE 0 END AS is_declining + FROM daily_stats ) SELECT - timestamp, - symbol, - uptick / downtick as tick, - (uptick / downtick) / (upvol / downvol) as trin -FROM tick_vol; + SUM(is_advancing) AS advancing, + SUM(is_declining) AS declining, + SUM(CASE WHEN is_advancing = 1 THEN total_volume ELSE 0 END) AS advancing_volume, + SUM(CASE WHEN is_declining = 1 THEN total_volume ELSE 0 END) AS declining_volume, + (SUM(is_advancing)::double / NULLIF(SUM(is_declining), 0)) / + (SUM(CASE WHEN is_advancing = 1 THEN total_volume ELSE 0 END)::double / + NULLIF(SUM(CASE WHEN is_declining = 1 THEN total_volume ELSE 0 END), 0)) AS trin +FROM classified; ``` -**Intraday periods (reset at intervals):** -```questdb-sql demo title="Tick and Trin reset every hour" -WITH tick_vol AS ( - SELECT - timestamp, - side, - quantity, - SUM(CASE WHEN side = 'sell' THEN 1.0 END) - OVER (PARTITION BY timestamp_floor('h', timestamp) ORDER BY timestamp) as downtick, - SUM(CASE WHEN side = 'buy' THEN 1.0 END) - OVER (PARTITION BY timestamp_floor('h', timestamp) ORDER BY timestamp) as uptick, - SUM(CASE WHEN side = 'sell' THEN quantity END) - OVER (PARTITION BY timestamp_floor('h', timestamp) ORDER BY timestamp) as downvol, - SUM(CASE WHEN side = 'buy' THEN quantity END) - OVER (PARTITION BY timestamp_floor('h', timestamp) ORDER BY timestamp) as upvol - FROM fx_trades - WHERE timestamp IN yesterday() AND symbol = 'EURUSD' +### TRIN time-series + +Track how market breadth evolves throughout the day. For each candle interval, compare each symbol's close to its previous close to classify it as advancing or declining. Each row returns the market-wide TRIN at that point in time. + +```questdb-sql demo title="TRIN time-series with 5-minute candles" +WITH candles AS ( + SELECT timestamp, symbol, + last(price) AS close_price, + sum(quantity) AS total_volume + FROM fx_trades + WHERE timestamp IN today() + SAMPLE BY 5m +), +with_previous AS ( + SELECT timestamp, symbol, total_volume, close_price, + LAG(close_price) OVER (PARTITION BY symbol ORDER BY timestamp) AS last_close + FROM candles +), +classified AS ( + SELECT timestamp, symbol, total_volume, + CASE WHEN close_price > last_close THEN 1 ELSE 0 END AS is_advancing, + CASE WHEN close_price < last_close THEN 1 ELSE 0 END AS is_declining + FROM with_previous + WHERE last_close IS NOT NULL ) SELECT - timestamp, - uptick / downtick as tick, - (uptick / downtick) / (upvol / downvol) as trin -FROM tick_vol; + timestamp, + SUM(is_advancing) AS advancing, + SUM(is_declining) AS declining, + SUM(CASE WHEN is_advancing = 1 THEN total_volume ELSE 0 END) AS advancing_volume, + SUM(CASE WHEN is_declining = 1 THEN total_volume ELSE 0 END) AS declining_volume, + (SUM(is_advancing)::double / NULLIF(SUM(is_declining), 0)) / + (SUM(CASE WHEN is_advancing = 1 THEN total_volume ELSE 0 END)::double / + NULLIF(SUM(CASE WHEN is_declining = 1 THEN total_volume ELSE 0 END), 0)) AS trin +FROM classified; ``` -**Daily summary values only:** -```questdb-sql demo title="Tick and Trin daily summary" -WITH tick_vol AS ( - SELECT - SUM(CASE WHEN side = 'sell' THEN 1.0 END) as downtick, - SUM(CASE WHEN side = 'buy' THEN 1.0 END) as uptick, - SUM(CASE WHEN side = 'sell' THEN quantity END) as downvol, - SUM(CASE WHEN side = 'buy' THEN quantity END) as upvol - FROM fx_trades - WHERE timestamp IN yesterday() AND symbol = 'EURUSD' -) -SELECT - uptick / downtick as tick, - (uptick / downtick) / (upvol / downvol) as trin -FROM tick_vol; -``` +### Interpreting TRIN + +- **< 1.0**: Volume favoring advances (bullish) +- **> 1.0**: Volume favoring declines (bearish) +- **= 1.0**: Neutral -:::tip Market Analysis Applications -- **Intraday momentum**: Track Tick throughout the day to identify accumulation/distribution patterns -- **Overbought/oversold**: Extreme Trin readings often precede short-term reversals -- **Market breadth**: Persistently high/low values indicate broad market strength or weakness -- **Divergence trading**: When price makes new highs/lows but Trin doesn't confirm, it suggests weakening momentum +:::note +TRIN can produce counterintuitive results. If advances outnumber declines 2:1 and advancing volume also leads 2:1, TRIN equals 1.0 (neutral) despite bullish conditions. Always consider TRIN alongside the raw advancing/declining counts. ::: -:::info Related Documentation +:::info Related documentation - [Window functions](/docs/query/functions/window-functions/syntax/) -- [SUM aggregate](/docs/query/functions/aggregation/#sum) -- [CASE expressions](/docs/query/sql/case/) +- [LAG function](/docs/query/functions/window-functions/reference/#lag) +- [SAMPLE BY](/docs/query/sql/sample-by/) ::: diff --git a/documentation/cookbook/sql/finance/volume-profile.md b/documentation/cookbook/sql/finance/volume-profile.md index 2a44b0c70..5ce2d184d 100644 --- a/documentation/cookbook/sql/finance/volume-profile.md +++ b/documentation/cookbook/sql/finance/volume-profile.md @@ -1,5 +1,5 @@ --- -title: Volume Profile +title: Volume profile sidebar_label: Volume profile description: Calculate volume profile by grouping trades into price bins --- @@ -23,7 +23,7 @@ ORDER BY price_bin; Since QuestDB does an implicit GROUP BY on all non-aggregated columns, you can omit the explicit GROUP BY clause. -## Dynamic Tick Size +## Dynamic tick size For consistent histograms across different price ranges, calculate the tick size dynamically to always produce approximately 50 bins: diff --git a/documentation/cookbook/sql/finance/volume-spike.md b/documentation/cookbook/sql/finance/volume-spike.md index 383d4a9aa..d1f860c1d 100644 --- a/documentation/cookbook/sql/finance/volume-spike.md +++ b/documentation/cookbook/sql/finance/volume-spike.md @@ -1,5 +1,5 @@ --- -title: Volume Spike Detection +title: Volume spike detection sidebar_label: Volume spikes description: Detect volume spikes by comparing current volume against previous volume using LAG --- diff --git a/documentation/cookbook/sql/finance/vwap.md b/documentation/cookbook/sql/finance/vwap.md index ecb1c7fa3..b92031149 100644 --- a/documentation/cookbook/sql/finance/vwap.md +++ b/documentation/cookbook/sql/finance/vwap.md @@ -1,121 +1,91 @@ --- -title: Volume Weighted Average Price (VWAP) +title: Volume weighted average price (VWAP) sidebar_label: VWAP description: Calculate cumulative volume weighted average price using window functions for intraday trading analysis --- Calculate the cumulative Volume Weighted Average Price (VWAP) for intraday trading analysis. VWAP is a trading benchmark that represents the average price at which an asset has traded throughout the day, weighted by volume. It's widely used by institutional traders to assess execution quality and identify trend strength. -## Problem: Calculate Running VWAP +## Problem: Calculate running VWAP You want to calculate the cumulative VWAP for a trading day, where each point shows the average price weighted by volume from market open until that moment. This helps traders determine if current prices are above or below the day's volume-weighted average. -## Solution: Use Window Functions for Cumulative Sums +## Solution: Use typical price from OHLC data -While QuestDB doesn't have a built-in VWAP window function, we can calculate it using cumulative `SUM` window functions for both traded value and volume: +The industry standard for VWAP uses the **typical price** formula from OHLC (Open, High, Low, Close) candles: -```questdb-sql demo title="Calculate cumulative VWAP over 10-minute intervals" +``` +Typical Price = (High + Low + Close) / 3 +VWAP = Σ(Typical Price × Volume) / Σ(Volume) +``` + +This approximation is used because most trading platforms work with OHLC data rather than tick-level trades. We use the `fx_trades_ohlc_1m` materialized view which provides 1-minute candles: + +```questdb-sql demo title="Calculate cumulative VWAP" WITH sampled AS ( - SELECT - timestamp, symbol, - SUM(quantity) AS volume, - SUM(price * quantity) AS traded_value - FROM fx_trades - WHERE timestamp IN yesterday() - AND symbol = 'EURUSD' - SAMPLE BY 10m -), cumulative AS ( - SELECT timestamp, symbol, - SUM(traded_value) - OVER (ORDER BY timestamp) AS cumulative_value, - SUM(volume) - OVER (ORDER BY timestamp) AS cumulative_volume - FROM sampled + SELECT + timestamp, symbol, + total_volume, + ((high + low + close) / 3) * total_volume AS traded_value + FROM fx_trades_ohlc_1m + WHERE timestamp IN yesterday() AND symbol = 'EURUSD' +), +cumulative AS ( + SELECT + timestamp, symbol, + SUM(traded_value) OVER (ORDER BY timestamp) AS cumulative_value, + SUM(total_volume) OVER (ORDER BY timestamp) AS cumulative_volume + FROM sampled ) -SELECT timestamp, symbol, - cumulative_value/cumulative_volume AS vwap - FROM cumulative; +SELECT timestamp, symbol, cumulative_value / cumulative_volume AS vwap +FROM cumulative; ``` This query: -1. Aggregates trades into 10-minute intervals, calculating total volume and total traded value (price × amount) for each interval -2. Uses window functions to compute running totals of both traded value and volume from the start of the day +1. Reads 1-minute OHLC candles and calculates typical price × volume for each candle +2. Uses window functions to compute running totals of both traded value and volume 3. Divides cumulative traded value by cumulative volume to get VWAP at each timestamp -## How It Works - -VWAP is calculated as: - -``` -VWAP = Total Traded Value / Total Volume - = Σ(Price × Volume) / Σ(Volume) -``` +## How it works The key insight is using `SUM(...) OVER (ORDER BY timestamp)` to create running totals: -- `cumulative_value`: Running sum of (price × amount) from market open +- `cumulative_value`: Running sum of (typical price × volume) from market open - `cumulative_volume`: Running sum of volume from market open -- Final VWAP: Dividing these cumulative values gives the volume-weighted average at each point in time - -### Window Function Behavior +- Final VWAP: Dividing these cumulative values gives the volume-weighted average at each point When using `SUM() OVER (ORDER BY timestamp)` without specifying a frame clause, QuestDB defaults to summing from the first row to the current row, which is exactly what we need for cumulative VWAP. -## Adapting the Query +## Multiple symbols -**Different time intervals:** -```questdb-sql demo title="VWAP with 1-minute resolution" -WITH sampled AS ( - SELECT - timestamp, symbol, - SUM(quantity) AS volume, - SUM(price * quantity) AS traded_value - FROM fx_trades - WHERE timestamp IN yesterday() - AND symbol = 'EURUSD' - SAMPLE BY 1m -- Changed from 10m to 1m -), cumulative AS ( - SELECT timestamp, symbol, - SUM(traded_value) - OVER (ORDER BY timestamp) AS cumulative_value, - SUM(volume) - OVER (ORDER BY timestamp) AS cumulative_volume - FROM sampled -) -SELECT timestamp, symbol, - cumulative_value/cumulative_volume AS vwap - FROM cumulative; -``` +To calculate VWAP for multiple symbols simultaneously, add `PARTITION BY symbol` to the window functions: -**Multiple symbols:** ```questdb-sql demo title="VWAP for multiple symbols" - WITH sampled AS ( - SELECT - timestamp, symbol, - SUM(quantity) AS volume, - SUM(price * quantity) AS traded_value - FROM fx_trades - WHERE timestamp IN yesterday() - AND symbol IN ('EURUSD', 'GBPUSD', 'JPYUSD') - SAMPLE BY 10m -), cumulative AS ( - SELECT timestamp, symbol, - SUM(traded_value) - OVER (ORDER BY timestamp) AS cumulative_value, - SUM(volume) - OVER (ORDER BY timestamp) AS cumulative_volume - FROM sampled + SELECT + timestamp, symbol, + total_volume, + ((high + low + close) / 3) * total_volume AS traded_value + FROM fx_trades_ohlc_1m + WHERE timestamp IN yesterday() + AND symbol IN ('EURUSD', 'GBPUSD', 'USDJPY') +), +cumulative AS ( + SELECT + timestamp, symbol, + SUM(traded_value) OVER (PARTITION BY symbol ORDER BY timestamp) AS cumulative_value, + SUM(total_volume) OVER (PARTITION BY symbol ORDER BY timestamp) AS cumulative_volume + FROM sampled ) -SELECT timestamp, symbol, - cumulative_value/cumulative_volume AS vwap - FROM cumulative; +SELECT timestamp, symbol, cumulative_value / cumulative_volume AS vwap +FROM cumulative; ``` -Note the addition of `PARTITION BY symbol` to calculate separate VWAP values for each symbol. +The `PARTITION BY symbol` ensures each symbol's VWAP is calculated independently, resetting the cumulative sums for each symbol. + +## Different time ranges -**Different time ranges:** ```sql --- Current trading day (today) +-- Current trading day WHERE timestamp IN today() -- Specific date @@ -125,15 +95,15 @@ WHERE timestamp IN '2026-01-12' WHERE timestamp >= dateadd('h', -1, now()) ``` -:::tip Trading Use Cases +:::tip Trading use cases - **Execution quality**: Institutional traders compare their execution prices against VWAP to assess trade quality - **Trend identification**: Price consistently above VWAP suggests bullish momentum; below suggests bearish - **Support/resistance**: VWAP often acts as dynamic support or resistance during the trading day - **Mean reversion**: Traders use deviations from VWAP to identify potential reversal points ::: -:::info Related Documentation +:::info Related documentation - [Window functions](/docs/query/functions/window-functions/syntax/) - [SUM aggregate](/docs/query/functions/aggregation/#sum) -- [SAMPLE BY](/docs/query/sql/sample-by/) +- [Materialized views](/docs/concepts/materialized-views/) ::: diff --git a/documentation/cookbook/sql/time-series/distribute-discrete-values.md b/documentation/cookbook/sql/time-series/distribute-discrete-values.md index 382a49119..424b3944c 100644 --- a/documentation/cookbook/sql/time-series/distribute-discrete-values.md +++ b/documentation/cookbook/sql/time-series/distribute-discrete-values.md @@ -1,5 +1,5 @@ --- -title: Distribute Discrete Values Across Time Intervals +title: Distribute discrete values across time intervals sidebar_label: Distribute discrete values description: Spread cumulative measurements across time intervals using sessions and window functions --- diff --git a/documentation/cookbook/sql/time-series/epoch-timestamps.md b/documentation/cookbook/sql/time-series/epoch-timestamps.md index 2c0dc417a..175f91104 100644 --- a/documentation/cookbook/sql/time-series/epoch-timestamps.md +++ b/documentation/cookbook/sql/time-series/epoch-timestamps.md @@ -1,5 +1,5 @@ --- -title: Query with Epoch Timestamps +title: Query with epoch timestamps sidebar_label: Epoch timestamps description: Use epoch timestamps for timestamp filtering in QuestDB --- diff --git a/documentation/cookbook/sql/time-series/fill-from-one-column.md b/documentation/cookbook/sql/time-series/fill-from-one-column.md index 8ca6baa59..7d4997bdf 100644 --- a/documentation/cookbook/sql/time-series/fill-from-one-column.md +++ b/documentation/cookbook/sql/time-series/fill-from-one-column.md @@ -1,5 +1,5 @@ --- -title: Fill Missing Intervals with Value from Another Column +title: Fill missing intervals with value from another column sidebar_label: Fill from one column description: Use window functions to propagate values from one column to fill multiple columns in SAMPLE BY queries --- diff --git a/documentation/cookbook/sql/time-series/fill-keyed-arbitrary-interval.md b/documentation/cookbook/sql/time-series/fill-keyed-arbitrary-interval.md index 0a029bfe8..ec2604a2b 100644 --- a/documentation/cookbook/sql/time-series/fill-keyed-arbitrary-interval.md +++ b/documentation/cookbook/sql/time-series/fill-keyed-arbitrary-interval.md @@ -1,20 +1,25 @@ --- -title: FILL on Keyed Queries with Arbitrary Intervals -sidebar_label: FILL keyed arbitrary interval -description: Use FILL with keyed queries across arbitrary time intervals by sandwiching data with null boundary rows +title: FILL on keyed queries with arbitrary intervals +sidebar_label: FILL arbitrary intervals +description: Use FILL with keyed queries and any FILL strategy across arbitrary time intervals by sandwiching data with null boundary rows --- -When using `SAMPLE BY` with `FILL` on keyed queries (queries with non-aggregated columns like symbol), the `FROM/TO` syntax doesn't work. This recipe shows how to fill gaps across an arbitrary time interval for keyed queries. +You want to sample data and fill any potential gaps with interpolated values, using a time interval defined by a starting +and ending timestamp, not only between the first and last existing row in the filtered results. + ## Problem -Keyed queries - queries that include non-aggregated columns beyond the timestamp - do not support the `SAMPLE BY FROM x TO y` syntax when using `FILL`. Without this feature, gaps are only filled between the first and last existing row in the filtered results, not across your desired time interval. +QuestDB has a built-in [`SAMPLE BY .. FROM/TO`](/docs/query/sql/sample-by/#from-to) syntax available for non-keyed queries (queries that include only aggregated columns beyond the timestamp), and for the `NULL` FILL strategy. + +If you use `FROM/TO` in a keyed query (for example, an OHLC with timestamp, symbol, and aggregations) you will get the +following error: _FROM-TO intervals are not supported for keyed SAMPLE BY queries_. -For example, if you want to sample by symbol and timestamp bucket with `FILL` for a specific time range, standard approaches will not fill gaps at the beginning or end of your interval. ## Solution -"Sandwich" your data by adding artificial boundary rows at the start and end of your time interval using `UNION ALL`. These rows contain your target timestamps with nulls for all other columns: +"Sandwich" your data by adding artificial boundary rows at the start and end of your time interval using `UNION ALL`. These rows contain your target timestamps with nulls for all other columns. Then you can use `FILL` without the `FROM/TO` keywords and get results +for every sampled interval within those arbitrary dates. ```questdb-sql demo title="FILL arbitrary interval with keyed SAMPLE BY" diff --git a/documentation/cookbook/sql/time-series/fill-prev-with-history.md b/documentation/cookbook/sql/time-series/fill-prev-with-history.md index c31c28aa5..12ea8793a 100644 --- a/documentation/cookbook/sql/time-series/fill-prev-with-history.md +++ b/documentation/cookbook/sql/time-series/fill-prev-with-history.md @@ -1,5 +1,5 @@ --- -title: FILL PREV with Historical Values +title: FILL PREV with historical values sidebar_label: FILL PREV with history description: Use FILL(PREV) with a filler row to carry historical values into a filtered time interval --- diff --git a/documentation/cookbook/sql/time-series/filter-by-week.md b/documentation/cookbook/sql/time-series/filter-by-week.md index 64242b815..644ba3ced 100644 --- a/documentation/cookbook/sql/time-series/filter-by-week.md +++ b/documentation/cookbook/sql/time-series/filter-by-week.md @@ -1,5 +1,5 @@ --- -title: Filter Data by Week Number +title: Filter data by week number sidebar_label: Filter by week description: Query data by ISO week number using week_of_year() or dateadd() for better performance --- @@ -15,7 +15,7 @@ SELECT * FROM trades WHERE week_of_year(timestamp) = 24; ``` -## Solution 2: Using dateadd() (Faster) +## Solution 2: Using dateadd() (faster) However, depending on your table size, especially if you are not filtering by any timestamp, you might prefer this alternative, as it executes faster: @@ -27,7 +27,7 @@ WHERE timestamp >= dateadd('w', 23, '2025-01-01') You need to be careful with that query, as it will start counting time from Jan 1st 1970, which is not a Monday. -## Solution 3: Start at First Monday of Year +## Solution 3: Start at first Monday of year This alternative would start at the Monday of the week that includes January 1st: diff --git a/documentation/cookbook/sql/time-series/force-designated-timestamp.md b/documentation/cookbook/sql/time-series/force-designated-timestamp.md index 083e3bd32..da49887ab 100644 --- a/documentation/cookbook/sql/time-series/force-designated-timestamp.md +++ b/documentation/cookbook/sql/time-series/force-designated-timestamp.md @@ -1,12 +1,12 @@ --- -title: Force a Designated Timestamp +title: Force a designated timestamp sidebar_label: Force designated timestamp description: Learn how to explicitly set a designated timestamp column in QuestDB queries using the TIMESTAMP keyword --- Sometimes you need to force a designated timestamp in your query. This happens when you want to run operations like `SAMPLE BY` with a non-designated timestamp column, or when QuestDB applies certain functions or joins and loses track of the designated timestamp. -## Problem: Lost Designated Timestamp +## Problem: Lost designated timestamp When you run this query on the demo instance, you'll notice the `time` column is not recognized as a designated timestamp because we cast it to a string and back: @@ -24,7 +24,7 @@ LIMIT 10; Without a designated timestamp, you cannot use time-series operations like `SAMPLE BY`. -## Solution: Use the TIMESTAMP Keyword +## Solution: Use the TIMESTAMP keyword You can force the designated timestamp using the `TIMESTAMP()` keyword, which allows you to run time-series operations: @@ -47,7 +47,7 @@ SELECT * FROM t LATEST BY symbol; The `TIMESTAMP(time)` clause explicitly tells QuestDB which column to use as the designated timestamp, enabling `LATEST BY` and other time-series operations. This query gets the most recent price for each symbol in the last hour. -## Common Case: UNION Queries +## Common case: UNION queries The designated timestamp is often lost when using `UNION` or `UNION ALL`. This is intentional - QuestDB cannot guarantee that the combined results are in order, and designated timestamps must always be in ascending order. @@ -68,7 +68,7 @@ LIMIT 10; This query combines the last minute of data twice using `UNION ALL`, then restores the designated timestamp. -## Querying External Parquet Files +## Querying external Parquet files When querying external parquet files using `read_parquet()`, the result does not have a designated timestamp. You need to force it using `TIMESTAMP()` to enable time-series operations like `SAMPLE BY`: diff --git a/documentation/cookbook/sql/time-series/latest-activity-window.md b/documentation/cookbook/sql/time-series/latest-activity-window.md index 48e8cc387..25c9f0358 100644 --- a/documentation/cookbook/sql/time-series/latest-activity-window.md +++ b/documentation/cookbook/sql/time-series/latest-activity-window.md @@ -1,5 +1,5 @@ --- -title: Query Last N Minutes of Activity +title: Query last N minutes of activity sidebar_label: Latest activity window description: Get rows from the last N minutes of recorded activity using subqueries with LIMIT -1 --- diff --git a/documentation/cookbook/sql/time-series/latest-n-per-partition.md b/documentation/cookbook/sql/time-series/latest-n-per-partition.md index a303e2df5..13be6ca66 100644 --- a/documentation/cookbook/sql/time-series/latest-n-per-partition.md +++ b/documentation/cookbook/sql/time-series/latest-n-per-partition.md @@ -1,12 +1,12 @@ --- -title: Get Latest N Records Per Partition +title: Get latest N records per partition sidebar_label: Latest N per partition description: Retrieve the most recent N rows for each distinct value using window functions and filtering --- Retrieve the most recent N rows for each distinct partition value (e.g., latest 5 trades per symbol, last 10 readings per sensor). While `LATEST ON` returns only the single most recent row per partition, this pattern extends it to get multiple recent rows per partition. -## Problem: Need Multiple Recent Rows Per Group +## Problem: Need multiple recent rows per group You want to get the latest N rows for each distinct value in a column. For example: - Latest 5 trades for each trading symbol @@ -23,7 +23,7 @@ LATEST ON timestamp PARTITION BY symbol; But you need multiple rows per symbol. -## Solution: Use ROW_NUMBER() Window Function +## Solution: Use ROW_NUMBER() window function Use `row_number()` to rank rows within each partition, then filter to keep only the top N: @@ -43,7 +43,7 @@ ORDER BY symbol, timestamp DESC; This returns up to 5 most recent trades for each symbol from the last day. -## How It Works +## How it works The query uses a two-step approach: @@ -71,7 +71,7 @@ The query uses a two-step approach: With `WHERE rn <= 3`, we keep rows 1-3 for each symbol. -## Adapting the Query +## Adapting the query **Different partition columns:** ```sql @@ -136,7 +136,7 @@ FROM ranked WHERE rn <= 5; ``` -## Alternative: Use Negative LIMIT +## Alternative: Use negative LIMIT For a simpler approach when you need the latest N rows **total** (not per partition), use negative LIMIT: @@ -157,7 +157,7 @@ LIMIT -100; **But this doesn't work per partition** - it returns 100 total rows, not 100 per symbol. -## Performance Optimization +## Performance optimization **Filter by timestamp first:** ```sql @@ -184,7 +184,7 @@ WHERE timestamp in today() AND symbol IN ('BTC-USDT', 'ETH-USDT', 'SOL-USDT') ``` -## Top N with Aggregates +## Top N with aggregates Combine with aggregates to get summary statistics for top N: diff --git a/documentation/cookbook/sql/time-series/remove-outliers.md b/documentation/cookbook/sql/time-series/remove-outliers.md index 6d908985a..c7b2c84c1 100644 --- a/documentation/cookbook/sql/time-series/remove-outliers.md +++ b/documentation/cookbook/sql/time-series/remove-outliers.md @@ -1,5 +1,5 @@ --- -title: Remove Outliers from Candle Data +title: Remove outliers from candle data sidebar_label: Remove OHLC outliers description: Filter outliers in OHLC candles using window functions to compare against moving averages --- diff --git a/documentation/cookbook/sql/time-series/sample-by-interval-bounds.md b/documentation/cookbook/sql/time-series/sample-by-interval-bounds.md index bca427729..5f76d7067 100644 --- a/documentation/cookbook/sql/time-series/sample-by-interval-bounds.md +++ b/documentation/cookbook/sql/time-series/sample-by-interval-bounds.md @@ -1,5 +1,5 @@ --- -title: Right Interval Bound with SAMPLE BY +title: Right interval bound with SAMPLE BY sidebar_label: Interval bounds description: Shift SAMPLE BY timestamps to use right interval bound instead of left bound --- diff --git a/documentation/cookbook/sql/time-series/session-windows.md b/documentation/cookbook/sql/time-series/session-windows.md index 7df8446f4..c819cc6dd 100644 --- a/documentation/cookbook/sql/time-series/session-windows.md +++ b/documentation/cookbook/sql/time-series/session-windows.md @@ -1,12 +1,12 @@ --- -title: Calculate Sessions and Elapsed Time +title: Calculate sessions and elapsed time sidebar_label: Session windows description: Identify sessions by detecting state changes and calculate elapsed time between events using window functions --- Calculate sessions and elapsed time by identifying when state changes occur in time-series data. This "flip-flop" or "session" pattern is useful for analyzing user sessions, vehicle rides, machine operating cycles, or any scenario where you need to track duration between state transitions. -## Problem: Track Time Between State Changes +## Problem: Track time between state changes You have a table tracking vehicle lock status over time and want to calculate ride duration. A ride starts when `lock_status` changes from `true` (locked) to `false` (unlocked), and ends when it changes back to `true`. @@ -31,7 +31,7 @@ CREATE TABLE vehicle_events ( You want to calculate the duration of each ride. -## Solution: Session Detection with Window Functions +## Solution: Session detection with window functions Use window functions to detect state changes, assign session IDs, then calculate durations: @@ -85,11 +85,11 @@ WHERE lock_status = false AND prev_ts IS NOT NULL; | 10:25:00 | V001 | 1200 | | 10:45:00 | V001 | 900 | -## How It Works +## How it works The query uses a five-step approach: -### 1. Get Previous Status (`prevEvents`) +### 1. Get previous status (`prevEvents`) ```sql lag(lock_status::int) OVER (PARTITION BY vehicle_id ORDER BY timestamp) @@ -97,7 +97,7 @@ lag(lock_status::int) OVER (PARTITION BY vehicle_id ORDER BY timestamp) For each row, get the status from the previous row. Convert boolean to integer (0/1) since `lag` doesn't support boolean types directly. -### 2. Detect State Changes (`ride_sessions`) +### 2. Detect state changes (`ride_sessions`) ```sql SUM(CASE WHEN lock_status != prev_status THEN 1 ELSE 0 END) @@ -110,7 +110,7 @@ Whenever status changes, increment a counter. This creates sequential session ID - Ride 2: After second state change - ... -### 3. Create Global Session IDs (`global_sessions`) +### 3. Create global session IDs (`global_sessions`) ```sql concat(vehicle_id, '#', ride) @@ -118,7 +118,7 @@ concat(vehicle_id, '#', ride) Combine vehicle_id with ride number to create unique session identifiers across all vehicles. -### 4. Get Session Start Times (`totals`) +### 4. Get session start times (`totals`) ```sql SELECT first(timestamp) as ts, ... @@ -128,7 +128,7 @@ GROUP BY session For each session, get the timestamp and status at the beginning of that session. -### 5. Calculate Duration (`prev_ts`) +### 5. Calculate duration (`prev_ts`) ```sql lag(timestamp) OVER (PARTITION BY vehicle_id ORDER BY timestamp) @@ -136,7 +136,7 @@ lag(timestamp) OVER (PARTITION BY vehicle_id ORDER BY timestamp) Get the timestamp from the previous session (for the same vehicle), then use `datediff('s', prev_ts, timestamp)` to calculate duration in seconds. -### Filter for Rides +### Filter for rides ```sql WHERE lock_status = false @@ -144,7 +144,7 @@ WHERE lock_status = false Only show sessions where status is `false` (unlocked), which represents completed rides. The duration is from the previous session end (lock) to this session start (unlock). -## Monthly Aggregation +## Monthly aggregation Calculate total ride duration per vehicle per month: @@ -194,7 +194,7 @@ GROUP BY month, vehicle_id ORDER BY month, vehicle_id; ``` -## Adapting to Different Use Cases +## Adapting to different use cases **User website sessions (1 hour timeout):** ```sql diff --git a/documentation/cookbook/sql/time-series/sparse-sensor-data.md b/documentation/cookbook/sql/time-series/sparse-sensor-data.md index 0e46939c0..c905cde10 100644 --- a/documentation/cookbook/sql/time-series/sparse-sensor-data.md +++ b/documentation/cookbook/sql/time-series/sparse-sensor-data.md @@ -1,5 +1,5 @@ --- -title: Join Strategies for Sparse Sensor Data +title: Join strategies for sparse sensor data sidebar_label: Sparse sensor data description: Compare CROSS JOIN, LEFT JOIN, and ASOF JOIN strategies for combining data from sensors stored in separate tables --- @@ -28,7 +28,7 @@ LIMIT 100000; This works, but it is not super fast (1sec for 10 million rows, in a table with 120 sensor columns and with 10k different vehicle_ids), and it is also not very efficient because `null` columns take some bytes on disk. -## Solution: Multiple Narrow Tables with Joins +## Solution: Multiple narrow tables with joins A single table works, but there is a more efficient (although a bit more cumbersome if you compose queries by hand) way to do this. diff --git a/documentation/schema-design-essentials.md b/documentation/schema-design-essentials.md index acb10dd8c..e5b863486 100644 --- a/documentation/schema-design-essentials.md +++ b/documentation/schema-design-essentials.md @@ -147,6 +147,7 @@ For timezone handling at query time, see | `LONG` | 64 bits | -9.2E18 to 9.2E18 | | `FLOAT` | 32 bits | Single precision IEEE 754 | | `DOUBLE` | 64 bits | Double precision IEEE 754 | +| `DECIMAL` | 1-32 bytes | [Variable based on precision](/docs/query/datatypes/decimal/#storage) | Choose the smallest type that fits your data to save storage. @@ -227,6 +228,31 @@ the view are instant regardless of base table size. See [Materialized Views](/docs/concepts/materialized-views/) for details. +## Views + +When query performance is acceptable and you don't need materialization, use views to abstract complex queries: + +```questdb-sql +CREATE VIEW recent_trades AS ( + SELECT * FROM trades + WHERE timestamp > dateadd('d', -7, now()) +); +``` + +Views can be parameterized using `DECLARE OVERRIDABLE`: + +```questdb-sql +CREATE VIEW trades_above AS ( + DECLARE OVERRIDABLE @min_price := 100 + SELECT * FROM trades WHERE price >= @min_price +); + +-- Override at query time +DECLARE @min_price := 500 SELECT * FROM trades_above; +``` + +See [Views](/docs/concepts/views/) for details. + ## Common mistakes ### Using VARCHAR for categorical data diff --git a/documentation/sidebars.js b/documentation/sidebars.js index b723766d3..9b63e465d 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -749,6 +749,7 @@ module.exports = { "cookbook/sql/finance/vwap", "cookbook/sql/finance/bollinger-bands", "cookbook/sql/finance/tick-trin", + "cookbook/sql/finance/aggressor-volume-imbalance", "cookbook/sql/finance/volume-profile", "cookbook/sql/finance/volume-spike", "cookbook/sql/finance/rolling-stddev",