diff --git a/__del_test__.tmp b/__del_test__.tmp deleted file mode 100644 index e69de29..0000000 diff --git a/docs/pxf/7x/administering.md b/docs/pxf/7x/administering.md deleted file mode 100644 index f0726cb..0000000 --- a/docs/pxf/7x/administering.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -title: Managing the PXF cluster -navTitle: Managing the PXF cluster -description: Learn how to start, stop, and check the status of PXF, and how to apply a configuration change. ---- - -Once PXF is running, you manage it from the coordinator host using the `pxf cluster` commands. - -## Understanding PXF directories and configuration files - -PXF uses two main directories: - -- `$PXF_HOME` is the read-only directory where PXF is installed, for example `/usr/local/pxf`. It holds the PXF service itself, along with the connector site templates under `templates/` that you copy into a server configuration. -- `$PXF_BASE` is the writable runtime directory you create when you [initialize your cluster](configuring.md#initializing-and-starting-pxf), for example `$HOME/pxf-base`. It holds your configuration, including your server configurations under `servers/`. Keeping `$PXF_BASE` separate from `$PXF_HOME` means a PXF upgrade doesn't overwrite your configuration. - -See [Connecting to external data](connecting/index.md) for how PXF servers and profiles work. - -### Relocating $PXF_BASE - -Move `$PXF_BASE` to a new location if you're running low on disk space or reorganizing storage across hosts: - -1. Stop PXF on every host: - - ```bash - pxf cluster stop - ``` - -1. On the coordinator, move `$PXF_BASE` to the new location and update the environment variable, including in `~/.bashrc`: - - ```bash - mv $PXF_BASE /new/path/to/pxf-base - export PXF_BASE=/new/path/to/pxf-base - ``` - -1. Sync the relocated directory out to every segment host, then start PXF: - - ```bash - pxf cluster sync - pxf cluster start - ``` - -1. Once PXF is confirmed running from the new location, clean up the old `$PXF_BASE` directory on each segment host. - -## Starting, stopping, and restarting PXF - -Start, stop, or restart PXF across the whole cluster with a single command, rather than repeating it on every host. Only the `gpadmin` user can run these commands: - -```bash -pxf cluster start -pxf cluster stop -pxf cluster restart -``` - -## Applying a configuration change - -Sync any change to a file under `$PXF_BASE/conf`, or to a server configuration, out to every segment host, then restart PXF. For example, to increase the PXF Java heap size: - -1. Edit `PXF_JVM_OPTS` in `$PXF_BASE/conf/pxf-env.sh` on the coordinator: - - ```bash - export PXF_JVM_OPTS="-Xmx4g -Xms2g" - ``` - -1. Sync the change to every segment host: - - ```bash - pxf cluster sync - ``` - -1. Restart PXF to apply it. Changes made only on the coordinator have no effect until you sync and restart: - - ```bash - pxf cluster restart - ``` - -### Configuring advanced settings - -Adjust settings under `$PXF_BASE/conf` to fit your environment, the same way as any other configuration change: - -- Change the listen address, overall log level, or thread limit by editing `pxf-application.properties`. -- Change the JVM heap size by editing `pxf-env.sh`. -- Set the hostname or port WHPG uses to reach PXF by setting the `PXF_HOST` and `PXF_PORT` environment variables in `~/.bashrc` on the relevant host. This change also requires restarting WHPG itself, not just PXF. -- Fine-tune logging for individual loggers by editing `pxf-log4j2.xml`. -- Define a custom PXF profile in `pxf-profiles.xml`, pairing a profile name with Java classes that implement PXF's plugin interfaces to read or write it. - -See [PXF configuration files](reference/configuration-files.md) for the full list of properties in each file. - -## Monitoring PXF - -`pxf cluster status` confirms PXF is both running and responsive on every host: - -```bash -pxf cluster status -``` - -PXF also exposes HTTP endpoints on each host for monitoring tools, at `http://:5888/`: - -| Endpoint | Description | -|---|---| -| `actuator/health` | The status of the PXF service. | -| `actuator/info` | Build information for the PXF service. | -| `actuator/metrics` | JVM, Tomcat, system, and PXF-specific metrics. | -| `actuator/prometheus` | The same metrics, in a format a Prometheus server can scrape. | diff --git a/docs/pxf/7x/configuring.md b/docs/pxf/7x/configuring.md deleted file mode 100644 index 3e195ea..0000000 --- a/docs/pxf/7x/configuring.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: Configuring and starting PXF for WarehousePG -navTitle: Configuring and starting -description: Learn how to configure PXF's runtime environment, initialize it across your cluster, and start the PXF service. ---- - -After you install PXF on every host in your WarehousePG (WHPG) cluster, set up its environment, initialize and start the service, and create the `pxf` extension. - -## Setting environment variables - -Set `$PXF_HOME` to the directory created during installation, so PXF and your shell can find the PXF service and its default configuration templates. PXF also needs a separate, writable runtime directory, [`$PXF_BASE`](administering.md#understanding-pxf-directories-and-configuration-files), where your per-cluster configuration, server definitions, and logs live. Keeping `$PXF_BASE` separate from `$PXF_HOME` means a PXF upgrade doesn't overwrite your configuration. - -On the coordinator, set `PXF_HOME`, `PXF_BASE`, and add the PXF `bin` directory to your `PATH`: - -```bash -export PXF_HOME=/usr/local/pxf -export PXF_BASE=$HOME/pxf-base -export PATH="$PXF_HOME/bin:$PATH" -``` - -Add these lines to `~/.bashrc` on the coordinator so they persist across sessions. - -## Initializing and starting PXF - -Set up PXF's runtime configuration across the cluster and start the service, so PXF is ready to handle queries. Run the following commands from the coordinator host. Each `pxf cluster` command applies the action to every host in the cluster. - -1. Create the runtime configuration directory on every host: - - ```bash - pxf cluster prepare - ``` - - This command creates `$PXF_BASE` on every host and copies the default configuration templates from `$PXF_HOME/conf` into it. If `$PXF_BASE` doesn't already exist, `pxf cluster prepare` creates it for you. - -1. Set `JAVA_HOME` in `$PXF_BASE/conf/pxf-env.sh` on the coordinator: - - ```bash - echo "export JAVA_HOME=$(readlink -f $(which java) | sed 's:/bin/java$::')" >> $PXF_BASE/conf/pxf-env.sh - ``` - -1. Sync the change to every host: - - ```bash - pxf cluster sync - ``` - -1. Start the PXF Java service on every host, listening on port 5888 by default: - - ```bash - pxf cluster start - ``` - -1. Confirm PXF is running on every host: - - ```bash - pxf cluster status - __OUTPUT__ - Checking status of PXF servers on coordinator host and 2 segment hosts... - PXF is running on 3 out of 3 hosts - ``` - -## Creating the PXF extension - -Create the `pxf` extension in each database that needs external table access. `pxf cluster register`, run as the last step of [installing PXF](installing.md#installing-on-the-cluster), already placed the extension's control, SQL, and shared library files under `$GPHOME` on every host, so you don't need to repeat it here. - -1. Connect to the target database and create the extension: - - ```sql - CREATE EXTENSION IF NOT EXISTS pxf; - ``` - - If the extension already exists from a previous PXF version, update it instead: - - ```sql - ALTER EXTENSION pxf UPDATE; - ``` - - Repeat this step in every database where you want to query external tables. - -1. Grant roles access. Only WHPG superusers can use the `pxf` protocol by default, so grant `SELECT` to let a role read external tables through PXF, and `INSERT` to let it write them: - - ```sql - GRANT SELECT ON PROTOCOL pxf TO ; - GRANT INSERT ON PROTOCOL pxf TO ; - ``` - -::: info Note -If you need to drop the `pxf` extension, use `DROP EXTENSION pxf CASCADE` to drop it along with any external table still using the `pxf` protocol. -::: - -## Next steps - -See [Managing the PXF cluster](administering.md) for the PXF configuration files and ongoing cluster management commands, or go straight to configuring a connector, such as [Object stores](connecting/object-stores/index.md) or [Hadoop](connecting/hadoop/index.md), to reach an external data source. If you'd rather query through standard Postgres foreign tables instead of `pxf://` external tables, see [Using the foreign data wrapper](foreign-data-wrapper.md). diff --git a/docs/pxf/7x/connecting/hadoop/hbase.md b/docs/pxf/7x/connecting/hadoop/hbase.md deleted file mode 100644 index 4833318..0000000 --- a/docs/pxf/7x/connecting/hadoop/hbase.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -title: Connecting to HBase -navTitle: HBase -description: Configure PXF to read HBase table data and query it from WarehousePG. ---- - -Configure a server for HBase, then query its table data through external tables. PXF only supports reading from HBase, not writing to it. - -## Configuring the server - -Create a server that connects to an HBase cluster. - -::: info Note -If you plan to filter on HBase external tables, copy `$PXF_HOME/share/pxf-hbase-*.jar` to every node in the HBase cluster first, and add its location to `$HBASE_CLASSPATH`. HBase's region servers need this JAR to execute a pushed-down filter. -::: - -1. Create a server directory under `$PXF_BASE/servers`, and copy the `hbase-site.xml` template into it. For example, to configure a server named `hbase_server`: - - ```bash - mkdir $PXF_BASE/servers/hbase_server - cp $PXF_HOME/templates/hbase-site.xml $PXF_BASE/servers/hbase_server - ``` - -1. Edit `hbase-site.xml` with your connection details: - - ```xml - - - - hbase.rootdir - hdfs://:/hbase - - - hbase.zookeeper.quorum - - - - ``` - - Where: - - `hbase.rootdir` is the location on HDFS where HBase stores its data. - - `hbase.zookeeper.quorum` is the ZooKeeper ensemble HBase uses for coordination. - - See [Configuration templates](../../reference/configuration-templates.md#hbase-sitexml) for the full list of `hbase-site.xml` properties. - -1. Sync the change to every segment host, then restart PXF to apply it: - - ```bash - pxf cluster sync - pxf cluster restart - ``` - -## Reading data - -PXF maps each HBase column to a `:` column name. The following example reads an HBase table named `employees`, with `personal` and `job` column families, using the `hbase_server` server: - -```sql -CREATE EXTERNAL TABLE employee_hbase ("personal:name" text, "job:department" text, "job:salary" int) - LOCATION ('pxf://employees?PROFILE=hbase&SERVER=hbase_server') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); - -SELECT * FROM employee_hbase; -``` - -WarehousePG (WHPG) caps column names at 63 characters and requires them to be character-based, while an HBase qualifier name can be longer or contain binary data. When a qualifier doesn't fit those rules, map it indirectly through a lookup table instead. Create a table named `pxflookup` in HBase with a single column family named `mapping`, then add a row per mapped qualifier, using the target HBase table's name as the row key and the WHPG column alias as the qualifier under `mapping`: - -``` -create 'pxflookup', 'mapping' -put 'pxflookup', 'employees', 'mapping:name', 'personal:name' -put 'pxflookup', 'employees', 'mapping:dept', 'job:department' -``` - -Reference the aliases directly as column names when you create the external table: - -```sql -CREATE EXTERNAL TABLE employee_hbase_mapped (name text, dept text) - LOCATION ('pxf://employees?PROFILE=hbase&SERVER=hbase_server') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); -``` - -Add a `recordkey bytea` column to also read the row's key, HBase's unique per-row identifier that isn't stored in any column family. Type it `text` instead of `bytea` if you want to filter on it in a `WHERE` clause and have that filter pushed down to HBase. diff --git a/docs/pxf/7x/connecting/hadoop/hdfs.md b/docs/pxf/7x/connecting/hadoop/hdfs.md deleted file mode 100644 index 699f9ef..0000000 --- a/docs/pxf/7x/connecting/hadoop/hdfs.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: Connecting to HDFS -navTitle: HDFS -description: Configure PXF to read data from the Hadoop Distributed File System (HDFS) and query it from WarehousePG. ---- - -Configure a server for the Hadoop Distributed File System (HDFS), then read and write its data through external tables. If you also plan to use Hive, configure HDFS first, since Hive builds on this same connection. - -## Configuring the server - -Create a server that connects to HDFS. - -1. Create a server directory under `$PXF_BASE/servers`, and copy the `core-site.xml` template from `$PXF_HOME/templates` into it. For example, to configure a server named `hdfssrvcfg`: - - ```bash - mkdir -p $PXF_BASE/servers/hdfssrvcfg - cp $PXF_HOME/templates/core-site.xml $PXF_BASE/servers/hdfssrvcfg - ``` - -1. Edit `fs.defaultFS` in that file to point to your HDFS NameNode: - - ```xml - - fs.defaultFS - hdfs://: - - ``` - - See [Configuration templates](../../reference/configuration-templates.md#core-sitexml) for the full list of `core-site.xml` properties. - -1. Sync the change to every segment host, then restart PXF to apply it: - - ```bash - pxf cluster sync - pxf cluster restart - ``` - -## Reading data - -Read data from HDFS by creating a readable external table with the profile for its format and the server you configured. For example, to read a CSV file using the `hdfssrvcfg` server: - -```sql -CREATE EXTERNAL TABLE pxf_hdfs_example (id int, name text, age int) - LOCATION ('pxf:///data.csv?PROFILE=hdfs:csv&SERVER=hdfssrvcfg') - FORMAT 'CSV' (delimiter=','); - -SELECT * FROM pxf_hdfs_example; -``` - -PXF also supports structured formats like Parquet, through the same profile-based syntax: - -```sql -CREATE EXTERNAL TABLE pxf_parquet_read (id int, name text) - LOCATION ('pxf://parquet_data?PROFILE=hdfs:parquet&SERVER=hdfssrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); - -SELECT * FROM pxf_parquet_read; -``` - -See [PXF profiles](../../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. - -## Writing data - -Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to HDFS, then a separate readable external table at the same location to query it back: - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_parquet_write (id int, name text) - LOCATION ('pxf://parquet_data?PROFILE=hdfs:parquet&SERVER=hdfssrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); - -INSERT INTO pxf_parquet_write VALUES (1, 'New York'); -``` - -```sql -CREATE EXTERNAL TABLE pxf_parquet_read_back (id int, name text) - LOCATION ('pxf://parquet_data?PROFILE=hdfs:parquet&SERVER=hdfssrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); - -SELECT * FROM pxf_parquet_read_back; -``` - -The same pattern applies to other connectors, using their own profile prefix, for example `s3:parquet`. See [Object stores](../object-stores/index.md) for an example. - -## Next steps - -With HDFS configured, you're ready to connect to [Hive](hive.md), which builds on this same connection, or [HBase](hbase.md). If your cluster uses Kerberos, see [Authenticating with Kerberos](kerberos.md). diff --git a/docs/pxf/7x/connecting/hadoop/hive.md b/docs/pxf/7x/connecting/hadoop/hive.md deleted file mode 100644 index 7a26fa5..0000000 --- a/docs/pxf/7x/connecting/hadoop/hive.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: Connecting to Hive -navTitle: Hive -description: Configure PXF to read Hive table data and query it from WarehousePG. ---- - -Configure a server for Hive, then query its table data through external tables. PXF only supports reading from Hive, not writing to it. Configure the Hadoop Distributed File System (HDFS) first, since Hive builds on that same connection. - -## Configuring the server - -Hive tables live on HDFS, so PXF reuses the server you already configured for HDFS, for example `hdfssrvcfg` from [HDFS](hdfs.md). Add `hive-site.xml` to that same server directory. - -1. Copy `hive-site.xml` from the templates directory into the server: - - ```bash - cp $PXF_HOME/templates/hive-site.xml $PXF_BASE/servers/hdfssrvcfg - ``` - -1. Edit `hive-site.xml` to point to your Hive metastore: - - ```xml - - - - hive.metastore.uris - thrift://:9083 - - - ``` - - `hive.metastore.uris` is the Thrift URI PXF uses to reach your Hive metastore. See [Configuration templates](../../reference/configuration-templates.md#hive-sitexml) for the full list of `hive-site.xml` properties. - -1. Sync the change to every segment host, then restart PXF to apply it: - - ```bash - pxf cluster sync - pxf cluster restart - ``` - -## Reading data - -Read a Hive table by creating a readable external table with the `hive` profile and the server you configured. For example, to read a table named `employee` in the `hr` Hive database, using the `hdfssrvcfg` server: - -```sql -CREATE EXTERNAL TABLE emp_hive (id int, name text) - LOCATION ('pxf://hr.employee?PROFILE=hive&SERVER=hdfssrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); - -SELECT * FROM emp_hive; -``` - -The path in `LOCATION` is `.`. - -The `hive` profile works with any format Hive supports, reading through Hive's SerDe (serializer/deserializer), the class Hive itself uses to parse that format. For a table stored in text, RCFile, or ORC, switch to the matching `hive:` profile instead for faster reads that skip the SerDe and go straight to the files. - -`hive:orc` also supports `VECTORIZE=true`, to read up to 1024 rows at once. Vectorized reads don't support complex types or the `timestamp` data type: - -```sql -CREATE EXTERNAL TABLE emp_hive_orc_vectorized (id int, name text) - LOCATION ('pxf://hr.employee?PROFILE=hive:orc&SERVER=hdfssrvcfg&VECTORIZE=true') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); -``` - -See [PXF profiles](../../reference/profiles.md#hive) for the full list of Hive profiles. - -## Filtering on partitions - -For a Hive table partitioned on one or more columns, a `WHERE` clause on a partition column lets PXF skip non-matching partitions instead of reading and filtering every row in WarehousePG (WHPG). Which comparisons trigger this depends on the partition column's type: - -- On a string-typed column, `=`, `<>`, `<`, `<=`, `>`, and `>=` all work. -- On an integral-typed column (such as `int`), `=` and `<>` work when you enable `hive.metastore.integral.jdo.pushdown` in `hive-site.xml`. - -You must define partition columns at the end of your `CREATE EXTERNAL TABLE` column list, matching Hive's names and order. - -Hive excludes a row from partition queries if its value doesn't match the partition column's type, commonly a `NULL`. PXF instead surfaces that row under an actual `NULL`, so `IS NULL` queries can return different row counts between Hive and PXF. diff --git a/docs/pxf/7x/connecting/hadoop/index.md b/docs/pxf/7x/connecting/hadoop/index.md deleted file mode 100644 index 39075d9..0000000 --- a/docs/pxf/7x/connecting/hadoop/index.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: Connecting to Hadoop -navTitle: Connecting to Hadoop -description: Configure PXF to connect to a Hadoop cluster and query HDFS, Hive, and HBase data from WarehousePG. ---- - -PXF's Hadoop connectors, the Hadoop Distributed File System (HDFS), Hive, and HBase, share the same underlying cluster configuration. Configure the connection to HDFS first if you also plan to use Hive, since Hive builds on top of it. - -- [HDFS](hdfs.md): Read and write files directly on HDFS. -- [Hive](hive.md): Read Hive table data. -- [HBase](hbase.md): Read HBase table data. -- [Authenticating with Kerberos](kerberos.md): Connect to a Kerberized Hadoop cluster and configure user impersonation. diff --git a/docs/pxf/7x/connecting/hadoop/kerberos.md b/docs/pxf/7x/connecting/hadoop/kerberos.md deleted file mode 100644 index 094a3b0..0000000 --- a/docs/pxf/7x/connecting/hadoop/kerberos.md +++ /dev/null @@ -1,56 +0,0 @@ ---- -title: Authenticating with Kerberos -navTitle: Authenticating with Kerberos -description: Configure PXF to authenticate with a Kerberized Hadoop cluster and impersonate the connecting WHPG user. ---- - -Authenticate PXF to a Kerberized Hadoop cluster using a Kerberos principal and keytab, instead of connecting as an unauthenticated user. This authentication method applies to any Hadoop connector, the Hadoop Distributed File System (HDFS), Hive, or HBase. - -## Enabling Kerberos authentication - -Add these settings to the same server directory you already configured for HDFS, Hive, or HBase. - -::: info Note -This procedure requires a Kerberos principal and keytab for PXF, generated with `kadmin` (MIT Kerberos) or `ktpass` (Active Directory) on your Kerberos Key Distribution Center (KDC), or provided by your Kerberos administrator. -::: - -1. Copy the `pxf-site.xml` template into your server directory: - - ```bash - cp $PXF_HOME/templates/pxf-site.xml $PXF_BASE/servers/ - ``` - -1. Edit `pxf.service.kerberos.principal` and `pxf.service.kerberos.keytab` with your principal and keytab path: - - ```xml - - pxf.service.kerberos.principal - gpadmin/_HOST@EXAMPLE.COM - - - pxf.service.kerberos.keytab - ${pxf.base}/keytabs/pxf.service.keytab - - ``` - - PXF replaces `_HOST` in the principal with each host's fully qualified domain name. See [Configuration templates](../../reference/configuration-templates.md#pxf-sitexml) for the full list of `pxf-site.xml` properties. - -1. Copy the keytab file to `$PXF_BASE/keytabs` on every segment host, at the same path. `pxf cluster sync` doesn't distribute this directory, so you need to place it there yourself. - -1. Sync the server configuration to every segment host, then restart PXF to apply it: - - ```bash - pxf cluster sync - pxf cluster restart - ``` - -To connect to a Kerberized Hive metastore over JDBC instead, set `hadoop.security.authentication` to `kerberos` in that server's `jdbc-site.xml`. See [Connecting to SQL databases over JDBC](../jdbc.md). - -## Tuning Kerberos parameters - -Edit optional properties the same way as [Enabling Kerberos authentication](#enabling-kerberos-authentication) above, in that same server's `pxf-site.xml`, then sync and restart PXF to apply the change: - -- `pxf.service.user.impersonation`: By default, PXF impersonates the WarehousePG (WHPG) user running the query when it connects to the external system, so the Hadoop cluster sees requests as coming from that user rather than from PXF's own principal. Set this to `false` to disable impersonation and connect as the PXF principal for every query instead. -- `pxf.service.user.name`: Impersonates a specific proxy user rather than the connecting WHPG user. Use this on unsecured clusters where a proxy user is already configured, or to make every WHPG user appear as this one user on the Hadoop side. -- `pxf.service.kerberos.constrained-delegation`: Set to `true` to impersonate using Kerberos constrained delegation (S4U2Self/Proxy) instead of requiring PXF's principal to be a Hadoop proxy user, for clusters that support it. This delegation mode needs additional configuration on your Active Directory or IPA server. -- `pxf.service.kerberos.ticket-renew-window`: The fraction of a Kerberos ticket's lifespan PXF waits through before refreshing it, `0.8` by default. Set it to `0` to request a new ticket on every request. diff --git a/docs/pxf/7x/connecting/index.md b/docs/pxf/7x/connecting/index.md deleted file mode 100644 index 250810e..0000000 --- a/docs/pxf/7x/connecting/index.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -title: Connecting to external data -navTitle: Connecting to external data -description: Learn how PXF servers and profiles work, and how to reference them when you create an external table. ---- - -Reach external data from WarehousePG (WHPG) by pointing a PXF external table at a server, for connection details, and a profile, for the connector and data format to read or write it with. Understand how servers, profiles, and formats work before you configure one for the connector you need, on [Object stores](object-stores/index.md), [Hadoop](hadoop/index.md), [Connecting to SQL databases over JDBC](jdbc.md), or [Network file system](network-file-system.md). - -Every connector, regardless of the external source it reaches, relies on the same two pieces of configuration: a server for connection details and a profile for the data format, such as Parquet, Avro, or JSON. An external table's `LOCATION` clause references both: - -```sql -CREATE EXTERNAL TABLE sales (id int, name text, amount numeric) - LOCATION ('pxf://data/sales.csv?PROFILE=s3:text&SERVER=example') - FORMAT 'CSV' (delimiter=','); -``` - -## Understanding PXF servers - -PXF connects to an external data source through a named server configuration to read or write data. Each server is a directory under `$PXF_BASE/servers` that contains one or more site XML files with the connection settings for that source, such as the endpoint URL and credentials. - -When you initialize your cluster, `pxf cluster prepare` creates an empty `default` server. To read and write from several sources, create multiple servers by adding a separate subdirectory for each one under `$PXF_BASE/servers`, with its own site XML files. `$PXF_HOME/templates` holds a sample site XML file for each connector, ready to copy into your server directory and edit. See [Configuration templates](../reference/configuration-templates.md) for the properties in each. - -::: info Note -If `LOCATION` omits `SERVER`, PXF falls back to `$PXF_BASE/servers/default`. -::: - -## Understanding PXF profiles - -A PXF profile name follows a `:` pattern, for example `hdfs:parquet` or `s3:avro`, and PXF uses it to pick which connector and format code handles the request. - -PXF ships with a built-in profile for each connector and format combination it supports, compiled into the PXF service. See [PXF profiles](../reference/profiles.md) for the full list. - -PXF also supports custom profiles, defined in [`pxf-profiles.xml`](../reference/configuration-files.md#pxf-profilesxml) and backed by your own Java plugin classes. - -The `format` part of a profile works the same way regardless of which connector it's paired with. PXF supports text, CSV, Parquet, ORC, Avro, JSON, and SequenceFile, depending on the connector. See [PXF profiles](../reference/profiles.md) for the full list of formats and which connectors support each. - -## Choosing a connector - -Configure the connector that matches your external data source. - -- [Object stores](object-stores/index.md): Amazon S3, MinIO, Azure Blob Storage, Azure Data Lake Storage, and Google Cloud Storage. -- [Hadoop](hadoop/index.md): HDFS, Hive, and HBase. -- [Connecting to SQL databases over JDBC](jdbc.md): external SQL databases through a JDBC driver. -- [Network file system](network-file-system.md): a shared network file system. diff --git a/docs/pxf/7x/connecting/jdbc.md b/docs/pxf/7x/connecting/jdbc.md deleted file mode 100644 index 62fac7c..0000000 --- a/docs/pxf/7x/connecting/jdbc.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: Connecting to a SQL database over JDBC -navTitle: Connecting to SQL databases over JDBC -description: Configure PXF to read and write data in an external SQL database over JDBC, and query it from WarehousePG. ---- - -Configure a server for your SQL database, then read and write its data through external tables over JDBC. - -## Configuring the server - -Create a server that connects to an external SQL database over JDBC. - -1. Create the server directory and copy the `jdbc-site.xml` template into it: - - ```bash - mkdir -p $PXF_BASE/servers/jdbcsrvcfg - cp $PXF_HOME/templates/jdbc-site.xml $PXF_BASE/servers/jdbcsrvcfg - ``` - -1. Edit `jdbc-site.xml` with your connection details. For example, to connect to a WHPG database named `gpadmin` using the bundled PostgreSQL JDBC driver: - - ```xml - - - - jdbc.driver - org.postgresql.Driver - - - jdbc.url - jdbc:postgresql://:/gpadmin - - - jdbc.user - - - - jdbc.password - - - - ``` - - PXF bundles the PostgreSQL and Hive JDBC drivers. For any other database, copy the vendor's JDBC driver JAR to `$PXF_BASE/lib` before applying the change. - - `jdbc-site.xml` also has commented-out properties for connection pooling, batch and fetch sizes, session and connection-level settings, and user impersonation. Uncomment and set only the ones you need. See [Configuration templates](../reference/configuration-templates.md#jdbc-sitexml) for the full list. - -1. Sync the change to every segment host, then restart PXF to apply it: - - ```bash - pxf cluster sync - pxf cluster restart - ``` - -## Reading data - -Read data from the external database by creating a readable external table with the `jdbc` profile. For example, to read the `test` table from the `gpadmin` database configured above: - -```sql -CREATE EXTERNAL TABLE jdbc_read_example (id int) - LOCATION ('pxf://public.test?PROFILE=jdbc&SERVER=jdbcsrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); - -SELECT * FROM jdbc_read_example; -``` - -The path in `LOCATION` is `.`. This same pattern works against any JDBC-compliant database, not only WHPG, once you point `jdbc.url` and `jdbc.driver` at that database and provide its driver JAR. - -The data still lives in the source database. You can load it into a local table with `CREATE TABLE AS`: - -```sql -CREATE TABLE test_local AS SELECT * FROM jdbc_read_example; -``` - -## Writing data - -Create a writable external table with the same `jdbc` profile to insert data into the external database: - -```sql -CREATE WRITABLE EXTERNAL TABLE jdbc_write_example (id int) - LOCATION ('pxf://public.test?PROFILE=jdbc&SERVER=jdbcsrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); - -INSERT INTO jdbc_write_example SELECT * FROM some_local_table; -``` - -To query the data, create a separate readable external table at the same location: - -```sql -CREATE EXTERNAL TABLE pxf_read_back (id int) - LOCATION ('pxf://public.test?PROFILE=jdbc&SERVER=jdbcsrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); - -SELECT * FROM pxf_read_back; -``` diff --git a/docs/pxf/7x/connecting/network-file-system.md b/docs/pxf/7x/connecting/network-file-system.md deleted file mode 100644 index 64e10f3..0000000 --- a/docs/pxf/7x/connecting/network-file-system.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -title: Connecting to a network file system -navTitle: Connecting to a network file system -description: Configure PXF to read and write data on a network file system, and query it from WarehousePG. ---- - -Configure a server for your network file system, then read and write its data through external tables. - -## Configuring the server - -Create a server that points PXF at a directory mounted at the same path on every WarehousePG (WHPG) host. - -1. Create a server directory under `$PXF_BASE/servers`, and copy the `pxf-site.xml` template from `$PXF_HOME/templates` into it. For example, to configure a server named `nfssrvcfg`: - - ```bash - mkdir -p $PXF_BASE/servers/nfssrvcfg - cp $PXF_HOME/templates/pxf-site.xml $PXF_BASE/servers/nfssrvcfg - ``` - -1. Uncomment `pxf.fs.basePath` and set it to the mount point, and set `pxf.service.user.impersonation` to `false`, since this connector always accesses files as the OS user running PXF rather than the connecting WHPG user: - - ```xml - - pxf.fs.basePath - /mnt/extdata/pxffs - - - pxf.service.user.impersonation - false - - ``` - - The path in `LOCATION` is relative to `pxf.fs.basePath`. See [Configuration templates](../reference/configuration-templates.md#pxf-sitexml) for the full list of `pxf-site.xml` properties. - -1. Sync the change to every segment host, then restart PXF to apply it: - - ```bash - pxf cluster sync - pxf cluster restart - ``` - -## Reading data - -Read a file from the mounted directory by creating a readable external table with the profile for its format and the server you configured. For example, to read a CSV file using the `nfssrvcfg` server: - -```sql -CREATE EXTERNAL TABLE pxf_read_example (id int, name text, age int) - LOCATION ('pxf://data.csv?PROFILE=file:text&SERVER=nfssrvcfg') - FORMAT 'CSV' (delimiter=','); - -SELECT * FROM pxf_read_example; -``` - -PXF also supports structured formats like Parquet, through the same profile-based syntax: - -```sql -CREATE EXTERNAL TABLE pxf_parquet_example ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf://parquet_data/?PROFILE=file:parquet&SERVER=nfssrvcfg') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') - ENCODING 'UTF8'; -``` - -The path in `LOCATION` can't be relative, and can't include the `$` character. See [PXF profiles](../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. - -## Writing data - -Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to the mounted directory: - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_write_example ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf://parquet_data/?PROFILE=file:parquet&SERVER=nfssrvcfg') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') - ENCODING 'UTF8'; - -INSERT INTO pxf_write_example SELECT id, created, status FROM some_local_table; -``` - -To query the data, create a separate readable external table at the same location: - -```sql -CREATE EXTERNAL TABLE pxf_read_back ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf://parquet_data/?PROFILE=file:parquet&SERVER=nfssrvcfg') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') - ENCODING 'UTF8'; - -SELECT * FROM pxf_read_back; -``` diff --git a/docs/pxf/7x/connecting/object-stores/azure.md b/docs/pxf/7x/connecting/object-stores/azure.md deleted file mode 100644 index 6bcc84f..0000000 --- a/docs/pxf/7x/connecting/object-stores/azure.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -title: Connecting to Azure -navTitle: Azure -description: Configure PXF to read and write data in Azure Blob Storage or Azure Data Lake Storage, and query it from WarehousePG. ---- - -Configure a server for Azure Blob Storage or Azure Data Lake Storage, then read and write its data through external tables. - -## Configuring the server - -Create a server directory under `$PXF_BASE/servers`, copy the matching XML template from `$PXF_HOME/templates` into it, and edit it with your connection details. - -### Blob Storage - -```bash -mkdir -p $PXF_BASE/servers/wasbssrvcfg -cp $PXF_HOME/templates/wasbs-site.xml $PXF_BASE/servers/wasbssrvcfg -``` - -Edit `wasbs-site.xml` with your storage account name and key: - -```xml - - - - fs.azure.account.key..blob.core.windows.net - - - -``` - -See [Configuration templates](../../reference/configuration-templates.md#wasbs-sitexml) for the full list of `wasbs-site.xml` properties. - -### Data Lake Storage Gen2 - -```bash -mkdir -p $PXF_BASE/servers/abfsssrvcfg -cp $PXF_HOME/templates/abfss-site.xml $PXF_BASE/servers/abfsssrvcfg -``` - -Edit `abfss-site.xml` with the client ID, secret, and token endpoint from an Azure AD app registration: - -```xml - - - - fs.azure.account.auth.type - OAuth - - - fs.azure.account.oauth.provider.type - org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider - - - fs.azure.account.oauth2.client.endpoint - - - - fs.azure.account.oauth2.client.id - - - - fs.azure.account.oauth2.client.secret - - - -``` - -See [Configuration templates](../../reference/configuration-templates.md#abfss-sitexml) for the full list of `abfss-site.xml` properties. - -Sync the change to every segment host, then restart PXF to apply it: - -```bash -pxf cluster sync -pxf cluster restart -``` - -## Reading data - -Read data from the object store by creating a readable external table with the profile for its format and the server you configured. - -### Blob Storage - -```sql -CREATE EXTERNAL TABLE pxf_read_example (id int, name text, age int) - LOCATION ('pxf://@.blob.core.windows.net//data.csv?PROFILE=wasbs:text&SERVER=wasbssrvcfg') - FORMAT 'CSV' (delimiter=','); - -SELECT * FROM pxf_read_example; -``` - -### Data Lake Storage Gen2 - -```sql -CREATE EXTERNAL TABLE pxf_read_example_adls (id int, name text, age int) - LOCATION ('pxf://.dfs.core.windows.net//data.csv?PROFILE=abfss:text&SERVER=abfsssrvcfg') - FORMAT 'CSV' (delimiter=','); - -SELECT * FROM pxf_read_example_adls; -``` - -PXF also supports structured formats like Parquet, through the same profile-based syntax: - -```sql -CREATE EXTERNAL TABLE pxf_parquet_example ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf://@.blob.core.windows.net//?PROFILE=wasbs:parquet&SERVER=wasbssrvcfg') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') - ENCODING 'UTF8'; -``` - -See [PXF profiles](../../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. - -## Writing data - -Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to Azure, then a separate readable external table at the same location to query it back. - -### Blob Storage - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_write_example ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf://@.blob.core.windows.net//?PROFILE=wasbs:parquet&SERVER=wasbssrvcfg&COMPRESSION_CODEC=snappy') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') - ENCODING 'UTF8'; - -INSERT INTO pxf_write_example SELECT id, created, status FROM some_local_table; -``` - -```sql -CREATE EXTERNAL TABLE pxf_read_back ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf://@.blob.core.windows.net//?PROFILE=wasbs:parquet&SERVER=wasbssrvcfg') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') - ENCODING 'UTF8'; - -SELECT * FROM pxf_read_back; -``` - -### Data Lake Storage Gen2 - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_write_example_adls ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf://.dfs.core.windows.net//?PROFILE=abfss:parquet&SERVER=abfsssrvcfg&COMPRESSION_CODEC=snappy') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') - ENCODING 'UTF8'; - -INSERT INTO pxf_write_example_adls SELECT id, created, status FROM some_local_table; -``` - -```sql -CREATE EXTERNAL TABLE pxf_read_back_adls ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf://.dfs.core.windows.net//?PROFILE=abfss:parquet&SERVER=abfsssrvcfg') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') - ENCODING 'UTF8'; - -SELECT * FROM pxf_read_back_adls; -``` diff --git a/docs/pxf/7x/connecting/object-stores/gcs.md b/docs/pxf/7x/connecting/object-stores/gcs.md deleted file mode 100644 index 8583f7c..0000000 --- a/docs/pxf/7x/connecting/object-stores/gcs.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: Connecting to Google Cloud Storage -navTitle: Google Cloud Storage -description: Configure PXF to read and write data in Google Cloud Storage, and query it from WarehousePG. ---- - -Configure a server for Google Cloud Storage, then read and write its data through external tables. - -## Configuring the server - -Create a server that connects to Google Cloud Storage (GCS). - -1. Create a server directory under `$PXF_BASE/servers`, and copy the `gs-site.xml` template from `$PXF_HOME/templates` into it. For example, to configure a server named `gcssrvcfg`: - - ```bash - mkdir -p $PXF_BASE/servers/gcssrvcfg - cp $PXF_HOME/templates/gs-site.xml $PXF_BASE/servers/gcssrvcfg - ``` - -1. Edit `gs-site.xml` with the path to a Google Cloud service account JSON key file, readable by the PXF service on every host: - - ```xml - - - - google.cloud.auth.service.account.enable - true - - - google.cloud.auth.service.account.json.keyfile - - - - ``` - - See [Configuration templates](../../reference/configuration-templates.md#gs-sitexml) for the full list of `gs-site.xml` properties. - -1. Sync the change to every segment host, then restart PXF to apply it: - - ```bash - pxf cluster sync - pxf cluster restart - ``` - -## Reading data - -Read data from the object store by creating a readable external table with the profile for its format and the server you configured. For example, to read a CSV file using the `gcssrvcfg` server: - -```sql -CREATE EXTERNAL TABLE pxf_read_example (id int, name text, age int) - LOCATION ('pxf:////data.csv?PROFILE=gs:text&SERVER=gcssrvcfg') - FORMAT 'CSV' (delimiter=','); - -SELECT * FROM pxf_read_example; -``` - -PXF also supports structured formats like Parquet, through the same profile-based syntax: - -```sql -CREATE EXTERNAL TABLE pxf_parquet_example ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf:////?PROFILE=gs:parquet&SERVER=gcssrvcfg&COMPRESSION_CODEC=snappy') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') - ENCODING 'UTF8'; -``` - -See [PXF profiles](../../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. - -::: info Note -`gs:json` supports reading only, unlike the equivalent JSON profiles for other object stores. -::: - -## Writing data - -Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to GCS: - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_write_example ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf:////?PROFILE=gs:parquet&SERVER=gcssrvcfg&COMPRESSION_CODEC=snappy') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') - ENCODING 'UTF8'; - -INSERT INTO pxf_write_example SELECT id, created, status FROM some_local_table; -``` - -To query the data, create a separate readable external table at the same location: - -```sql -CREATE EXTERNAL TABLE pxf_read_back ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf:////?PROFILE=gs:parquet&SERVER=gcssrvcfg') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') - ENCODING 'UTF8'; - -SELECT * FROM pxf_read_back; -``` diff --git a/docs/pxf/7x/connecting/object-stores/index.md b/docs/pxf/7x/connecting/object-stores/index.md deleted file mode 100644 index 88b017b..0000000 --- a/docs/pxf/7x/connecting/object-stores/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: Connecting to an object store -navTitle: Connecting to object stores -description: Configure PXF to read and write data in Amazon S3, MinIO, Azure, or Google Cloud Storage, and query it from WarehousePG. ---- - -Configure a server for your object store, then read and write its data through external tables. - -- [S3-compatible object stores](s3.md): Amazon S3 and MinIO. -- [Azure](azure.md): Azure Blob Storage and Azure Data Lake Storage. -- [Google Cloud Storage](gcs.md). diff --git a/docs/pxf/7x/connecting/object-stores/s3.md b/docs/pxf/7x/connecting/object-stores/s3.md deleted file mode 100644 index b180c76..0000000 --- a/docs/pxf/7x/connecting/object-stores/s3.md +++ /dev/null @@ -1,230 +0,0 @@ ---- -title: Connecting to an S3-compatible object store -navTitle: S3-compatible stores -description: Configure PXF to read and write data in Amazon S3 or MinIO, and query it from WarehousePG. ---- - -Configure a server for an S3-compatible object store, then read and write its data through external tables. - -## Configuring the server - -Create a server that connects to an S3-compatible object store, such as Amazon S3 or MinIO. - -1. Create a server directory under `$PXF_BASE/servers`, and copy an XML template from `$PXF_HOME/templates` into it. For example, to configure a server named `s3srvcfg` for Amazon S3: - - ```bash - mkdir -p $PXF_BASE/servers/s3srvcfg - cp $PXF_HOME/templates/s3-site.xml $PXF_BASE/servers/s3srvcfg - ``` - -1. Edit the site XML file to authenticate to S3, using whichever of the following methods fits your environment. - - ### Static access keys - - Set your AWS access key and secret key directly: - - ```xml - - - - fs.s3a.access.key - - - - fs.s3a.secret.key - - - - fs.s3a.fast.upload - true - - - ``` - - ### Temporary credentials, through AWS Assume Role - - Obtain temporary credentials yourself, for example through `aws sts assume-role`, then set them along with the session token, and switch the credential provider to one that accepts a session token: - - ```xml - - fs.s3a.access.key - - - - fs.s3a.secret.key - - - - fs.s3a.session.token - - - - fs.s3a.aws.credentials.provider - org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider - - ``` - - ### Named AWS profile - - Point PXF at a profile in `~/.aws/credentials` instead of storing keys in `s3-site.xml`: - - ```xml - - fs.s3a.aws.credentials.provider - com.amazonaws.auth.profile.ProfileCredentialsProvider - - ``` - - Add your keys to `~/.aws/credentials` on every host, under the `default` profile or a named profile: - - ```ini - [test] - aws_access_key_id = - aws_secret_access_key = - ``` - - If you used a named profile instead of `default`, select it by setting `AWS_PROFILE` in `$PXF_BASE/conf/pxf-env.sh`: - - ```bash - export AWS_PROFILE=test - ``` - - ### Credential process - - Use the same `ProfileCredentialsProvider` as the named profile method, but point the profile at a script instead of static keys. The AWS SDK runs the script and reads its stdout as JSON, containing the access key, secret key, session token, and expiration, so this method works with credentials that rotate: - - ```ini - [test] - credential_process = /path/to/credential-script.sh - ``` - - ### IAM role, through an EC2 instance profile - - On EC2, attach an IAM role to the instances running PXF, and omit credentials from `s3-site.xml` entirely. PXF then pulls temporary credentials from the instance metadata service automatically. Set the credential provider explicitly only if you need to rule out another provider taking precedence: - - ```xml - - fs.s3a.aws.credentials.provider - com.amazonaws.auth.InstanceProfileCredentialsProvider - - ``` - - ### Additional tuning properties - - `s3-site.xml` accepts any other Hadoop S3A property, such as `fs.s3a.buffer.dir` for local buffering during upload, `fs.s3a.multipart.size` for the multipart upload threshold, or `fs.s3a.connection.maximum` for connection pooling. See the [S3A section of the Hadoop-AWS module documentation](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#S3A) for the full list of available properties. - - MinIO and other S3-compatible stores also need an endpoint and path-style access, in addition to whichever authentication method you chose above: - - ```xml - - - - fs.s3a.endpoint - http://: - - - fs.s3a.access.key - - - - fs.s3a.secret.key - - - - fs.s3a.fast.upload - true - - - fs.s3a.path.style.access - true - - - ``` - - ::: info Note - Always include the URI scheme (`http://` or `https://`) in `fs.s3a.endpoint`. Omitting it causes connection errors such as `Unsupported or unrecognized SSL message` when PXF tries to reach the endpoint. - ::: - - See [Configuration templates](../../reference/configuration-templates.md#s3-sitexml) for the full list of `s3-site.xml` properties, or [minio-site.xml](../../reference/configuration-templates.md#minio-sitexml) for MinIO and other S3-compatible stores. - -1. Sync the change to every segment host, then restart PXF to apply it: - - ```bash - pxf cluster sync - pxf cluster restart - ``` - -## Reading data - -Read data from the object store by creating a readable external table with the profile for its format and the server you configured. For example, to read a CSV file using the `s3srvcfg` server: - -```sql -CREATE EXTERNAL TABLE pxf_read_example (id int, name text, age int) - LOCATION ('pxf:////data.csv?PROFILE=s3:text&SERVER=s3srvcfg') - FORMAT 'CSV' (delimiter=','); - -SELECT * FROM pxf_read_example; -``` - -PXF also supports structured formats like Parquet, through the same profile-based syntax: - -```sql -CREATE EXTERNAL TABLE pxf_parquet_example ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf:////?PROFILE=s3:parquet&SERVER=s3srvcfg&COMPRESSION_CODEC=snappy') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') - ENCODING 'UTF8'; -``` - -See [PXF profiles](../../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. - -## Writing data - -Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to the object store: - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_write_example ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf:////?PROFILE=s3:parquet&SERVER=s3srvcfg&COMPRESSION_CODEC=snappy') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') - ENCODING 'UTF8'; - -INSERT INTO pxf_write_example SELECT id, created, status FROM some_local_table; -``` - -To query the data, create a separate readable external table at the same location: - -```sql -CREATE EXTERNAL TABLE pxf_read_back ( - id bigint, - created timestamp without time zone, - status integer -) - LOCATION ('pxf:////?PROFILE=s3:parquet&SERVER=s3srvcfg') - FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') - ENCODING 'UTF8'; - -SELECT * FROM pxf_read_back; -``` - -## Overriding credentials per query - -Regardless of how the server authenticates, a user can override its access key and secret key for a single query, using custom options in the `LOCATION` clause: - -```sql -CREATE EXTERNAL TABLE pxf_s3_override (id int, name text) - LOCATION ('pxf:////data.csv?PROFILE=s3:text&SERVER=s3srvcfg&accesskey=&secretkey=') - FORMAT 'CSV' (delimiter=','); -``` - -::: warning Important -Credentials passed this way are visible as part of the external table definition. Avoid this method in a production environment. -::: - -This override works only for plain Amazon S3, not for Azure, Google Cloud Storage, or MinIO servers. diff --git a/docs/pxf/7x/foreign-data-wrapper.md b/docs/pxf/7x/foreign-data-wrapper.md deleted file mode 100644 index d11f718..0000000 --- a/docs/pxf/7x/foreign-data-wrapper.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: Querying data with the PXF foreign data wrapper -navTitle: Using the foreign data wrapper -description: Use the pxf_fdw extension to query PXF-connected data sources through standard Postgres foreign tables instead of external tables. ---- - -Query a PXF-connected data source through standard Postgres foreign tables, using the PXF foreign data wrapper `pxf_fdw`, instead of `pxf://` external tables. `pxf_fdw` gives you `CREATE SERVER`, `CREATE USER MAPPING`, and `CREATE FOREIGN TABLE` statements for the same connectors. Both interfaces read and write through the same server configurations, so switching between them changes only the SQL objects and grants you use, not the underlying data source setup. - -::: info Note -`pxf_fdw` is available only for WHPG 7 and later. -::: - -## Enabling the extension - -`pxf_fdw` runs through the same PXF service as `pxf://` external tables, so complete [Installing PXF](installing.md) first, then [Configuring and starting PXF](configuring.md) through [Initializing and starting PXF](configuring.md#initializing-and-starting-pxf). You don't need the [Creating the PXF extension](configuring.md#creating-the-pxf-extension) step that follows, since `pxf_fdw` is its own extension. - -Like `pxf`, `pxf_fdw` registers per database. `pxf cluster register` already installs the extension files on every host, so you only need to create it in each database where you want to use it: - -```sql -CREATE EXTENSION pxf_fdw; -``` - -## Available foreign data wrappers - -`pxf_fdw` provides one foreign data wrapper per PXF connector: - -| Foreign data wrapper | Connector | -|---|---| -| `hdfs_pxf_fdw` | HDFS | -| `hive_pxf_fdw` | Hive | -| `hbase_pxf_fdw` | HBase | -| `s3_pxf_fdw` | Amazon S3 and other S3-compatible object stores | -| `gs_pxf_fdw` | Google Cloud Storage | -| `abfss_pxf_fdw` | Azure Data Lake Storage Gen2 | -| `wasbs_pxf_fdw` | Azure Blob Storage | -| `jdbc_pxf_fdw` | External SQL databases through JDBC | -| `file_pxf_fdw` | Local file storage | - -## Creating a server - -A foreign data wrapper server points at the same `$PXF_BASE/servers/` directory you'd configure for external tables. See [Object stores](connecting/object-stores/index.md), [Hadoop](connecting/hadoop/index.md), [JDBC](connecting/jdbc.md), or [Network file system](connecting/network-file-system.md) for how to create and populate that directory for your connector. Once it exists, reference it with the `config` option: - -```sql -CREATE SERVER hdfs_hdp - FOREIGN DATA WRAPPER hdfs_pxf_fdw - OPTIONS (config 'hdfssrvcfg'); -``` - -`config` names the server directory, `$PXF_BASE/servers/hdfssrvcfg` in this example, not the FDW server name itself. - -## Creating a user mapping - -A user mapping grants a role permission to use a server. Add connector credentials here instead of in the server's site XML file if you want them to be user-specific rather than shared: - -```sql -CREATE USER MAPPING FOR gpadmin SERVER hdfs_hdp; -``` - -For example, an S3 user mapping can carry that user's own access and secret keys: - -```sql -CREATE USER MAPPING FOR francisco - SERVER s3_hdp - OPTIONS (accesskey 'FRANCISCOS_AWS_ACCESS_KEY', secretkey 'FRANCISCOS_AWS_SECRET_KEY'); -``` - -## Creating and querying a foreign table - -`CREATE FOREIGN TABLE` takes a `resource` option instead of a `LOCATION` clause. `resource` means the same path or identifier you'd use in a `pxf://` location, an HDFS path, an `.` pair, an object store bucket and key, and so on, depending on the connector. An optional `format` option selects the data format, using the same suffixes documented in [PXF profiles](reference/profiles.md), for example `format 'parquet'` for `hdfs:parquet`. Omit `format` for delimited text. - -```sql -CREATE FOREIGN TABLE hdfs_data (id int, name text) - SERVER hdfs_hdp - OPTIONS (resource '/data/pxf_data', format 'csv', delimiter ','); - -SELECT * FROM hdfs_data; -``` - -## Granting access - -Foreign tables use Postgres's standard FDW grants rather than the `GRANT ... ON PROTOCOL pxf` syntax described in [Managing the PXF cluster](administering.md). A role needs `USAGE` on the foreign server, its own user mapping, and the usual table-level grant: - -```sql -GRANT USAGE ON FOREIGN SERVER hdfs_hdp TO ; -GRANT SELECT ON hdfs_data TO ; -``` diff --git a/docs/pxf/7x/index.md b/docs/pxf/7x/index.md deleted file mode 100644 index 3f7d1ff..0000000 --- a/docs/pxf/7x/index.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: PXF for WarehousePG -description: Covers the usage of the WarehousePG Platform Extension Framework (PXF) to access external data sources from WarehousePG. ---- - -The WarehousePG Platform Extension Framework (PXF) gives WarehousePG (WHPG) access to data that lives outside the cluster, in systems like Hadoop, object stores, and external SQL databases. You interact with that external data through external tables, using a readable external table to query it and a writable external table to insert into it, the same way you'd work with any other WHPG table, or through a foreign data wrapper as an alternative. - -A `pxf` extension and a PXF service run on the coordinator, standby coordinator, and every segment host. When a query touches a PXF external table, each segment's PXF service connects to the external source independently and reads or writes its share of the data in parallel, so a PXF query scales with your cluster rather than routing through the coordinator. See [Architecture](overview/architecture.md) for how these pieces fit together. - -## Key features - -- **Broad connector support.** Reach Hadoop, object stores such as Amazon S3, MinIO, Azure Blob Storage, and Google Cloud Storage, SQL databases over JDBC, and network file systems. - -- **Multiple data formats.** Read and write text, CSV, Avro, JSON, RCFile, Parquet, SequenceFile, and ORC, depending on the connector. See [Compatibility](overview/compatibility.md) for supported sources, formats, and versions. - -- **Parallel, segment-based execution.** Each segment's PXF service handles its own portion of the data independently, so performance scales with the number of segments in your cluster. - -- **Full read and write support.** Query external data or insert into it using PXF external tables. - -- **A foreign data wrapper alternative.** Reach the same sources through standard Postgres foreign tables instead of external tables. See [Using the foreign data wrapper](foreign-data-wrapper.md) for that alternative. - -- **Cluster-wide management.** The `pxf cluster` command line tool syncs configuration and starts or stops the PXF service across every host from one place. See [Managing the PXF cluster](administering.md) for details. diff --git a/docs/pxf/7x/installing.md b/docs/pxf/7x/installing.md deleted file mode 100644 index f6c5ef9..0000000 --- a/docs/pxf/7x/installing.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: Installing PXF for WarehousePG -navTitle: Installing -description: Learn how to install the WarehousePG Platform Extension Framework (PXF) package across your WarehousePG cluster. ---- - -Build `whpg-pxf` from source, then distribute the build to every host in your WarehousePG (WHPG) cluster, including the coordinator, standby coordinator, and all segment hosts. - -## Prerequisites - -- **Build host:** GCC, `make`, `unzip`, a cURL development package (`libcurl-devel` on RHEL; use a version from source instead if you're on an older OS whose package only provides cURL 7.19), [Go](https://go.dev/doc/install) 1.21 or later, and JDK 8 specifically, since the server build uses Lombok, which requires JDK 8. Building directly on the WarehousePG coordinator avoids a cross-compile step, since it's already running the same OS as your segment hosts. -- **Every host in the cluster:** JDK 8 or 11 to run. The PXF service is a Java process that runs on every host, not only the one you build on. -- WarehousePG installed on every host in the cluster, running on RHEL 8 or 9. - -## Building from source - -Build PXF on the coordinator, from the `main` branch, which matches WHPG 7.x. - -1. Clone the repository: - - ```bash - git clone https://github.com/warehouse-pg/whpg-pxf.git - cd whpg-pxf - ``` - -1. Set `JAVA_HOME`, and source your WarehousePG environment so the build can find it: - - ```bash - export JAVA_HOME= - source /usr/local/greenplum-db/greenplum_path.sh - ``` - -1. Build PXF: - - ```bash - make - ``` - - This command compiles the PXF service, the `pxf` and `pxf cluster` CLI, and the WHPG extension, and runs PXF's unit tests. - -## Installing on the cluster - -1. Set `GPHOME`, `PXF_HOME`, and `PXF_BASE`, then install the build locally on the coordinator: - - ```bash - export GPHOME=/usr/local/greenplum-db - export PXF_HOME=/usr/local/pxf - export PXF_BASE=$HOME/pxf-base - sudo mkdir -p "$PXF_HOME" - sudo chown -R gpadmin:gpadmin "$GPHOME" "$PXF_HOME" - make install - ``` - - `make install` places the PXF service and CLI under `$PXF_HOME`, and copies the WHPG extension's control, SQL, and shared library files under `$GPHOME` on the coordinator. If you don't set `PXF_BASE`, it defaults to `PXF_HOME`, and a later PXF upgrade can delete your server configurations along with it, so always set it to a separate directory. - -1. Create a file `all_hosts` listing the coordinator, standby coordinator, and all segment hosts: - - ```ini - cdw - scdw - sdw1 - sdw2 - sdw3 - ``` - -1. Copy the built `$PXF_HOME` directory to every other host in the cluster: - - ```bash - gpsync -f all_hosts $PXF_HOME =:$PXF_HOME - ``` - -1. Set ownership of `$PXF_HOME` on every host, so `gpadmin` can run PXF: - - ```bash - gpssh -f all_hosts -u gpadmin -e "sudo chown -R gpadmin:gpadmin $PXF_HOME" - ``` - -1. Install the WHPG extension files under `$GPHOME` on every host, since `make install` only did this on the coordinator: - - ```bash - pxf cluster register - ``` - -## Next steps - -See [Configuring and starting PXF](configuring.md) to set up and start the PXF service across your cluster. If you plan to query through `pxf_fdw` instead of `pxf://` external tables, see [Using the foreign data wrapper](foreign-data-wrapper.md) for its own package requirements. diff --git a/docs/pxf/7x/overview/architecture.md b/docs/pxf/7x/overview/architecture.md deleted file mode 100644 index 953ddf7..0000000 --- a/docs/pxf/7x/overview/architecture.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -title: PXF for WarehousePG architecture -navTitle: Architecture -description: Overview of the PXF architecture and how a query reaches external data. ---- - -The WarehousePG Platform Extension Framework (PXF) architecture consists of an extension registered inside WarehousePG (WHPG), and a Java service that runs alongside WHPG on the coordinator, standby coordinator, and every segment host. You interact with PXF through external tables, using a readable external table to query external data and a writable external table to insert into it. See [Using the foreign data wrapper](../foreign-data-wrapper.md) for an alternative interface. - -![The pxf cluster CLI, run from a shell, reaches a WHPG cluster with a coordinator, cdw, and two segment hosts, sdw1 and sdw2. Each host runs a pxf extension paired with a PXF service. Only the segment hosts' PXF services connect independently to an external data source, such as an object store, Hadoop, Hive, HBase, or a SQL database.](./images/pxf-architecture.svg) - -## Components - -The PXF architecture consists of three main components: the `pxf` extension inside WHPG and the PXF service, both running on every host, and the `pxf cluster` CLI, which you run from a shell to manage the other two. - -### The pxf extension - -You create the `pxf` extension inside each database that needs external table access. It implements the `pxf://` protocol, and its role differs by host when a query scans a [PXF external table](../../../whpg/7x/admin_guide/external/external-tables/about_exttab_7.md). On the coordinator (cdw), it only takes part in planning, working out which filters and columns to push down and estimating cost for the query plan, without contacting the PXF service or the external source. On each segment (sdw), the extension executes its share of that plan, reading the table's connector, server, and path details and forwarding the request to the local PXF service. - -### PXF service - -The PXF service is a long-running Java (JVM) process that runs on the coordinator, standby coordinator, and every WHPG segment host. It listens on a local port (5888 by default) for requests from the `pxf` extension. Each PXF service instance reads its own local server configuration, which must be kept in sync across the cluster. - -### The pxf cluster CLI - -The `pxf` and `pxf cluster` commands are a separate command-line tool for managing the PXF service across every host. Running one of its commands connects to WHPG to look up the cluster's hosts, then reaches each one over SSH to copy configuration or start and stop its local PXF service. This CLI tool doesn't take part in query execution, and it doesn't depend on the PXF service running on the host you invoke it from, so it works even if that host's PXF service is down. - -## Key concepts - -You [connect to external data](../connecting/index.md) through a PXF external table, which brings together a few pieces of configuration to tell PXF what to read and how to reach it. - -- **External table**: The WHPG table you use to read or write external data through PXF. Create a [readable or writable external table](../../../whpg/7x/admin_guide/external/external-tables/about_exttab_7.md) with a `pxf://` location, the same way as any other WHPG external table. -- **Connector**: PXF's implementation for reading from and writing to a specific kind of external data store, such as Hadoop, an S3-compatible object store, or a JDBC-accessible database. See [Object stores](../connecting/object-stores/index.md), [Hadoop](../connecting/hadoop/index.md), [Connecting to SQL databases over JDBC](../connecting/jdbc.md), and [Network file system](../connecting/network-file-system.md) for the connectors PXF supports. -- **Server**: A named configuration for a connector, telling PXF how to reach a specific external source, including details such as the endpoint URL and credentials. A table's `SERVER` setting determines which connection settings its PXF requests use. See [Understanding PXF servers](../connecting/index.md#understanding-pxf-servers). -- **Profile**: A named mapping of a connector to a data format, for example `s3:parquet`. A table's `PROFILE` setting determines which connector and format code handles the request. See [Understanding PXF profiles](../connecting/index.md#understanding-pxf-profiles). - -An external table's `LOCATION` clause specifies both a profile and a server: - -```sql -CREATE EXTERNAL TABLE sales (id int, name text, amount numeric) - LOCATION ('pxf://data/sales.csv?PROFILE=s3:text&SERVER=example') - FORMAT 'CSV' (delimiter=','); -``` - -## Query execution flow - -A query against a PXF external table follows the same path on every segment, from the `pxf` extension out to the external source and back. - -1. A query against a PXF external table reaches the WHPG segments as part of the query's overall execution plan. -2. On each segment, the `pxf` extension forwards the relevant portion of the request to the local PXF service, including the profile, server name, and any parameters from the `LOCATION` clause. -3. Each PXF service independently connects to the external source, such as an object store, Hadoop cluster, or external database, using the server configuration, and reads or writes its portion of the data. -4. For reads, results stream back through the `pxf` extension into the query's execution plan on that segment. For writes, the extension confirms the PXF service accepted the data before the operation completes on that segment. - -Because each segment's PXF service handles its own portion of the data in parallel, a PXF query scales with the number of segments in your cluster rather than routing through a single coordinator process. - -### Filter pushdown - -When you query a PXF external table with a `WHERE` clause, PXF can push part of that clause down to the external data source instead of transferring every row to WHPG and filtering there. WHPG controls pushdown through the [`gp_external_enable_filter_pushdown`](../../../whpg/7x/ref_guide/config_params/guc-list.md#gp_external_enable_filter_pushdown) server configuration parameter. See [Compatibility](compatibility.md) to check which operators and data types support it. PXF falls back to filtering in WHPG for any part of the clause the connector or profile doesn't push down. - -### Column projection - -PXF also applies column projection automatically. With column projection, PXF asks the external data source for only the columns your query selects, rather than transferring every column and discarding the rest in WHPG. See [Compatibility](compatibility.md) to check which connectors and profiles support it. diff --git a/docs/pxf/7x/overview/compatibility.md b/docs/pxf/7x/overview/compatibility.md deleted file mode 100644 index fe4b8a2..0000000 --- a/docs/pxf/7x/overview/compatibility.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: PXF for WarehousePG compatibility -navTitle: Compatibility -description: Supported WarehousePG versions, platforms, and PXF versions for the WarehousePG Platform Extension Framework (PXF). ---- - -This page covers PXF for WHPG 7.x. See [PXF for WHPG 6.x](../../6x/overview/compatibility.md) for that version instead. - -### Platform compatibility - -RHEL 8 or RHEL 9. - -::: info Note -[PXF foreign data wrapper](../foreign-data-wrapper.md) is available starting with WHPG 7.x. It isn't available for WHPG 6.x. -::: - -### System requirements - -- Java 8 or Java 11 on every host in the cluster. - -### External system compatibility - -PXF bundles a client library for each connector. These versions determine which external system versions PXF can reach. - -| Connector | Compatible version | -|---|---| -| HDFS | Hadoop 3.3.x | -| Hive | Hive 2.3.x metastore | -| HBase | HBase 2.6.x, coordinated with ZooKeeper 3.8.x | -| Amazon S3 and other S3-compatible object stores, including MinIO | S3 API, through the AWS SDK for Java 1.12.x | -| Azure Blob Storage and Azure Data Lake Storage Gen2 | Bundled with PXF's Hadoop 3.3.x client | -| Google Cloud Storage | GCS connector 1.9.x | -| SQL databases, through JDBC | Any JDBC 4.x-compliant driver you provide. PXF bundles the PostgreSQL driver, currently 42.7.x, by default. | - -### Operations by connector - -| Connector | Read | Write | Supported formats | -|---|---|---|---| -| HDFS | Yes | Yes | Delimited text, CSV, fixed-width, Parquet, ORC, Avro, JSON, SequenceFile, and Avro in a SequenceFile | -| Object stores (S3, Azure, Google Cloud Storage) | Yes | Yes | Same formats as HDFS | -| Network file system | Yes | Yes | Delimited text, CSV, fixed-width, Parquet, ORC, Avro, and JSON | -| Hive | Yes | No | Whatever format the Hive table itself uses, delimited text, SequenceFile, RCFile, ORC, Parquet, or Avro | -| HBase | Yes | No | Not applicable, HBase's own column-family model | -| SQL databases, through JDBC | Yes | Yes | Not applicable, the external database's native column types | - -### Filter pushdown support - -| Profile | Comparisons (`<`, `>`, `<=`, `>=`, `=`, `<>`) | `LIKE` | `IS [NOT] NULL` | `IN` | `AND` / `OR` | `NOT` | -|---|---|---|---|---|---|---| -| `jdbc` | Yes | Yes⁴ | Yes | No | Yes | Yes | -| `*:parquet` | Yes¹ | No | Yes¹ | Yes¹ | Yes¹ | Yes¹ | -| `*:orc`, except `hive:orc` | Yes¹ ³ | No | Yes¹ ³ | Yes¹ ³ | Yes¹ ³ | Yes¹ ³ | -| `s3:parquet` and `s3:text` with S3 Select | Yes | No | Yes | Yes | Yes | Yes | -| `hbase` | Yes | No | Yes | No | Yes | No | -| `hive:text`, `hive` on a text-format table | Yes² | No | No | No | Yes² | No | -| `hive:rc`, `hive` on an RCFile table | Yes² | No | Yes | Yes | Yes² | Yes | -| `hive:orc`, `hive` on an ORC table | Yes² | No | Yes | Yes | Yes² | Yes | -| `hive` on a Parquet table | Yes² | No | No | Yes | Yes² | Yes | -| `hive:orc` with `VECTORIZE=true` | Yes² | No | No | No | Yes² | No | - -- ¹ PXF applies the predicate itself instead of the remote system. Doing so doesn't reduce the data transferred over the network, but it does reduce how much of that data PXF holds in memory at once while processing the result. -- ² PXF supports partition pruning based on partition keys. -- ³ PXF bases filtering on file-level, stripe-level, and row-level ORC statistics. -- ⁴ The `jdbc` profile supports `LIKE` only for `TEXT` columns. - -PXF doesn't support filter pushdown for any other profile, including `*:avro`, `*:AvroSequenceFile`, `*:SequenceFile`, `*:json`, `*:text`, `*:csv`, `*:fixedwidth`, and `*:text:multi`. - -Supported data types for pushdown include `INT2`, `INT4`, `INT8`, `CHAR`, `TEXT`, `VARCHAR`, `FLOAT`, `BOOL`, and `NUMERIC`, except with `hive` on a Parquet table. `DATE` and `TIMESTAMP` push down only with the JDBC connector, `hive:rc`, `hive:orc`, and `hive` on an RCFile or ORC table. - -### Column projection support - -| Data source | Profiles | -|---|---| -| SQL database, through JDBC | `jdbc` | -| Hive | `hive` on a text, Parquet, RCFile, or ORC table, `hive:rc`, `hive:orc` | -| HDFS | `hdfs:orc`, `hdfs:parquet` | -| Network file system | `file:orc`, `file:parquet` | -| Object stores (S3, Azure, Google Cloud Storage) | `s3:orc`, `s3:parquet`, `wasbs:orc`, `wasbs:parquet`, `abfss:orc`, `abfss:parquet`, `gs:orc`, `gs:parquet` | - -PXF may still turn projection off for a specific query if it can't serialize the query's filter, for example when a `WHERE` clause resolves to a `boolean` column rather than a comparison. diff --git a/docs/pxf/7x/overview/images/pxf-architecture.svg b/docs/pxf/7x/overview/images/pxf-architecture.svg deleted file mode 100644 index 2f13b42..0000000 --- a/docs/pxf/7x/overview/images/pxf-architecture.svg +++ /dev/null @@ -1,71 +0,0 @@ - - PXF architecture: the pxf cluster CLI, the pxf extension, and the PXF service on the coordinator and every WHPG segment host, connecting out to an external data source - The pxf cluster CLI, run from a shell, reaches into a WHPG cluster with a coordinator, cdw, and two segment hosts, sdw1 and sdw2, grouped separately to show that the coordinator plays a different role. Each host runs a pxf extension paired with a PXF service. Each segment host's PXF service connects independently to an external data source such as an object store, Hadoop, Hive, HBase, or a SQL database. - - - - - - - - - - - - - - - - - - pxf cluster CLI - - - - - WHPG cluster - - - cdw - - pxf extension - - - PXF service - - - - - - sdw1 - - pxf extension - - - PXF service - - - sdw2 - - pxf extension - - - PXF service - - - - - - - - External data source - - - Object store - - Hadoop (HDFS) - - Hive / HBase - - SQL database - diff --git a/docs/pxf/7x/overview/index.md b/docs/pxf/7x/overview/index.md deleted file mode 100644 index 2b39da3..0000000 --- a/docs/pxf/7x/overview/index.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: Overview of PXF for WarehousePG -navTitle: Overview -description: Learn what the WarehousePG Platform Extension Framework (PXF) does and which external sources it connects to. ---- - -Data managed by your organization often lives outside your WarehousePG (WHPG) cluster, in systems like Hadoop, object stores, or other SQL databases. The WarehousePG Platform Extension Framework (PXF) gives you access to that external data through built-in connectors that map an external data source to a WHPG table definition. You read and write external data through PXF the same way you query or insert into a WHPG external table, using standard SQL. - -## Supported connectors and formats - -PXF connects to external data through [profiles](../connecting/index.md#understanding-pxf-profiles), which pair a connector, such as an object store or Hadoop, with a data format. Supported connectors include: - -- Object stores compatible with the S3 API, including Amazon S3 and MinIO -- Hadoop (HDFS) -- Hive -- HBase -- SQL databases, through the Java Database Connectivity (JDBC) connector - -Supported data formats include text, CSV, Avro, JSON, RCFile, Parquet, SequenceFile, and ORC, depending on the connector. - -PXF supports both reading external data into WHPG and writing WHPG data out to external storage, through readable and writable external tables respectively. PXF also supports a foreign data wrapper, letting you reach the same connectors through standard Postgres foreign tables instead of external tables. See [Using the foreign data wrapper](../foreign-data-wrapper.md) for that alternative. - -## Explore PXF - -- [Architecture](architecture.md): Understand the PXF components and how a query reaches external data. -- [Compatibility](compatibility.md): Check supported WHPG versions, platforms, and PXF versions. -- [Known issues](known_issues.md): Review current limitations and workarounds. diff --git a/docs/pxf/7x/overview/known_issues.md b/docs/pxf/7x/overview/known_issues.md deleted file mode 100644 index 51a77de..0000000 --- a/docs/pxf/7x/overview/known_issues.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: Known issues -navTitle: Known issues -description: Learn about known issues and configuration gotchas for PXF on WarehousePG. ---- - -This release includes the following known issues and limitations. Where applicable, workarounds are included to help mitigate the impact. These issues are actively tracked and are planned for resolution in a future release. - -- If `fs.s3a.endpoint` in a server's site XML file doesn't include the URI scheme (`http://` or `https://`), PXF fails with a connection error such as `Unsupported or unrecognized SSL message`, and the PXF logs don't show additional detail. Always include the scheme in the endpoint value. See [S3-compatible object stores](../connecting/object-stores/s3.md). - -- Third-party JDBC driver JARs placed in the global `$PXF_BASE/lib` directory can cause classloader conflicts with libraries PXF already ships. Place them in the server-specific directory (`$PXF_BASE/servers//`) instead, so PXF loads them in an isolated classloader. diff --git a/docs/pxf/7x/reference/commands.md b/docs/pxf/7x/reference/commands.md deleted file mode 100644 index c65a2e5..0000000 --- a/docs/pxf/7x/reference/commands.md +++ /dev/null @@ -1,203 +0,0 @@ ---- -title: PXF commands -navTitle: Commands -description: Reference for the pxf and pxf cluster commands. ---- - -PXF provides two related commands. Run `pxf ` to act on the local host only, or `pxf cluster ` to run the same action across every host in the cluster over SSH, using the coordinator's `PXF_HOME`, `PXF_BASE`, and other environment variables. Use the `pxf cluster` form for routine cluster administration, covered in [Managing the PXF cluster](../administering.md). - -## Cluster commands - -Run these from the coordinator host. - -### pxf cluster prepare - -Creates `$PXF_BASE` on every host and copies the default configuration templates from `$PXF_HOME/conf` into it. See [Initializing and starting PXF](../configuring.md#initializing-and-starting-pxf) for usage in context. - -```bash -pxf cluster prepare -``` - -### pxf cluster register - -Installs the PXF extension's control, SQL, and shared library files under `$GPHOME` on every host. Needed because `make install` only places these files under `$GPHOME` on the coordinator. See [Installing on the cluster](../installing.md#installing-on-the-cluster) for usage in context. - -```bash -pxf cluster register -``` - -### pxf cluster start - -Starts the PXF service on every host. See [Starting, stopping, and restarting PXF](../administering.md#starting-stopping-and-restarting-pxf) for usage in context. - -```bash -pxf cluster start -``` - -### pxf cluster stop - -Stops the PXF service on every host. See [Starting, stopping, and restarting PXF](../administering.md#starting-stopping-and-restarting-pxf) for usage in context. - -```bash -pxf cluster stop -``` - -### pxf cluster restart - -Stops, then starts, the PXF service on every host. See [Starting, stopping, and restarting PXF](../administering.md#starting-stopping-and-restarting-pxf) for usage in context. - -```bash -pxf cluster restart -``` - -### pxf cluster status - -Reports whether the PXF service is both running and responsive on every host. See [Monitoring PXF](../administering.md#monitoring-pxf) for usage in context. - -```bash -pxf cluster status -``` - -### pxf cluster sync - -Copies `$PXF_BASE/{conf,lib,servers}` from the coordinator to every other host. See [Applying a configuration change](../administering.md#applying-a-configuration-change) and [Relocating $PXF_BASE](../administering.md#relocating-pxf_base) for usage in context. - -```bash -pxf cluster sync [options] -``` - -| Option | Description | -|--------|-------------| -| `--delete` | Also remove files on the remote hosts that no longer exist on the coordinator. | - -**Example:** - -```bash -pxf cluster sync --delete -``` - -### pxf cluster migrate - -Migrates configuration from an older PXF installation into a new `$PXF_BASE`. - -```bash -pxf cluster migrate -``` - -### pxf cluster init - -(Deprecated) Installs the PXF extension under `$GPHOME`. Use [`pxf cluster register`](#pxf-cluster-register) instead. - -```bash -pxf cluster init -``` - -### pxf cluster reset - -(Deprecated) No operation. - -```bash -pxf cluster reset -``` - -## Local commands - -Run these on a single host to act on that host's PXF instance only. - -### pxf prepare - -Creates `$PXF_BASE` on the local host and copies the default configuration templates into it. Equivalent to [`pxf cluster prepare`](#pxf-cluster-prepare), scoped to this host. - -```bash -pxf prepare -``` - -### pxf register - -Installs the PXF extension under `$GPHOME` on the local host. Useful after a WHPG upgrade. - -```bash -pxf register -``` - -### pxf start - -Starts the local PXF service instance. - -```bash -pxf start -``` - -### pxf stop - -Stops the local PXF service instance. - -```bash -pxf stop -``` - -### pxf restart - -Restarts the local PXF service instance. - -```bash -pxf restart -``` - -### pxf status - -Reports whether the local PXF service instance is running. - -```bash -pxf status -``` - -### pxf sync - -Copies `$PXF_BASE/{conf,lib,servers}` from the local host to another host. Unlike [`pxf cluster sync`](#pxf-cluster-sync), you specify the target host explicitly. - -```bash -pxf sync [options] -``` - -| Option | Description | -|--------|-------------| -| `--delete` | Also remove files on `` that no longer exist locally. | - -**Example:** - -```bash -pxf sync sdw3 --delete -``` - -### pxf migrate - -Migrates configuration from an older PXF installation into a new `$PXF_BASE` on the local host. - -```bash -pxf migrate -``` - -### pxf version - -Shows the PXF server version. - -```bash -pxf version -``` - -### pxf init - -(Deprecated) Installs the PXF extension under `$GPHOME` on the local host. Use [`pxf register`](#pxf-register) instead. - -```bash -pxf init -``` - -### pxf reset - -(Deprecated) No operation. - -```bash -pxf reset -``` diff --git a/docs/pxf/7x/reference/configuration-files.md b/docs/pxf/7x/reference/configuration-files.md deleted file mode 100644 index d058583..0000000 --- a/docs/pxf/7x/reference/configuration-files.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -title: PXF configuration files -navTitle: Configuration files -description: Reference for the PXF configuration files under $PXF_BASE/conf and the properties you can set in each. ---- - -`$PXF_BASE/conf` holds the files you use to configure the PXF service itself, separate from the server configurations you use to connect to external data sources. See [Managing the PXF cluster](../administering.md) for how to apply a change to any of these files. - -## pxf-application.properties - -Configures the PXF Service application. Commonly changed properties: - -| Property | Description | Default | -|---|---|---| -| `server.address` | The address PXF listens on. | `localhost` | -| `pxf.max.threads` | The maximum number of PXF Tomcat threads. | 200 | -| `pxf.connection.timeout` | The connection timeout for read operations. | 5m | -| `pxf.connection.upload-timeout` | The connection timeout for write operations. | 5m | -| `pxf.log.level` | The log level for the PXF service. | info | - -## pxf-env.sh - -Configures the PXF Java virtual machine and runtime paths. Commonly changed properties: - -| Property | Description | Default | -|---|---|---| -| `JAVA_HOME` | The path to the Java JRE home directory. | `/usr/java/default` | -| `PXF_LOGDIR` | The PXF log directory. | `$PXF_BASE/logs` | -| `PXF_RUNDIR` | The PXF run directory. | `$PXF_BASE/run` | -| `PXF_JVM_OPTS` | The default options for the PXF JVM, including heap size. | `-Xmx2g -Xms1g` | -| `PXF_OOM_KILL` | Whether PXF terminates itself automatically on an out-of-memory error. | true | -| `PXF_OOM_DUMP_PATH` | The absolute path to the dump file PXF generates on an out-of-memory error. | No dump file (empty) | -| `PXF_LOADER_PATH` | Additional directories and JARs for PXF to class-load. | (empty) | -| `LD_LIBRARY_PATH` | Additional directories and native libraries for PXF to load. | (empty) | - -You already edited this file to set `JAVA_HOME` in [Configuring and starting PXF](../configuring.md). - -`PXF_HOST` and `PXF_PORT` aren't part of `pxf-env.sh`. They're plain environment variables you set in `~/.bashrc` on a host, and they tell the `pxf` extension where to reach the PXF service, rather than configuring the service itself. Set them only if you're running PXF on a non-default port or on a host other than the one WHPG expects. - -## pxf-log4j2.xml - -Configures PXF and third-party library logging. The root logger, which covers PXF's own code, logs at the level set by `pxf.log.level` in `pxf-application.properties`, `info` by default. Third-party libraries PXF depends on, such as `org.apache.hadoop` and `org.apache.parquet`, already have their own `` entries tuned to a quieter level, so they don't flood the log at `info`. - -Add a `` element inside `` to set a level for a specific package, for example to debug PXF's Hive connector: - -```xml - -``` - -## pxf-profiles.xml - -Defines custom PXF profiles, pairing a name with the Java classes that read or write data for it. PXF's built-in profiles are already defined for you, so you only need this file if you're adding a custom profile: - -```xml - - - my_profile - A custom profile - - com.example.MyFragmenter - com.example.MyAccessor - com.example.MyResolver - - - -``` - -`fragmenter`, `accessor`, and `resolver` are fully qualified names of Java classes implementing PXF's plugin interfaces. Reference the profile from a `LOCATION` clause with `PROFILE=my_profile`. diff --git a/docs/pxf/7x/reference/configuration-templates.md b/docs/pxf/7x/reference/configuration-templates.md deleted file mode 100644 index 8d873df..0000000 --- a/docs/pxf/7x/reference/configuration-templates.md +++ /dev/null @@ -1,158 +0,0 @@ ---- -title: PXF configuration templates -navTitle: Configuration templates -description: Reference for the site XML templates under $PXF_HOME/templates and the properties in each. ---- - -`$PXF_HOME/templates` holds a template site XML file for each connector. Copy the template for your connector into a server directory under `$PXF_BASE/servers/`, then edit it as described on the relevant connector page, for example [Object stores](../connecting/object-stores/index.md) or [Hadoop](../connecting/hadoop/index.md). Reference `` as the `SERVER` parameter in an external table's `LOCATION` clause. - -## core-site.xml - -| Property | Default | Description | -|---|---|---| -| `fs.defaultFS` | `hdfs://0.0.0.0:8020` | The address of the HDFS NameNode. | - -## hdfs-site.xml - -| Property | Default | Description | -|---|---|---| -| `dfs.permissions.enabled` | `true` | Enforces HDFS file permission checks. | -| `dfs.block.access.token.enable` | `true` | Requires a block access token for datanode communication. | -| `dfs.support.append` | `true` | Allows appending to existing files. | -| `hadoop.proxyuser.gpadmin.hosts` | `*` | The hosts `gpadmin` can proxy from when impersonation is enabled. | -| `hadoop.proxyuser.gpadmin.groups` | `*` | The groups `gpadmin` can impersonate when impersonation is enabled. | -| `hadoop.security.authorization` | `true` | Enables service-level authorization checks. | - -## hive-site.xml - -| Property | Default | Description | -|---|---|---| -| `hive.metastore.uris` | `thrift://localhost:9083` | The URI PXF uses to reach the Hive metastore. | -| `hive.metastore.integral.jdo.pushdown` | `true` | Pushes down predicates on integral partition columns to the metastore, improving performance for tables with many partitions. | - -## hbase-site.xml - -| Property | Default | Description | -|---|---|---| -| `hbase.rootdir` | `hdfs://0.0.0.0:8020/hbase` | The location on HDFS where HBase stores its data. | -| `hbase.zookeeper.quorum` | `127.0.0.1` | The ZooKeeper ensemble HBase uses for coordination. | - -## mapred-site.xml - -| Property | Default | Description | -|---|---|---| -| `mapreduce.framework.name` | `local` | The MapReduce execution framework. | -| `mapreduce.input.fileinputformat.input.dir.recursive` | `true` | Reads files in subdirectories recursively. | - -## yarn-site.xml - -| Property | Default | Description | -|---|---|---| -| `yarn.resourcemanager.address` | `0.0.0.0:8032` | The address of the YARN ResourceManager. | - -## s3-site.xml - -For Amazon S3 itself, without the endpoint or path-style properties MinIO and other S3-compatible stores need. See `minio-site.xml` for those. - -| Property | Default | Description | -|---|---|---| -| `fs.s3a.access.key` | Required, no default | Your AWS access key ID. | -| `fs.s3a.secret.key` | Required, no default | Your AWS secret access key. | -| `fs.s3a.fast.upload` | `true` | Uses fast upload buffering for writes. | - -## minio-site.xml - -For MinIO and other S3-compatible object stores. - -| Property | Default | Description | -|---|---|---| -| `fs.s3a.endpoint` | Required, no default | The object store's endpoint URL, including the URI scheme. | -| `fs.s3a.access.key` | Required, no default | The access key for the object store. | -| `fs.s3a.secret.key` | Required, no default | The secret key for the object store. | -| `fs.s3a.fast.upload` | `true` | Uses fast upload buffering for writes. | -| `fs.s3a.path.style.access` | `true` | Uses path-style bucket addressing, required by MinIO and most non-AWS S3-compatible stores. | - -Both `s3-site.xml` and `minio-site.xml` also accept any other Hadoop S3A property, such as `fs.s3a.buffer.dir` for local buffering during upload, `fs.s3a.multipart.size` for the multipart upload threshold, or `fs.s3a.connection.maximum` for connection pooling. See the [S3A section of the Hadoop-AWS module documentation](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#S3A) for the full list of available properties. See [Connecting to an S3-compatible object store](../connecting/object-stores/s3.md) for the properties specific to each authentication method. - -## gs-site.xml - -For Google Cloud Storage. - -| Property | Default | Description | -|---|---|---| -| `google.cloud.auth.service.account.enable` | `true` | Authenticates using a Google Cloud service account. | -| `google.cloud.auth.service.account.json.keyfile` | Required, no default | The path to your Google Cloud service account JSON key file. | -| `fs.AbstractFileSystem.gs.impl` | `com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS` | The file system implementation class for Google Cloud Storage. | - -## abfss-site.xml - -For Azure Data Lake Storage Gen2. - -| Property | Default | Description | -|---|---|---| -| `fs.azure.account.auth.type` | `OAuth` | The authentication type. | -| `fs.azure.account.oauth.provider.type` | `org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider` | The OAuth token provider class. | -| `fs.azure.account.oauth2.client.endpoint` | Required, no default | Your Azure OAuth client endpoint. | -| `fs.azure.account.oauth2.client.id` | Required, no default | Your Azure OAuth client ID. | -| `fs.azure.account.oauth2.client.secret` | Required, no default | Your Azure OAuth client secret. | - -## wasbs-site.xml - -For Azure Blob Storage. - -| Property | Default | Description | -|---|---|---| -| `fs.adl.oauth2.access.token.provider.type` | `ClientCredential` | The OAuth token provider type. | -| `fs.azure.account.key..blob.core.windows.net` | Required, no default | The access key for your Azure Blob Storage account. Replace `` in the property name with your account name. | -| `fs.AbstractFileSystem.wasbs.impl` | `org.apache.hadoop.fs.azure.Wasbs` | The file system implementation class for Azure Blob Storage. | - -## jdbc-site.xml - -### Connection properties - -| Property | Default | Description | -|---|---|---| -| `jdbc.driver` | Required, no default | The JDBC driver class name, for example `org.postgresql.Driver`. | -| `jdbc.url` | Required, no default | The JDBC connection URL, for example `jdbc:postgresql://localhost/postgres`. | -| `jdbc.user` | Required, no default | The username for connecting to the external database. | -| `jdbc.password` | Required, no default | The password for connecting to the external database. | - -### Optional properties - -These properties are commented out in the template. Uncomment and set only the ones you need. - -| Property | Default | Description | -|---|---|---| -| `jdbc.pool.enabled` | `true` | Enables connection pooling. | -| `jdbc.pool.property.maximumPoolSize` | `15` | The maximum number of pooled connections to the external database. | -| `jdbc.pool.property.connectionTimeout` | `30000` | The maximum time, in milliseconds, to wait for a connection from the pool. | -| `jdbc.pool.property.idleTimeout` | `30000` | The maximum time, in milliseconds, a connection sits idle in the pool. | -| `jdbc.pool.property.minimumIdle` | `0` | The minimum number of idle connections to maintain in the pool. | -| `jdbc.pool.qualifier` | None | Creates a separate connection pool for each value of this property. Useful for per-user pools alongside `jdbc.session.property` settings that switch users after connecting. | -| `jdbc.statement.writeSize` | `100` | The number of rows batched together before writing. | -| `jdbc.statement.fetchSize` | `1000` | The number of rows fetched at a time when reading. The MySQL JDBC driver instead streams rows one at a time by default. | -| `jdbc.connection.transactionIsolation` | None | The transaction isolation level: `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ`, or `SERIALIZABLE`. Confirm your database supports the level you choose. | -| `jdbc.session.property.` | None | Sets a session-level variable in the external database before running a query. Add one property per variable, replacing `` in the property name. | -| `jdbc.connection.property.` | None | Sets a JDBC connection property. Add one property per value, replacing `` in the property name. | -| `pxf.service.user.impersonation` | `false` | Impersonates the connecting WHPG user when connecting to the external database. | -| `hadoop.security.authentication` | None | Set to `kerberos` when connecting to a Kerberized Hive metastore. | -| `jdbc.read.prepared-statement` | `false` | Uses a `PreparedStatement` instead of a `Statement` when reading. Some drivers, such as Teradata's FastExport, require this. | - -## pxf-site.xml - -For Kerberos authentication, user impersonation, predicate pushdown, and ORC and Parquet write behavior. This template applies to any server, not only Hadoop connectors. See [Authenticating with Kerberos](../connecting/hadoop/kerberos.md) for the Kerberos and impersonation properties in context. - -| Property | Default | Description | -|---|---|---| -| `pxf.service.kerberos.principal` | `gpadmin/_HOST@EXAMPLE.COM` | The Kerberos principal the PXF service uses. PXF replaces `_HOST` with the host's fully qualified domain name. | -| `pxf.service.kerberos.keytab` | `${pxf.base}/keytabs/pxf.service.keytab` | The path to the Kerberos keytab file. | -| `pxf.service.user.impersonation` | `true` | Impersonates the connecting WHPG user when accessing the external system. | -| `pxf.service.kerberos.constrained-delegation` | `false` | Uses Kerberos constrained delegation (S4U2Self/Proxy) for impersonation instead of requiring PXF to be a Hadoop proxy user. Requires additional Active Directory or IPA server configuration. | -| `pxf.service.kerberos.ticket-renew-window` | `0.8` | The fraction of a Kerberos ticket's lifespan PXF waits through before refreshing it. | -| `pxf.service.user.name` | `${user.name}` | Overrides the proxy user PXF presents to the external system. Set this on unsecured clusters with a proxy user already configured, or to make every WHPG user appear as this one user. | -| `pxf.fs.basePath` | None | The base path PXF uses when building a file URI for a `file:*` profile. Required for any server that accesses data using a `file:*` profile. | -| `pxf.ppd.hive` | `true` | Enables predicate pushdown for Hive profiles. | -| `pxf.sasl.connection.retries` | `5` | The number of retries when a NameNode refuses a SASL connection with a `GSS initiate failed` error. | -| `pxf.orc.write.decimal.overflow` | `round` | The behavior when writing a bare `NUMERIC` column, mapped internally to `DECIMAL(38, 10)`, and a value's integer digit count exceeds what that precision and scale allow: `error` fails the write, `round` rounds the value to fit, or `ignore` writes `NULL`. Define the column as `NUMERIC(, )` instead of a bare `NUMERIC` for more control over when an overflow happens. | -| `pxf.orc.write.timezone.utc` | `true` | Writes ORC timestamp values using UTC instead of the PXF JVM's local timezone. | -| `pxf.parquet.write.decimal.overflow` | `round` | The behavior when writing a bare `NUMERIC` column, mapped internally to `DECIMAL(38, 18)`, and a value's integer digit count exceeds what that precision and scale allow: `error` fails the write, `round` rounds the value to fit, or `ignore` writes `NULL`. Define the column as `NUMERIC(, )` instead of a bare `NUMERIC` for more control over when an overflow happens. | diff --git a/docs/pxf/7x/reference/index.md b/docs/pxf/7x/reference/index.md deleted file mode 100644 index d872f8b..0000000 --- a/docs/pxf/7x/reference/index.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: PXF reference -navTitle: Reference -description: Reference material for PXF configuration files and commands. ---- - -Reference material for administering PXF. - -- [Commands](commands.md): The `pxf` and `pxf cluster` commands. -- [Profiles](profiles.md): The built-in profiles PXF ships for each connector. -- [Configuration files](configuration-files.md): The files under `$PXF_BASE/conf` and the properties you can set in each. -- [Configuration templates](configuration-templates.md): The site XML templates under `$PXF_HOME/templates` and the properties in each. diff --git a/docs/pxf/7x/reference/profiles.md b/docs/pxf/7x/reference/profiles.md deleted file mode 100644 index 939bc7f..0000000 --- a/docs/pxf/7x/reference/profiles.md +++ /dev/null @@ -1,160 +0,0 @@ ---- -title: PXF profiles -navTitle: Profiles -description: Reference for the built-in profiles PXF ships for each connector. ---- - -Specify a profile as the `PROFILE` parameter in an external table's `LOCATION` clause. See [Understanding PXF profiles](../connecting/index.md#understanding-pxf-profiles) for what a profile is. - -## HDFS, object stores, and network file system - -`hdfs`, `s3`, `abfss` (Azure Data Lake Storage), `wasbs` (Azure Blob Storage), `gs` (Google Cloud Storage), and `file` (network file system) all share the same format suffixes. Combine a connector prefix with a format suffix to form a profile name, for example `s3:parquet` or `file:csv`. - -| Format | Profile suffix | Available on | Notes | -|---|---|---|---| -| Delimited text | `:text` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Plain text. | -| CSV | `:csv` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | | -| Multi-line text | `:text:multi` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Supports multi-line records. | -| Fixed-width text | `:fixedwidth` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Reads columns at fixed byte offsets. | -| Parquet | `:parquet` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Columnar, binary, with the schema embedded in the file. | -| ORC | `:orc` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Columnar, binary. | -| Avro | `:avro` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Row-based, binary, paired with a schema. | -| JSON | `:json` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | | -| SequenceFile | `:SequenceFile` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs` | Hadoop's binary key-value container format. | -| Avro in a SequenceFile | `:AvroSequenceFile` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs` | Avro-encoded records stored inside a SequenceFile container. | - -`file:SequenceFile` and `file:AvroSequenceFile` aren't available, since SequenceFile is a Hadoop-specific container format. See [Object stores](../connecting/object-stores/index.md), [Hadoop](../connecting/hadoop/index.md), and [Network file system](../connecting/network-file-system.md) for connecting to each of these sources. - -Only `:text` and `:csv` use `FORMAT 'TEXT'` or `FORMAT 'CSV'`. Every other format needs `FORMAT 'CUSTOM'`, with `FORMATTER='pxfwritable_import'` on a readable external table or `FORMATTER='pxfwritable_export'` on a writable one. - -## Hive - -| Profile | Description | -|---|---| -| `hive` | Reads any Hive-supported storage format through Hive's SerDe layer. | -| `hive:text` | Reads a Hive table stored as text files directly, skipping the SerDe layer. | -| `hive:rc` | Reads a Hive table stored as RCFile directly, skipping the SerDe layer. | -| `hive:orc` | Reads a Hive table stored as ORC directly, skipping the SerDe layer. Supports vectorized reads, see [Hive](../connecting/hadoop/hive.md) for the `VECTORIZE` option. | - -The `hive` profile reads through Hive's SerDe (serializer/deserializer), the class Hive itself uses to parse the table's storage format, so it works with any format Hive supports. A `hive:` profile instead skips the SerDe and reads the underlying files directly, which is faster but only available for the formats listed above. See [Hive](../connecting/hadoop/hive.md) for connecting to a Hive metastore. - -## HBase - -| Profile | Description | -|---|---| -| `hbase` | Reads an HBase table. See [HBase](../connecting/hadoop/hbase.md). | - -## JDBC - -| Profile | Description | -|---|---| -| `jdbc` | Reads and writes data through a JDBC driver. See [Connecting to SQL databases over JDBC](../connecting/jdbc.md). | - -## Examples - -The following examples show the options and syntax specific to each format, using a mix of connectors to illustrate that the same options apply regardless of which connector you pair the format with. - -### Multi-byte delimiters - -The `:text` and `:csv` profiles only support a single-byte delimiter. To read data with a multi-byte or multi-character delimiter, such as a currency symbol, use a `:csv` profile with the `pxfdelimited_import` custom formatter instead: - -```sql -CREATE EXTERNAL TABLE mbyte_delim (id int, city text, country text) - LOCATION ('pxf://data/multibyte_currency?PROFILE=s3:csv&SERVER=s3srvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfdelimited_import', DELIMITER='¤'); -``` - -`DELIMITER` is required and accepts a string up to 32 bytes. Add `QUOTE` and `ESCAPE` if your data quotes or escapes values: - -```sql -CREATE EXTERNAL TABLE mbyte_delim_quoted (id int, city text, country text) - LOCATION ('pxf://data/multibyte_quoted?PROFILE=s3:csv&SERVER=s3srvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfdelimited_import', DELIMITER='¤', QUOTE '"', ESCAPE '\'); -``` - -If your data's line ending isn't a line feed, add `NEWLINE` (`CR` or `CRLF`) to both the formatter options and the `LOCATION` clause. This formatter only supports reading data, not writing it. - -### Avro - -Avro is a compact binary format that pairs each record with a schema describing its fields. A schema looks like this: - -```json -{ - "type": "record", - "name": "User", - "fields": [ - { "name": "id", "type": "int" }, - { "name": "name", "type": "string" } - ] -} -``` - -Create a readable external table with an `:avro` profile, pointing at an existing Avro file: - -```sql -CREATE EXTERNAL TABLE pxf_avro_read (id int, name text, details text[]) - LOCATION ('pxf://data/emp_data.avro?PROFILE=gs:avro&SERVER=gcssrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); - -SELECT * FROM pxf_avro_read; -``` - -If you don't provide a `SCHEMA` option, PXF generates the Avro schema from the external table's column definitions, using each column name as the Avro field name: - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_avro_write (id int, username text, followers text[]) - LOCATION ('pxf://data/pxf_write.avro?PROFILE=gs:avro&SERVER=gcssrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); - -INSERT INTO pxf_avro_write VALUES (1, 'Mickey', ARRAY['Pluto', 'Donald', 'Mini']); -``` - -Create a separate readable external table at the same location to query the data back, since you can't query a writable external table directly. To use your own schema file instead of letting PXF generate one, add the `SCHEMA` option to the `LOCATION` clause: - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_avro_write_with_schema (id int, username text, followers text[]) - LOCATION ('pxf://data/pxf_write.avro?PROFILE=gs:avro&SERVER=gcssrvcfg&SCHEMA=/path/to/avro_schema.avsc') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); -``` - -### JSON - -PXF reads and writes JSON in two layouts. The default expects one JSON object per line, known as JSON Lines. An alternate mode expects a single root object whose value is an array of records, useful when each file already holds a complete JSON document rather than one record per line. - -Use dot notation to project nested fields into columns: - -```sql -CREATE EXTERNAL TABLE pxf_json_read (created_at text, "user.id" int, "user.location" text) - LOCATION ('pxf://data/pxf_examples/events.jsonl?PROFILE=file:json&SERVER=filesrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); - -SELECT * FROM pxf_json_read; -``` - -For a file that holds a single root object with an array of records instead of one object per line, add `IDENTIFIER`, naming the field that marks the start of each record: - -```sql -CREATE EXTERNAL TABLE pxf_json_read_single (created_at text, "user.id" int) - LOCATION ('pxf://data/pxf_examples/events.json?PROFILE=file:json&SERVER=filesrvcfg&IDENTIFIER=created_at') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); -``` - -PXF writes each row as its own JSON Lines record by default: - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_json_write (created_at text, id int, location text) - LOCATION ('pxf://data/pxf_examples/events_out?PROFILE=file:json&SERVER=filesrvcfg') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); - -INSERT INTO pxf_json_write VALUES ('2026-07-21', 1, 'Austin'); -``` - -Add `ROOT` to write a single root object holding an array of records instead, naming the root-level attribute: - -```sql -CREATE WRITABLE EXTERNAL TABLE pxf_json_write_single (created_at text, id int, location text) - LOCATION ('pxf://data/pxf_examples/events_root?PROFILE=file:json&SERVER=filesrvcfg&ROOT=records') - FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); -``` - -Create a separate readable external table at the same location, with a matching `IDENTIFIER`, to query the data back, since you can't query a writable external table directly. PXF only writes scalar columns or one-dimensional arrays, so a writable external table can't include nested or column-projected fields. diff --git a/docs/pxf/7x/release_notes/6.10_rel_notes.md b/docs/pxf/7x/release_notes/6.10_rel_notes.md deleted file mode 100644 index 3c0e325..0000000 --- a/docs/pxf/7x/release_notes/6.10_rel_notes.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: PXF 6.10.2 release notes -navTitle: Version 6.10.2 -description: Release notes for PXF for WarehousePG version 6.10.2. ---- - -## PXF 6.10.2 - -Released: 26 Feb 2026 - -### Enhancements - -- Added `TIMESTAMPTZ` support to the JDBC profile. -- Replaced the `adl` profile with `abfss`, for Azure Data Lake Storage Gen2 support. - -### Bug fixes - -- Fixed the `pxf.control` and `pxf_fdw.control` extension files installing to the PXF package directory instead of `$GPHOME/share/postgresql/extension`, where WHPG expects them. The package now installs both files to the correct location directly, so `pxf cluster register` is no longer required after a default installation. diff --git a/docs/pxf/7x/release_notes/index.md b/docs/pxf/7x/release_notes/index.md deleted file mode 100644 index 4b2eba7..0000000 --- a/docs/pxf/7x/release_notes/index.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: PXF release notes -navTitle: Release notes -description: Release notes provide information on what's new in each release of PXF for WarehousePG. ---- - -PXF for WHPG 7.x hasn't had a release yet. This page will list releases once one ships. See [PXF for WHPG 6.x release notes](../../6x/release_notes/index.md) for the 6.x release history.