diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 57b812a..376b26b 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -34,7 +34,8 @@ export default defineConfig( items: [ { text: '7.x', link: '/docs/7x' }, { text: '6.x', link: '/docs/6x' }, - { text: 'Backup & restore', link: '/whpg-backup/' } + { text: 'Backup & restore', link: '/whpg-backup/' }, + { text: 'PXF 6.x', link: '/pxf/6x/' } ] }, { text: 'GitHub', link: 'https://github.com/warehouse-pg/warehouse-pg' }, @@ -101,6 +102,49 @@ export default defineConfig( { text: "Reference", link: "/whpg-backup/reference/" } ], + }, + { + text: "PXF 6.x", + link: "/pxf/6x/", + collapsed: false, + items: [ + { text: "Release notes", link: "/pxf/6x/release_notes/" }, + { text: "Overview", link: "/pxf/6x/overview/" }, + { text: "Installing", link: "/pxf/6x/installing" }, + { text: "Configuring and starting", link: "/pxf/6x/configuring" }, + { text: "Administering", link: "/pxf/6x/administering" }, + { + text: "Connecting to external data", + link: "/pxf/6x/connecting/", + collapsed: true, + items: [ + { + text: "Object stores", + link: "/pxf/6x/connecting/object-stores/", + collapsed: true, + items: [ + { text: "S3-compatible stores", link: "/pxf/6x/connecting/object-stores/s3" }, + { text: "Azure", link: "/pxf/6x/connecting/object-stores/azure" }, + { text: "Google Cloud Storage", link: "/pxf/6x/connecting/object-stores/gcs" }, + ], + }, + { + text: "Hadoop", + link: "/pxf/6x/connecting/hadoop/", + collapsed: true, + items: [ + { text: "HDFS", link: "/pxf/6x/connecting/hadoop/hdfs" }, + { text: "Hive", link: "/pxf/6x/connecting/hadoop/hive" }, + { text: "HBase", link: "/pxf/6x/connecting/hadoop/hbase" }, + { text: "Authenticating with Kerberos", link: "/pxf/6x/connecting/hadoop/kerberos" }, + ], + }, + { text: "SQL databases over JDBC", link: "/pxf/6x/connecting/jdbc" }, + { text: "Network file system", link: "/pxf/6x/connecting/network-file-system" }, + ], + }, + { text: "Reference", link: "/pxf/6x/reference/" } + ], } ] diff --git a/docs/pxf/6x/administering.md b/docs/pxf/6x/administering.md new file mode 100644 index 0000000..f0726cb --- /dev/null +++ b/docs/pxf/6x/administering.md @@ -0,0 +1,103 @@ +--- +title: Managing the PXF cluster +navTitle: Managing the PXF cluster +description: Learn how to start, stop, and check the status of PXF, and how to apply a configuration change. +--- + +Once PXF is running, you manage it from the coordinator host using the `pxf cluster` commands. + +## Understanding PXF directories and configuration files + +PXF uses two main directories: + +- `$PXF_HOME` is the read-only directory where PXF is installed, for example `/usr/local/pxf`. It holds the PXF service itself, along with the connector site templates under `templates/` that you copy into a server configuration. +- `$PXF_BASE` is the writable runtime directory you create when you [initialize your cluster](configuring.md#initializing-and-starting-pxf), for example `$HOME/pxf-base`. It holds your configuration, including your server configurations under `servers/`. Keeping `$PXF_BASE` separate from `$PXF_HOME` means a PXF upgrade doesn't overwrite your configuration. + +See [Connecting to external data](connecting/index.md) for how PXF servers and profiles work. + +### Relocating $PXF_BASE + +Move `$PXF_BASE` to a new location if you're running low on disk space or reorganizing storage across hosts: + +1. Stop PXF on every host: + + ```bash + pxf cluster stop + ``` + +1. On the coordinator, move `$PXF_BASE` to the new location and update the environment variable, including in `~/.bashrc`: + + ```bash + mv $PXF_BASE /new/path/to/pxf-base + export PXF_BASE=/new/path/to/pxf-base + ``` + +1. Sync the relocated directory out to every segment host, then start PXF: + + ```bash + pxf cluster sync + pxf cluster start + ``` + +1. Once PXF is confirmed running from the new location, clean up the old `$PXF_BASE` directory on each segment host. + +## Starting, stopping, and restarting PXF + +Start, stop, or restart PXF across the whole cluster with a single command, rather than repeating it on every host. Only the `gpadmin` user can run these commands: + +```bash +pxf cluster start +pxf cluster stop +pxf cluster restart +``` + +## Applying a configuration change + +Sync any change to a file under `$PXF_BASE/conf`, or to a server configuration, out to every segment host, then restart PXF. For example, to increase the PXF Java heap size: + +1. Edit `PXF_JVM_OPTS` in `$PXF_BASE/conf/pxf-env.sh` on the coordinator: + + ```bash + export PXF_JVM_OPTS="-Xmx4g -Xms2g" + ``` + +1. Sync the change to every segment host: + + ```bash + pxf cluster sync + ``` + +1. Restart PXF to apply it. Changes made only on the coordinator have no effect until you sync and restart: + + ```bash + pxf cluster restart + ``` + +### Configuring advanced settings + +Adjust settings under `$PXF_BASE/conf` to fit your environment, the same way as any other configuration change: + +- Change the listen address, overall log level, or thread limit by editing `pxf-application.properties`. +- Change the JVM heap size by editing `pxf-env.sh`. +- Set the hostname or port WHPG uses to reach PXF by setting the `PXF_HOST` and `PXF_PORT` environment variables in `~/.bashrc` on the relevant host. This change also requires restarting WHPG itself, not just PXF. +- Fine-tune logging for individual loggers by editing `pxf-log4j2.xml`. +- Define a custom PXF profile in `pxf-profiles.xml`, pairing a profile name with Java classes that implement PXF's plugin interfaces to read or write it. + +See [PXF configuration files](reference/configuration-files.md) for the full list of properties in each file. + +## Monitoring PXF + +`pxf cluster status` confirms PXF is both running and responsive on every host: + +```bash +pxf cluster status +``` + +PXF also exposes HTTP endpoints on each host for monitoring tools, at `http://:5888/`: + +| Endpoint | Description | +|---|---| +| `actuator/health` | The status of the PXF service. | +| `actuator/info` | Build information for the PXF service. | +| `actuator/metrics` | JVM, Tomcat, system, and PXF-specific metrics. | +| `actuator/prometheus` | The same metrics, in a format a Prometheus server can scrape. | diff --git a/docs/pxf/6x/configuring.md b/docs/pxf/6x/configuring.md new file mode 100644 index 0000000..3c9236b --- /dev/null +++ b/docs/pxf/6x/configuring.md @@ -0,0 +1,93 @@ +--- +title: Configuring and starting PXF for WarehousePG +navTitle: Configuring and starting +description: Learn how to configure PXF's runtime environment, initialize it across your cluster, and start the PXF service. +--- + +After you install PXF on every host in your WarehousePG (WHPG) cluster, set up its environment, initialize and start the service, and create the `pxf` extension. + +## Setting environment variables + +Set `$PXF_HOME` to the directory created during installation, so PXF and your shell can find the PXF service and its default configuration templates. PXF also needs a separate, writable runtime directory, [`$PXF_BASE`](administering.md#understanding-pxf-directories-and-configuration-files), where your per-cluster configuration, server definitions, and logs live. Keeping `$PXF_BASE` separate from `$PXF_HOME` means a PXF upgrade doesn't overwrite your configuration. + +On the coordinator, set `PXF_HOME`, `PXF_BASE`, and add the PXF `bin` directory to your `PATH`: + +```bash +export PXF_HOME=/usr/local/pxf +export PXF_BASE=$HOME/pxf-base +export PATH="$PXF_HOME/bin:$PATH" +``` + +Add these lines to `~/.bashrc` on the coordinator so they persist across sessions. + +## Initializing and starting PXF + +Set up PXF's runtime configuration across the cluster and start the service, so PXF is ready to handle queries. Run the following commands from the coordinator host. Each `pxf cluster` command applies the action to every host in the cluster. + +1. Create the runtime configuration directory on every host: + + ```bash + pxf cluster prepare + ``` + + This command creates `$PXF_BASE` on every host and copies the default configuration templates from `$PXF_HOME/conf` into it. If `$PXF_BASE` doesn't already exist, `pxf cluster prepare` creates it for you. + +1. Set `JAVA_HOME` in `$PXF_BASE/conf/pxf-env.sh` on the coordinator: + + ```bash + echo "export JAVA_HOME=$(readlink -f $(which java) | sed 's:/bin/java$::')" >> $PXF_BASE/conf/pxf-env.sh + ``` + +1. Sync the change to every host: + + ```bash + pxf cluster sync + ``` + +1. Start the PXF Java service on every host, listening on port 5888 by default: + + ```bash + pxf cluster start + ``` + +1. Confirm PXF is running on every host: + + ```bash + pxf cluster status + __OUTPUT__ + Checking status of PXF servers on coordinator host and 2 segment hosts... + PXF is running on 3 out of 3 hosts + ``` + +## Creating the PXF extension + +Create the `pxf` extension in each database that needs external table access. `pxf cluster register`, run as the last step of [installing PXF](installing.md#installing-on-the-cluster), already placed the extension's control, SQL, and shared library files under `$GPHOME` on every host, so you don't need to repeat it here. + +1. Connect to the target database and create the extension: + + ```sql + CREATE EXTENSION IF NOT EXISTS pxf; + ``` + + If the extension already exists from a previous PXF version, update it instead: + + ```sql + ALTER EXTENSION pxf UPDATE; + ``` + + Repeat this step in every database where you want to query external tables. + +1. Grant roles access. Only WHPG superusers can use the `pxf` protocol by default, so grant `SELECT` to let a role read external tables through PXF, and `INSERT` to let it write them: + + ```sql + GRANT SELECT ON PROTOCOL pxf TO ; + GRANT INSERT ON PROTOCOL pxf TO ; + ``` + +::: info Note +If you need to drop the `pxf` extension, use `DROP EXTENSION pxf CASCADE` to drop it along with any external table still using the `pxf` protocol. +::: + +## Next steps + +See [Managing the PXF cluster](administering.md) for the PXF configuration files and ongoing cluster management commands, or go straight to configuring a connector, such as [Object stores](connecting/object-stores/index.md) or [Hadoop](connecting/hadoop/index.md), to reach an external data source. diff --git a/docs/pxf/6x/connecting/hadoop/hbase.md b/docs/pxf/6x/connecting/hadoop/hbase.md new file mode 100644 index 0000000..4833318 --- /dev/null +++ b/docs/pxf/6x/connecting/hadoop/hbase.md @@ -0,0 +1,81 @@ +--- +title: Connecting to HBase +navTitle: HBase +description: Configure PXF to read HBase table data and query it from WarehousePG. +--- + +Configure a server for HBase, then query its table data through external tables. PXF only supports reading from HBase, not writing to it. + +## Configuring the server + +Create a server that connects to an HBase cluster. + +::: info Note +If you plan to filter on HBase external tables, copy `$PXF_HOME/share/pxf-hbase-*.jar` to every node in the HBase cluster first, and add its location to `$HBASE_CLASSPATH`. HBase's region servers need this JAR to execute a pushed-down filter. +::: + +1. Create a server directory under `$PXF_BASE/servers`, and copy the `hbase-site.xml` template into it. For example, to configure a server named `hbase_server`: + + ```bash + mkdir $PXF_BASE/servers/hbase_server + cp $PXF_HOME/templates/hbase-site.xml $PXF_BASE/servers/hbase_server + ``` + +1. Edit `hbase-site.xml` with your connection details: + + ```xml + + + + hbase.rootdir + hdfs://:/hbase + + + hbase.zookeeper.quorum + + + + ``` + + Where: + - `hbase.rootdir` is the location on HDFS where HBase stores its data. + - `hbase.zookeeper.quorum` is the ZooKeeper ensemble HBase uses for coordination. + + See [Configuration templates](../../reference/configuration-templates.md#hbase-sitexml) for the full list of `hbase-site.xml` properties. + +1. Sync the change to every segment host, then restart PXF to apply it: + + ```bash + pxf cluster sync + pxf cluster restart + ``` + +## Reading data + +PXF maps each HBase column to a `:` column name. The following example reads an HBase table named `employees`, with `personal` and `job` column families, using the `hbase_server` server: + +```sql +CREATE EXTERNAL TABLE employee_hbase ("personal:name" text, "job:department" text, "job:salary" int) + LOCATION ('pxf://employees?PROFILE=hbase&SERVER=hbase_server') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); + +SELECT * FROM employee_hbase; +``` + +WarehousePG (WHPG) caps column names at 63 characters and requires them to be character-based, while an HBase qualifier name can be longer or contain binary data. When a qualifier doesn't fit those rules, map it indirectly through a lookup table instead. Create a table named `pxflookup` in HBase with a single column family named `mapping`, then add a row per mapped qualifier, using the target HBase table's name as the row key and the WHPG column alias as the qualifier under `mapping`: + +``` +create 'pxflookup', 'mapping' +put 'pxflookup', 'employees', 'mapping:name', 'personal:name' +put 'pxflookup', 'employees', 'mapping:dept', 'job:department' +``` + +Reference the aliases directly as column names when you create the external table: + +```sql +CREATE EXTERNAL TABLE employee_hbase_mapped (name text, dept text) + LOCATION ('pxf://employees?PROFILE=hbase&SERVER=hbase_server') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); +``` + +Add a `recordkey bytea` column to also read the row's key, HBase's unique per-row identifier that isn't stored in any column family. Type it `text` instead of `bytea` if you want to filter on it in a `WHERE` clause and have that filter pushed down to HBase. diff --git a/docs/pxf/6x/connecting/hadoop/hdfs.md b/docs/pxf/6x/connecting/hadoop/hdfs.md new file mode 100644 index 0000000..699f9ef --- /dev/null +++ b/docs/pxf/6x/connecting/hadoop/hdfs.md @@ -0,0 +1,86 @@ +--- +title: Connecting to HDFS +navTitle: HDFS +description: Configure PXF to read data from the Hadoop Distributed File System (HDFS) and query it from WarehousePG. +--- + +Configure a server for the Hadoop Distributed File System (HDFS), then read and write its data through external tables. If you also plan to use Hive, configure HDFS first, since Hive builds on this same connection. + +## Configuring the server + +Create a server that connects to HDFS. + +1. Create a server directory under `$PXF_BASE/servers`, and copy the `core-site.xml` template from `$PXF_HOME/templates` into it. For example, to configure a server named `hdfssrvcfg`: + + ```bash + mkdir -p $PXF_BASE/servers/hdfssrvcfg + cp $PXF_HOME/templates/core-site.xml $PXF_BASE/servers/hdfssrvcfg + ``` + +1. Edit `fs.defaultFS` in that file to point to your HDFS NameNode: + + ```xml + + fs.defaultFS + hdfs://: + + ``` + + See [Configuration templates](../../reference/configuration-templates.md#core-sitexml) for the full list of `core-site.xml` properties. + +1. Sync the change to every segment host, then restart PXF to apply it: + + ```bash + pxf cluster sync + pxf cluster restart + ``` + +## Reading data + +Read data from HDFS by creating a readable external table with the profile for its format and the server you configured. For example, to read a CSV file using the `hdfssrvcfg` server: + +```sql +CREATE EXTERNAL TABLE pxf_hdfs_example (id int, name text, age int) + LOCATION ('pxf:///data.csv?PROFILE=hdfs:csv&SERVER=hdfssrvcfg') + FORMAT 'CSV' (delimiter=','); + +SELECT * FROM pxf_hdfs_example; +``` + +PXF also supports structured formats like Parquet, through the same profile-based syntax: + +```sql +CREATE EXTERNAL TABLE pxf_parquet_read (id int, name text) + LOCATION ('pxf://parquet_data?PROFILE=hdfs:parquet&SERVER=hdfssrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); + +SELECT * FROM pxf_parquet_read; +``` + +See [PXF profiles](../../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. + +## Writing data + +Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to HDFS, then a separate readable external table at the same location to query it back: + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_parquet_write (id int, name text) + LOCATION ('pxf://parquet_data?PROFILE=hdfs:parquet&SERVER=hdfssrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); + +INSERT INTO pxf_parquet_write VALUES (1, 'New York'); +``` + +```sql +CREATE EXTERNAL TABLE pxf_parquet_read_back (id int, name text) + LOCATION ('pxf://parquet_data?PROFILE=hdfs:parquet&SERVER=hdfssrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); + +SELECT * FROM pxf_parquet_read_back; +``` + +The same pattern applies to other connectors, using their own profile prefix, for example `s3:parquet`. See [Object stores](../object-stores/index.md) for an example. + +## Next steps + +With HDFS configured, you're ready to connect to [Hive](hive.md), which builds on this same connection, or [HBase](hbase.md). If your cluster uses Kerberos, see [Authenticating with Kerberos](kerberos.md). diff --git a/docs/pxf/6x/connecting/hadoop/hive.md b/docs/pxf/6x/connecting/hadoop/hive.md new file mode 100644 index 0000000..7a26fa5 --- /dev/null +++ b/docs/pxf/6x/connecting/hadoop/hive.md @@ -0,0 +1,75 @@ +--- +title: Connecting to Hive +navTitle: Hive +description: Configure PXF to read Hive table data and query it from WarehousePG. +--- + +Configure a server for Hive, then query its table data through external tables. PXF only supports reading from Hive, not writing to it. Configure the Hadoop Distributed File System (HDFS) first, since Hive builds on that same connection. + +## Configuring the server + +Hive tables live on HDFS, so PXF reuses the server you already configured for HDFS, for example `hdfssrvcfg` from [HDFS](hdfs.md). Add `hive-site.xml` to that same server directory. + +1. Copy `hive-site.xml` from the templates directory into the server: + + ```bash + cp $PXF_HOME/templates/hive-site.xml $PXF_BASE/servers/hdfssrvcfg + ``` + +1. Edit `hive-site.xml` to point to your Hive metastore: + + ```xml + + + + hive.metastore.uris + thrift://:9083 + + + ``` + + `hive.metastore.uris` is the Thrift URI PXF uses to reach your Hive metastore. See [Configuration templates](../../reference/configuration-templates.md#hive-sitexml) for the full list of `hive-site.xml` properties. + +1. Sync the change to every segment host, then restart PXF to apply it: + + ```bash + pxf cluster sync + pxf cluster restart + ``` + +## Reading data + +Read a Hive table by creating a readable external table with the `hive` profile and the server you configured. For example, to read a table named `employee` in the `hr` Hive database, using the `hdfssrvcfg` server: + +```sql +CREATE EXTERNAL TABLE emp_hive (id int, name text) + LOCATION ('pxf://hr.employee?PROFILE=hive&SERVER=hdfssrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); + +SELECT * FROM emp_hive; +``` + +The path in `LOCATION` is `.`. + +The `hive` profile works with any format Hive supports, reading through Hive's SerDe (serializer/deserializer), the class Hive itself uses to parse that format. For a table stored in text, RCFile, or ORC, switch to the matching `hive:` profile instead for faster reads that skip the SerDe and go straight to the files. + +`hive:orc` also supports `VECTORIZE=true`, to read up to 1024 rows at once. Vectorized reads don't support complex types or the `timestamp` data type: + +```sql +CREATE EXTERNAL TABLE emp_hive_orc_vectorized (id int, name text) + LOCATION ('pxf://hr.employee?PROFILE=hive:orc&SERVER=hdfssrvcfg&VECTORIZE=true') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); +``` + +See [PXF profiles](../../reference/profiles.md#hive) for the full list of Hive profiles. + +## Filtering on partitions + +For a Hive table partitioned on one or more columns, a `WHERE` clause on a partition column lets PXF skip non-matching partitions instead of reading and filtering every row in WarehousePG (WHPG). Which comparisons trigger this depends on the partition column's type: + +- On a string-typed column, `=`, `<>`, `<`, `<=`, `>`, and `>=` all work. +- On an integral-typed column (such as `int`), `=` and `<>` work when you enable `hive.metastore.integral.jdo.pushdown` in `hive-site.xml`. + +You must define partition columns at the end of your `CREATE EXTERNAL TABLE` column list, matching Hive's names and order. + +Hive excludes a row from partition queries if its value doesn't match the partition column's type, commonly a `NULL`. PXF instead surfaces that row under an actual `NULL`, so `IS NULL` queries can return different row counts between Hive and PXF. diff --git a/docs/pxf/6x/connecting/hadoop/index.md b/docs/pxf/6x/connecting/hadoop/index.md new file mode 100644 index 0000000..39075d9 --- /dev/null +++ b/docs/pxf/6x/connecting/hadoop/index.md @@ -0,0 +1,12 @@ +--- +title: Connecting to Hadoop +navTitle: Connecting to Hadoop +description: Configure PXF to connect to a Hadoop cluster and query HDFS, Hive, and HBase data from WarehousePG. +--- + +PXF's Hadoop connectors, the Hadoop Distributed File System (HDFS), Hive, and HBase, share the same underlying cluster configuration. Configure the connection to HDFS first if you also plan to use Hive, since Hive builds on top of it. + +- [HDFS](hdfs.md): Read and write files directly on HDFS. +- [Hive](hive.md): Read Hive table data. +- [HBase](hbase.md): Read HBase table data. +- [Authenticating with Kerberos](kerberos.md): Connect to a Kerberized Hadoop cluster and configure user impersonation. diff --git a/docs/pxf/6x/connecting/hadoop/kerberos.md b/docs/pxf/6x/connecting/hadoop/kerberos.md new file mode 100644 index 0000000..094a3b0 --- /dev/null +++ b/docs/pxf/6x/connecting/hadoop/kerberos.md @@ -0,0 +1,56 @@ +--- +title: Authenticating with Kerberos +navTitle: Authenticating with Kerberos +description: Configure PXF to authenticate with a Kerberized Hadoop cluster and impersonate the connecting WHPG user. +--- + +Authenticate PXF to a Kerberized Hadoop cluster using a Kerberos principal and keytab, instead of connecting as an unauthenticated user. This authentication method applies to any Hadoop connector, the Hadoop Distributed File System (HDFS), Hive, or HBase. + +## Enabling Kerberos authentication + +Add these settings to the same server directory you already configured for HDFS, Hive, or HBase. + +::: info Note +This procedure requires a Kerberos principal and keytab for PXF, generated with `kadmin` (MIT Kerberos) or `ktpass` (Active Directory) on your Kerberos Key Distribution Center (KDC), or provided by your Kerberos administrator. +::: + +1. Copy the `pxf-site.xml` template into your server directory: + + ```bash + cp $PXF_HOME/templates/pxf-site.xml $PXF_BASE/servers/ + ``` + +1. Edit `pxf.service.kerberos.principal` and `pxf.service.kerberos.keytab` with your principal and keytab path: + + ```xml + + pxf.service.kerberos.principal + gpadmin/_HOST@EXAMPLE.COM + + + pxf.service.kerberos.keytab + ${pxf.base}/keytabs/pxf.service.keytab + + ``` + + PXF replaces `_HOST` in the principal with each host's fully qualified domain name. See [Configuration templates](../../reference/configuration-templates.md#pxf-sitexml) for the full list of `pxf-site.xml` properties. + +1. Copy the keytab file to `$PXF_BASE/keytabs` on every segment host, at the same path. `pxf cluster sync` doesn't distribute this directory, so you need to place it there yourself. + +1. Sync the server configuration to every segment host, then restart PXF to apply it: + + ```bash + pxf cluster sync + pxf cluster restart + ``` + +To connect to a Kerberized Hive metastore over JDBC instead, set `hadoop.security.authentication` to `kerberos` in that server's `jdbc-site.xml`. See [Connecting to SQL databases over JDBC](../jdbc.md). + +## Tuning Kerberos parameters + +Edit optional properties the same way as [Enabling Kerberos authentication](#enabling-kerberos-authentication) above, in that same server's `pxf-site.xml`, then sync and restart PXF to apply the change: + +- `pxf.service.user.impersonation`: By default, PXF impersonates the WarehousePG (WHPG) user running the query when it connects to the external system, so the Hadoop cluster sees requests as coming from that user rather than from PXF's own principal. Set this to `false` to disable impersonation and connect as the PXF principal for every query instead. +- `pxf.service.user.name`: Impersonates a specific proxy user rather than the connecting WHPG user. Use this on unsecured clusters where a proxy user is already configured, or to make every WHPG user appear as this one user on the Hadoop side. +- `pxf.service.kerberos.constrained-delegation`: Set to `true` to impersonate using Kerberos constrained delegation (S4U2Self/Proxy) instead of requiring PXF's principal to be a Hadoop proxy user, for clusters that support it. This delegation mode needs additional configuration on your Active Directory or IPA server. +- `pxf.service.kerberos.ticket-renew-window`: The fraction of a Kerberos ticket's lifespan PXF waits through before refreshing it, `0.8` by default. Set it to `0` to request a new ticket on every request. diff --git a/docs/pxf/6x/connecting/index.md b/docs/pxf/6x/connecting/index.md new file mode 100644 index 0000000..250810e --- /dev/null +++ b/docs/pxf/6x/connecting/index.md @@ -0,0 +1,44 @@ +--- +title: Connecting to external data +navTitle: Connecting to external data +description: Learn how PXF servers and profiles work, and how to reference them when you create an external table. +--- + +Reach external data from WarehousePG (WHPG) by pointing a PXF external table at a server, for connection details, and a profile, for the connector and data format to read or write it with. Understand how servers, profiles, and formats work before you configure one for the connector you need, on [Object stores](object-stores/index.md), [Hadoop](hadoop/index.md), [Connecting to SQL databases over JDBC](jdbc.md), or [Network file system](network-file-system.md). + +Every connector, regardless of the external source it reaches, relies on the same two pieces of configuration: a server for connection details and a profile for the data format, such as Parquet, Avro, or JSON. An external table's `LOCATION` clause references both: + +```sql +CREATE EXTERNAL TABLE sales (id int, name text, amount numeric) + LOCATION ('pxf://data/sales.csv?PROFILE=s3:text&SERVER=example') + FORMAT 'CSV' (delimiter=','); +``` + +## Understanding PXF servers + +PXF connects to an external data source through a named server configuration to read or write data. Each server is a directory under `$PXF_BASE/servers` that contains one or more site XML files with the connection settings for that source, such as the endpoint URL and credentials. + +When you initialize your cluster, `pxf cluster prepare` creates an empty `default` server. To read and write from several sources, create multiple servers by adding a separate subdirectory for each one under `$PXF_BASE/servers`, with its own site XML files. `$PXF_HOME/templates` holds a sample site XML file for each connector, ready to copy into your server directory and edit. See [Configuration templates](../reference/configuration-templates.md) for the properties in each. + +::: info Note +If `LOCATION` omits `SERVER`, PXF falls back to `$PXF_BASE/servers/default`. +::: + +## Understanding PXF profiles + +A PXF profile name follows a `:` pattern, for example `hdfs:parquet` or `s3:avro`, and PXF uses it to pick which connector and format code handles the request. + +PXF ships with a built-in profile for each connector and format combination it supports, compiled into the PXF service. See [PXF profiles](../reference/profiles.md) for the full list. + +PXF also supports custom profiles, defined in [`pxf-profiles.xml`](../reference/configuration-files.md#pxf-profilesxml) and backed by your own Java plugin classes. + +The `format` part of a profile works the same way regardless of which connector it's paired with. PXF supports text, CSV, Parquet, ORC, Avro, JSON, and SequenceFile, depending on the connector. See [PXF profiles](../reference/profiles.md) for the full list of formats and which connectors support each. + +## Choosing a connector + +Configure the connector that matches your external data source. + +- [Object stores](object-stores/index.md): Amazon S3, MinIO, Azure Blob Storage, Azure Data Lake Storage, and Google Cloud Storage. +- [Hadoop](hadoop/index.md): HDFS, Hive, and HBase. +- [Connecting to SQL databases over JDBC](jdbc.md): external SQL databases through a JDBC driver. +- [Network file system](network-file-system.md): a shared network file system. diff --git a/docs/pxf/6x/connecting/jdbc.md b/docs/pxf/6x/connecting/jdbc.md new file mode 100644 index 0000000..62fac7c --- /dev/null +++ b/docs/pxf/6x/connecting/jdbc.md @@ -0,0 +1,95 @@ +--- +title: Connecting to a SQL database over JDBC +navTitle: Connecting to SQL databases over JDBC +description: Configure PXF to read and write data in an external SQL database over JDBC, and query it from WarehousePG. +--- + +Configure a server for your SQL database, then read and write its data through external tables over JDBC. + +## Configuring the server + +Create a server that connects to an external SQL database over JDBC. + +1. Create the server directory and copy the `jdbc-site.xml` template into it: + + ```bash + mkdir -p $PXF_BASE/servers/jdbcsrvcfg + cp $PXF_HOME/templates/jdbc-site.xml $PXF_BASE/servers/jdbcsrvcfg + ``` + +1. Edit `jdbc-site.xml` with your connection details. For example, to connect to a WHPG database named `gpadmin` using the bundled PostgreSQL JDBC driver: + + ```xml + + + + jdbc.driver + org.postgresql.Driver + + + jdbc.url + jdbc:postgresql://:/gpadmin + + + jdbc.user + + + + jdbc.password + + + + ``` + + PXF bundles the PostgreSQL and Hive JDBC drivers. For any other database, copy the vendor's JDBC driver JAR to `$PXF_BASE/lib` before applying the change. + + `jdbc-site.xml` also has commented-out properties for connection pooling, batch and fetch sizes, session and connection-level settings, and user impersonation. Uncomment and set only the ones you need. See [Configuration templates](../reference/configuration-templates.md#jdbc-sitexml) for the full list. + +1. Sync the change to every segment host, then restart PXF to apply it: + + ```bash + pxf cluster sync + pxf cluster restart + ``` + +## Reading data + +Read data from the external database by creating a readable external table with the `jdbc` profile. For example, to read the `test` table from the `gpadmin` database configured above: + +```sql +CREATE EXTERNAL TABLE jdbc_read_example (id int) + LOCATION ('pxf://public.test?PROFILE=jdbc&SERVER=jdbcsrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); + +SELECT * FROM jdbc_read_example; +``` + +The path in `LOCATION` is `.`. This same pattern works against any JDBC-compliant database, not only WHPG, once you point `jdbc.url` and `jdbc.driver` at that database and provide its driver JAR. + +The data still lives in the source database. You can load it into a local table with `CREATE TABLE AS`: + +```sql +CREATE TABLE test_local AS SELECT * FROM jdbc_read_example; +``` + +## Writing data + +Create a writable external table with the same `jdbc` profile to insert data into the external database: + +```sql +CREATE WRITABLE EXTERNAL TABLE jdbc_write_example (id int) + LOCATION ('pxf://public.test?PROFILE=jdbc&SERVER=jdbcsrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); + +INSERT INTO jdbc_write_example SELECT * FROM some_local_table; +``` + +To query the data, create a separate readable external table at the same location: + +```sql +CREATE EXTERNAL TABLE pxf_read_back (id int) + LOCATION ('pxf://public.test?PROFILE=jdbc&SERVER=jdbcsrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); + +SELECT * FROM pxf_read_back; +``` diff --git a/docs/pxf/6x/connecting/network-file-system.md b/docs/pxf/6x/connecting/network-file-system.md new file mode 100644 index 0000000..64e10f3 --- /dev/null +++ b/docs/pxf/6x/connecting/network-file-system.md @@ -0,0 +1,99 @@ +--- +title: Connecting to a network file system +navTitle: Connecting to a network file system +description: Configure PXF to read and write data on a network file system, and query it from WarehousePG. +--- + +Configure a server for your network file system, then read and write its data through external tables. + +## Configuring the server + +Create a server that points PXF at a directory mounted at the same path on every WarehousePG (WHPG) host. + +1. Create a server directory under `$PXF_BASE/servers`, and copy the `pxf-site.xml` template from `$PXF_HOME/templates` into it. For example, to configure a server named `nfssrvcfg`: + + ```bash + mkdir -p $PXF_BASE/servers/nfssrvcfg + cp $PXF_HOME/templates/pxf-site.xml $PXF_BASE/servers/nfssrvcfg + ``` + +1. Uncomment `pxf.fs.basePath` and set it to the mount point, and set `pxf.service.user.impersonation` to `false`, since this connector always accesses files as the OS user running PXF rather than the connecting WHPG user: + + ```xml + + pxf.fs.basePath + /mnt/extdata/pxffs + + + pxf.service.user.impersonation + false + + ``` + + The path in `LOCATION` is relative to `pxf.fs.basePath`. See [Configuration templates](../reference/configuration-templates.md#pxf-sitexml) for the full list of `pxf-site.xml` properties. + +1. Sync the change to every segment host, then restart PXF to apply it: + + ```bash + pxf cluster sync + pxf cluster restart + ``` + +## Reading data + +Read a file from the mounted directory by creating a readable external table with the profile for its format and the server you configured. For example, to read a CSV file using the `nfssrvcfg` server: + +```sql +CREATE EXTERNAL TABLE pxf_read_example (id int, name text, age int) + LOCATION ('pxf://data.csv?PROFILE=file:text&SERVER=nfssrvcfg') + FORMAT 'CSV' (delimiter=','); + +SELECT * FROM pxf_read_example; +``` + +PXF also supports structured formats like Parquet, through the same profile-based syntax: + +```sql +CREATE EXTERNAL TABLE pxf_parquet_example ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf://parquet_data/?PROFILE=file:parquet&SERVER=nfssrvcfg') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') + ENCODING 'UTF8'; +``` + +The path in `LOCATION` can't be relative, and can't include the `$` character. See [PXF profiles](../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. + +## Writing data + +Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to the mounted directory: + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_write_example ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf://parquet_data/?PROFILE=file:parquet&SERVER=nfssrvcfg') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') + ENCODING 'UTF8'; + +INSERT INTO pxf_write_example SELECT id, created, status FROM some_local_table; +``` + +To query the data, create a separate readable external table at the same location: + +```sql +CREATE EXTERNAL TABLE pxf_read_back ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf://parquet_data/?PROFILE=file:parquet&SERVER=nfssrvcfg') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') + ENCODING 'UTF8'; + +SELECT * FROM pxf_read_back; +``` diff --git a/docs/pxf/6x/connecting/object-stores/azure.md b/docs/pxf/6x/connecting/object-stores/azure.md new file mode 100644 index 0000000..6bcc84f --- /dev/null +++ b/docs/pxf/6x/connecting/object-stores/azure.md @@ -0,0 +1,175 @@ +--- +title: Connecting to Azure +navTitle: Azure +description: Configure PXF to read and write data in Azure Blob Storage or Azure Data Lake Storage, and query it from WarehousePG. +--- + +Configure a server for Azure Blob Storage or Azure Data Lake Storage, then read and write its data through external tables. + +## Configuring the server + +Create a server directory under `$PXF_BASE/servers`, copy the matching XML template from `$PXF_HOME/templates` into it, and edit it with your connection details. + +### Blob Storage + +```bash +mkdir -p $PXF_BASE/servers/wasbssrvcfg +cp $PXF_HOME/templates/wasbs-site.xml $PXF_BASE/servers/wasbssrvcfg +``` + +Edit `wasbs-site.xml` with your storage account name and key: + +```xml + + + + fs.azure.account.key..blob.core.windows.net + + + +``` + +See [Configuration templates](../../reference/configuration-templates.md#wasbs-sitexml) for the full list of `wasbs-site.xml` properties. + +### Data Lake Storage Gen2 + +```bash +mkdir -p $PXF_BASE/servers/abfsssrvcfg +cp $PXF_HOME/templates/abfss-site.xml $PXF_BASE/servers/abfsssrvcfg +``` + +Edit `abfss-site.xml` with the client ID, secret, and token endpoint from an Azure AD app registration: + +```xml + + + + fs.azure.account.auth.type + OAuth + + + fs.azure.account.oauth.provider.type + org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider + + + fs.azure.account.oauth2.client.endpoint + + + + fs.azure.account.oauth2.client.id + + + + fs.azure.account.oauth2.client.secret + + + +``` + +See [Configuration templates](../../reference/configuration-templates.md#abfss-sitexml) for the full list of `abfss-site.xml` properties. + +Sync the change to every segment host, then restart PXF to apply it: + +```bash +pxf cluster sync +pxf cluster restart +``` + +## Reading data + +Read data from the object store by creating a readable external table with the profile for its format and the server you configured. + +### Blob Storage + +```sql +CREATE EXTERNAL TABLE pxf_read_example (id int, name text, age int) + LOCATION ('pxf://@.blob.core.windows.net//data.csv?PROFILE=wasbs:text&SERVER=wasbssrvcfg') + FORMAT 'CSV' (delimiter=','); + +SELECT * FROM pxf_read_example; +``` + +### Data Lake Storage Gen2 + +```sql +CREATE EXTERNAL TABLE pxf_read_example_adls (id int, name text, age int) + LOCATION ('pxf://.dfs.core.windows.net//data.csv?PROFILE=abfss:text&SERVER=abfsssrvcfg') + FORMAT 'CSV' (delimiter=','); + +SELECT * FROM pxf_read_example_adls; +``` + +PXF also supports structured formats like Parquet, through the same profile-based syntax: + +```sql +CREATE EXTERNAL TABLE pxf_parquet_example ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf://@.blob.core.windows.net//?PROFILE=wasbs:parquet&SERVER=wasbssrvcfg') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') + ENCODING 'UTF8'; +``` + +See [PXF profiles](../../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. + +## Writing data + +Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to Azure, then a separate readable external table at the same location to query it back. + +### Blob Storage + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_write_example ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf://@.blob.core.windows.net//?PROFILE=wasbs:parquet&SERVER=wasbssrvcfg&COMPRESSION_CODEC=snappy') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') + ENCODING 'UTF8'; + +INSERT INTO pxf_write_example SELECT id, created, status FROM some_local_table; +``` + +```sql +CREATE EXTERNAL TABLE pxf_read_back ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf://@.blob.core.windows.net//?PROFILE=wasbs:parquet&SERVER=wasbssrvcfg') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') + ENCODING 'UTF8'; + +SELECT * FROM pxf_read_back; +``` + +### Data Lake Storage Gen2 + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_write_example_adls ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf://.dfs.core.windows.net//?PROFILE=abfss:parquet&SERVER=abfsssrvcfg&COMPRESSION_CODEC=snappy') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') + ENCODING 'UTF8'; + +INSERT INTO pxf_write_example_adls SELECT id, created, status FROM some_local_table; +``` + +```sql +CREATE EXTERNAL TABLE pxf_read_back_adls ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf://.dfs.core.windows.net//?PROFILE=abfss:parquet&SERVER=abfsssrvcfg') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') + ENCODING 'UTF8'; + +SELECT * FROM pxf_read_back_adls; +``` diff --git a/docs/pxf/6x/connecting/object-stores/gcs.md b/docs/pxf/6x/connecting/object-stores/gcs.md new file mode 100644 index 0000000..8583f7c --- /dev/null +++ b/docs/pxf/6x/connecting/object-stores/gcs.md @@ -0,0 +1,106 @@ +--- +title: Connecting to Google Cloud Storage +navTitle: Google Cloud Storage +description: Configure PXF to read and write data in Google Cloud Storage, and query it from WarehousePG. +--- + +Configure a server for Google Cloud Storage, then read and write its data through external tables. + +## Configuring the server + +Create a server that connects to Google Cloud Storage (GCS). + +1. Create a server directory under `$PXF_BASE/servers`, and copy the `gs-site.xml` template from `$PXF_HOME/templates` into it. For example, to configure a server named `gcssrvcfg`: + + ```bash + mkdir -p $PXF_BASE/servers/gcssrvcfg + cp $PXF_HOME/templates/gs-site.xml $PXF_BASE/servers/gcssrvcfg + ``` + +1. Edit `gs-site.xml` with the path to a Google Cloud service account JSON key file, readable by the PXF service on every host: + + ```xml + + + + google.cloud.auth.service.account.enable + true + + + google.cloud.auth.service.account.json.keyfile + + + + ``` + + See [Configuration templates](../../reference/configuration-templates.md#gs-sitexml) for the full list of `gs-site.xml` properties. + +1. Sync the change to every segment host, then restart PXF to apply it: + + ```bash + pxf cluster sync + pxf cluster restart + ``` + +## Reading data + +Read data from the object store by creating a readable external table with the profile for its format and the server you configured. For example, to read a CSV file using the `gcssrvcfg` server: + +```sql +CREATE EXTERNAL TABLE pxf_read_example (id int, name text, age int) + LOCATION ('pxf:////data.csv?PROFILE=gs:text&SERVER=gcssrvcfg') + FORMAT 'CSV' (delimiter=','); + +SELECT * FROM pxf_read_example; +``` + +PXF also supports structured formats like Parquet, through the same profile-based syntax: + +```sql +CREATE EXTERNAL TABLE pxf_parquet_example ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf:////?PROFILE=gs:parquet&SERVER=gcssrvcfg&COMPRESSION_CODEC=snappy') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') + ENCODING 'UTF8'; +``` + +See [PXF profiles](../../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. + +::: info Note +`gs:json` supports reading only, unlike the equivalent JSON profiles for other object stores. +::: + +## Writing data + +Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to GCS: + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_write_example ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf:////?PROFILE=gs:parquet&SERVER=gcssrvcfg&COMPRESSION_CODEC=snappy') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') + ENCODING 'UTF8'; + +INSERT INTO pxf_write_example SELECT id, created, status FROM some_local_table; +``` + +To query the data, create a separate readable external table at the same location: + +```sql +CREATE EXTERNAL TABLE pxf_read_back ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf:////?PROFILE=gs:parquet&SERVER=gcssrvcfg') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') + ENCODING 'UTF8'; + +SELECT * FROM pxf_read_back; +``` diff --git a/docs/pxf/6x/connecting/object-stores/index.md b/docs/pxf/6x/connecting/object-stores/index.md new file mode 100644 index 0000000..88b017b --- /dev/null +++ b/docs/pxf/6x/connecting/object-stores/index.md @@ -0,0 +1,11 @@ +--- +title: Connecting to an object store +navTitle: Connecting to object stores +description: Configure PXF to read and write data in Amazon S3, MinIO, Azure, or Google Cloud Storage, and query it from WarehousePG. +--- + +Configure a server for your object store, then read and write its data through external tables. + +- [S3-compatible object stores](s3.md): Amazon S3 and MinIO. +- [Azure](azure.md): Azure Blob Storage and Azure Data Lake Storage. +- [Google Cloud Storage](gcs.md). diff --git a/docs/pxf/6x/connecting/object-stores/s3.md b/docs/pxf/6x/connecting/object-stores/s3.md new file mode 100644 index 0000000..b180c76 --- /dev/null +++ b/docs/pxf/6x/connecting/object-stores/s3.md @@ -0,0 +1,230 @@ +--- +title: Connecting to an S3-compatible object store +navTitle: S3-compatible stores +description: Configure PXF to read and write data in Amazon S3 or MinIO, and query it from WarehousePG. +--- + +Configure a server for an S3-compatible object store, then read and write its data through external tables. + +## Configuring the server + +Create a server that connects to an S3-compatible object store, such as Amazon S3 or MinIO. + +1. Create a server directory under `$PXF_BASE/servers`, and copy an XML template from `$PXF_HOME/templates` into it. For example, to configure a server named `s3srvcfg` for Amazon S3: + + ```bash + mkdir -p $PXF_BASE/servers/s3srvcfg + cp $PXF_HOME/templates/s3-site.xml $PXF_BASE/servers/s3srvcfg + ``` + +1. Edit the site XML file to authenticate to S3, using whichever of the following methods fits your environment. + + ### Static access keys + + Set your AWS access key and secret key directly: + + ```xml + + + + fs.s3a.access.key + + + + fs.s3a.secret.key + + + + fs.s3a.fast.upload + true + + + ``` + + ### Temporary credentials, through AWS Assume Role + + Obtain temporary credentials yourself, for example through `aws sts assume-role`, then set them along with the session token, and switch the credential provider to one that accepts a session token: + + ```xml + + fs.s3a.access.key + + + + fs.s3a.secret.key + + + + fs.s3a.session.token + + + + fs.s3a.aws.credentials.provider + org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider + + ``` + + ### Named AWS profile + + Point PXF at a profile in `~/.aws/credentials` instead of storing keys in `s3-site.xml`: + + ```xml + + fs.s3a.aws.credentials.provider + com.amazonaws.auth.profile.ProfileCredentialsProvider + + ``` + + Add your keys to `~/.aws/credentials` on every host, under the `default` profile or a named profile: + + ```ini + [test] + aws_access_key_id = + aws_secret_access_key = + ``` + + If you used a named profile instead of `default`, select it by setting `AWS_PROFILE` in `$PXF_BASE/conf/pxf-env.sh`: + + ```bash + export AWS_PROFILE=test + ``` + + ### Credential process + + Use the same `ProfileCredentialsProvider` as the named profile method, but point the profile at a script instead of static keys. The AWS SDK runs the script and reads its stdout as JSON, containing the access key, secret key, session token, and expiration, so this method works with credentials that rotate: + + ```ini + [test] + credential_process = /path/to/credential-script.sh + ``` + + ### IAM role, through an EC2 instance profile + + On EC2, attach an IAM role to the instances running PXF, and omit credentials from `s3-site.xml` entirely. PXF then pulls temporary credentials from the instance metadata service automatically. Set the credential provider explicitly only if you need to rule out another provider taking precedence: + + ```xml + + fs.s3a.aws.credentials.provider + com.amazonaws.auth.InstanceProfileCredentialsProvider + + ``` + + ### Additional tuning properties + + `s3-site.xml` accepts any other Hadoop S3A property, such as `fs.s3a.buffer.dir` for local buffering during upload, `fs.s3a.multipart.size` for the multipart upload threshold, or `fs.s3a.connection.maximum` for connection pooling. See the [S3A section of the Hadoop-AWS module documentation](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#S3A) for the full list of available properties. + + MinIO and other S3-compatible stores also need an endpoint and path-style access, in addition to whichever authentication method you chose above: + + ```xml + + + + fs.s3a.endpoint + http://: + + + fs.s3a.access.key + + + + fs.s3a.secret.key + + + + fs.s3a.fast.upload + true + + + fs.s3a.path.style.access + true + + + ``` + + ::: info Note + Always include the URI scheme (`http://` or `https://`) in `fs.s3a.endpoint`. Omitting it causes connection errors such as `Unsupported or unrecognized SSL message` when PXF tries to reach the endpoint. + ::: + + See [Configuration templates](../../reference/configuration-templates.md#s3-sitexml) for the full list of `s3-site.xml` properties, or [minio-site.xml](../../reference/configuration-templates.md#minio-sitexml) for MinIO and other S3-compatible stores. + +1. Sync the change to every segment host, then restart PXF to apply it: + + ```bash + pxf cluster sync + pxf cluster restart + ``` + +## Reading data + +Read data from the object store by creating a readable external table with the profile for its format and the server you configured. For example, to read a CSV file using the `s3srvcfg` server: + +```sql +CREATE EXTERNAL TABLE pxf_read_example (id int, name text, age int) + LOCATION ('pxf:////data.csv?PROFILE=s3:text&SERVER=s3srvcfg') + FORMAT 'CSV' (delimiter=','); + +SELECT * FROM pxf_read_example; +``` + +PXF also supports structured formats like Parquet, through the same profile-based syntax: + +```sql +CREATE EXTERNAL TABLE pxf_parquet_example ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf:////?PROFILE=s3:parquet&SERVER=s3srvcfg&COMPRESSION_CODEC=snappy') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') + ENCODING 'UTF8'; +``` + +See [PXF profiles](../../reference/profiles.md) for the full list of supported formats, including worked examples of Avro, JSON, and multi-byte delimiters. + +## Writing data + +Create a writable external table with the `pxfwritable_export` formatter to write WHPG data out to the object store: + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_write_example ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf:////?PROFILE=s3:parquet&SERVER=s3srvcfg&COMPRESSION_CODEC=snappy') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_export') + ENCODING 'UTF8'; + +INSERT INTO pxf_write_example SELECT id, created, status FROM some_local_table; +``` + +To query the data, create a separate readable external table at the same location: + +```sql +CREATE EXTERNAL TABLE pxf_read_back ( + id bigint, + created timestamp without time zone, + status integer +) + LOCATION ('pxf:////?PROFILE=s3:parquet&SERVER=s3srvcfg') + FORMAT 'CUSTOM' (FORMATTER = 'pxfwritable_import') + ENCODING 'UTF8'; + +SELECT * FROM pxf_read_back; +``` + +## Overriding credentials per query + +Regardless of how the server authenticates, a user can override its access key and secret key for a single query, using custom options in the `LOCATION` clause: + +```sql +CREATE EXTERNAL TABLE pxf_s3_override (id int, name text) + LOCATION ('pxf:////data.csv?PROFILE=s3:text&SERVER=s3srvcfg&accesskey=&secretkey=') + FORMAT 'CSV' (delimiter=','); +``` + +::: warning Important +Credentials passed this way are visible as part of the external table definition. Avoid this method in a production environment. +::: + +This override works only for plain Amazon S3, not for Azure, Google Cloud Storage, or MinIO servers. diff --git a/docs/pxf/6x/foreign-data-wrapper.md b/docs/pxf/6x/foreign-data-wrapper.md new file mode 100644 index 0000000..d11f718 --- /dev/null +++ b/docs/pxf/6x/foreign-data-wrapper.md @@ -0,0 +1,86 @@ +--- +title: Querying data with the PXF foreign data wrapper +navTitle: Using the foreign data wrapper +description: Use the pxf_fdw extension to query PXF-connected data sources through standard Postgres foreign tables instead of external tables. +--- + +Query a PXF-connected data source through standard Postgres foreign tables, using the PXF foreign data wrapper `pxf_fdw`, instead of `pxf://` external tables. `pxf_fdw` gives you `CREATE SERVER`, `CREATE USER MAPPING`, and `CREATE FOREIGN TABLE` statements for the same connectors. Both interfaces read and write through the same server configurations, so switching between them changes only the SQL objects and grants you use, not the underlying data source setup. + +::: info Note +`pxf_fdw` is available only for WHPG 7 and later. +::: + +## Enabling the extension + +`pxf_fdw` runs through the same PXF service as `pxf://` external tables, so complete [Installing PXF](installing.md) first, then [Configuring and starting PXF](configuring.md) through [Initializing and starting PXF](configuring.md#initializing-and-starting-pxf). You don't need the [Creating the PXF extension](configuring.md#creating-the-pxf-extension) step that follows, since `pxf_fdw` is its own extension. + +Like `pxf`, `pxf_fdw` registers per database. `pxf cluster register` already installs the extension files on every host, so you only need to create it in each database where you want to use it: + +```sql +CREATE EXTENSION pxf_fdw; +``` + +## Available foreign data wrappers + +`pxf_fdw` provides one foreign data wrapper per PXF connector: + +| Foreign data wrapper | Connector | +|---|---| +| `hdfs_pxf_fdw` | HDFS | +| `hive_pxf_fdw` | Hive | +| `hbase_pxf_fdw` | HBase | +| `s3_pxf_fdw` | Amazon S3 and other S3-compatible object stores | +| `gs_pxf_fdw` | Google Cloud Storage | +| `abfss_pxf_fdw` | Azure Data Lake Storage Gen2 | +| `wasbs_pxf_fdw` | Azure Blob Storage | +| `jdbc_pxf_fdw` | External SQL databases through JDBC | +| `file_pxf_fdw` | Local file storage | + +## Creating a server + +A foreign data wrapper server points at the same `$PXF_BASE/servers/` directory you'd configure for external tables. See [Object stores](connecting/object-stores/index.md), [Hadoop](connecting/hadoop/index.md), [JDBC](connecting/jdbc.md), or [Network file system](connecting/network-file-system.md) for how to create and populate that directory for your connector. Once it exists, reference it with the `config` option: + +```sql +CREATE SERVER hdfs_hdp + FOREIGN DATA WRAPPER hdfs_pxf_fdw + OPTIONS (config 'hdfssrvcfg'); +``` + +`config` names the server directory, `$PXF_BASE/servers/hdfssrvcfg` in this example, not the FDW server name itself. + +## Creating a user mapping + +A user mapping grants a role permission to use a server. Add connector credentials here instead of in the server's site XML file if you want them to be user-specific rather than shared: + +```sql +CREATE USER MAPPING FOR gpadmin SERVER hdfs_hdp; +``` + +For example, an S3 user mapping can carry that user's own access and secret keys: + +```sql +CREATE USER MAPPING FOR francisco + SERVER s3_hdp + OPTIONS (accesskey 'FRANCISCOS_AWS_ACCESS_KEY', secretkey 'FRANCISCOS_AWS_SECRET_KEY'); +``` + +## Creating and querying a foreign table + +`CREATE FOREIGN TABLE` takes a `resource` option instead of a `LOCATION` clause. `resource` means the same path or identifier you'd use in a `pxf://` location, an HDFS path, an `.` pair, an object store bucket and key, and so on, depending on the connector. An optional `format` option selects the data format, using the same suffixes documented in [PXF profiles](reference/profiles.md), for example `format 'parquet'` for `hdfs:parquet`. Omit `format` for delimited text. + +```sql +CREATE FOREIGN TABLE hdfs_data (id int, name text) + SERVER hdfs_hdp + OPTIONS (resource '/data/pxf_data', format 'csv', delimiter ','); + +SELECT * FROM hdfs_data; +``` + +## Granting access + +Foreign tables use Postgres's standard FDW grants rather than the `GRANT ... ON PROTOCOL pxf` syntax described in [Managing the PXF cluster](administering.md). A role needs `USAGE` on the foreign server, its own user mapping, and the usual table-level grant: + +```sql +GRANT USAGE ON FOREIGN SERVER hdfs_hdp TO ; +GRANT SELECT ON hdfs_data TO ; +``` diff --git a/docs/pxf/6x/index.md b/docs/pxf/6x/index.md new file mode 100644 index 0000000..8e8b637 --- /dev/null +++ b/docs/pxf/6x/index.md @@ -0,0 +1,20 @@ +--- +title: PXF for WarehousePG +description: Covers the usage of the WarehousePG Platform Extension Framework (PXF) to access external data sources from WarehousePG. +--- + +The WarehousePG Platform Extension Framework (PXF) gives WarehousePG (WHPG) access to data that lives outside the cluster, in systems like Hadoop, object stores, and external SQL databases. You interact with that external data through external tables, using a readable external table to query it and a writable external table to insert into it, the same way you'd work with any other WHPG table. + +A `pxf` extension and a PXF service run on the coordinator, standby coordinator, and every segment host. When a query touches a PXF external table, each segment's PXF service connects to the external source independently and reads or writes its share of the data in parallel, so a PXF query scales with your cluster rather than routing through the coordinator. See [Architecture](overview/architecture.md) for how these pieces fit together. + +## Key features + +- **Broad connector support.** Reach Hadoop, object stores such as Amazon S3, MinIO, Azure Blob Storage, and Google Cloud Storage, SQL databases over JDBC, and network file systems. + +- **Multiple data formats.** Read and write text, CSV, Avro, JSON, RCFile, Parquet, SequenceFile, and ORC, depending on the connector. See [Compatibility](overview/compatibility.md) for supported sources, formats, and versions. + +- **Parallel, segment-based execution.** Each segment's PXF service handles its own portion of the data independently, so performance scales with the number of segments in your cluster. + +- **Full read and write support.** Query external data or insert into it using PXF external tables. + +- **Cluster-wide management.** The `pxf cluster` command line tool syncs configuration and starts or stops the PXF service across every host from one place. See [Managing the PXF cluster](administering.md) for details. diff --git a/docs/pxf/6x/installing.md b/docs/pxf/6x/installing.md new file mode 100644 index 0000000..25190c4 --- /dev/null +++ b/docs/pxf/6x/installing.md @@ -0,0 +1,87 @@ +--- +title: Installing PXF for WarehousePG +navTitle: Installing +description: Learn how to install the WarehousePG Platform Extension Framework (PXF) package across your WarehousePG cluster. +--- + +Build `whpg-pxf` from source, then distribute the build to every host in your WarehousePG (WHPG) cluster, including the coordinator, standby coordinator, and all segment hosts. + +## Prerequisites + +- **Build host:** GCC, `make`, `unzip`, a cURL development package (`libcurl-devel` on RHEL; use a version from source instead if you're on an older OS whose package only provides cURL 7.19), [Go](https://go.dev/doc/install) 1.9 or later, and JDK 8 or 11. Building directly on the WarehousePG coordinator avoids a cross-compile step, since it's already running the same OS as your segment hosts. +- **Every host in the cluster:** JDK 8 or 11 to run. The PXF service is a Java process that runs on every host, not only the one you build on. +- WarehousePG installed on every host in the cluster, running on RHEL 7, 8, or 9. + +## Building from source + +Build PXF on the coordinator, from the `release-6.x` branch, which matches WHPG 6.x. + +1. Clone the repository and check out the `release-6.x` branch: + + ```bash + git clone https://github.com/warehouse-pg/whpg-pxf.git + cd whpg-pxf + git checkout release-6.x + ``` + +1. Set `JAVA_HOME`, and source your WarehousePG environment so the build can find it: + + ```bash + export JAVA_HOME= + source /usr/local/greenplum-db/greenplum_path.sh + ``` + +1. Build PXF: + + ```bash + make + ``` + + This command compiles the PXF service, the `pxf` and `pxf cluster` CLI, and the WHPG extension, and runs PXF's unit tests. + +## Installing on the cluster + +1. Set `GPHOME`, `PXF_HOME`, and `PXF_BASE`, then install the build locally on the coordinator: + + ```bash + export GPHOME=/usr/local/greenplum-db + export PXF_HOME=/usr/local/pxf + export PXF_BASE=$HOME/pxf-base + sudo mkdir -p "$PXF_HOME" + sudo chown -R gpadmin:gpadmin "$GPHOME" "$PXF_HOME" + make install + ``` + + `make install` places the PXF service and CLI under `$PXF_HOME`, and copies the WHPG extension's control, SQL, and shared library files under `$GPHOME` on the coordinator. If you don't set `PXF_BASE`, it defaults to `PXF_HOME`, and a later PXF upgrade can delete your server configurations along with it, so always set it to a separate directory. + +1. Create a file `all_hosts` listing the coordinator, standby coordinator, and all segment hosts: + + ```ini + cdw + scdw + sdw1 + sdw2 + sdw3 + ``` + +1. Copy the built `$PXF_HOME` directory to every other host in the cluster: + + ```bash + gpsync -f all_hosts $PXF_HOME =:$PXF_HOME + ``` + +1. Set ownership of `$PXF_HOME` on every host, so `gpadmin` can run PXF: + + ```bash + gpssh -f all_hosts -u gpadmin -e "sudo chown -R gpadmin:gpadmin $PXF_HOME" + ``` + +1. Install the WHPG extension files under `$GPHOME` on every host, since `make install` only did this on the coordinator: + + ```bash + pxf cluster register + ``` + +## Next steps + +See [Configuring and starting PXF](configuring.md) to set up and start the PXF service across your cluster. diff --git a/docs/pxf/6x/overview/architecture.md b/docs/pxf/6x/overview/architecture.md new file mode 100644 index 0000000..26278b4 --- /dev/null +++ b/docs/pxf/6x/overview/architecture.md @@ -0,0 +1,61 @@ +--- +title: PXF for WarehousePG architecture +navTitle: Architecture +description: Overview of the PXF architecture and how a query reaches external data. +--- + +The WarehousePG Platform Extension Framework (PXF) architecture consists of an extension registered inside WarehousePG (WHPG), and a Java service that runs alongside WHPG on the coordinator, standby coordinator, and every segment host. You interact with PXF through external tables, using a readable external table to query external data and a writable external table to insert into it. + +![The pxf cluster CLI, run from a shell, reaches a WHPG cluster with a coordinator, cdw, and two segment hosts, sdw1 and sdw2. Each host runs a pxf extension paired with a PXF service. Only the segment hosts' PXF services connect independently to an external data source, such as an object store, Hadoop, Hive, HBase, or a SQL database.](./images/pxf-architecture.svg) + +## Components + +The PXF architecture consists of three main components: the `pxf` extension inside WHPG and the PXF service, both running on every host, and the `pxf cluster` CLI, which you run from a shell to manage the other two. + +### The pxf extension + +You create the `pxf` extension inside each database that needs external table access. It implements the `pxf://` protocol, and its role differs by host when a query scans a [PXF external table](../../../whpg/6x/admin_guide/external/external-tables/index.md). On the coordinator (cdw), it only takes part in planning, working out which filters and columns to push down and estimating cost for the query plan, without contacting the PXF service or the external source. On each segment (sdw), the extension executes its share of that plan, reading the table's connector, server, and path details and forwarding the request to the local PXF service. + +### PXF service + +The PXF service is a long-running Java (JVM) process that runs on the coordinator, standby coordinator, and every WHPG segment host. It listens on a local port (5888 by default) for requests from the `pxf` extension. Each PXF service instance reads its own local server configuration, which must be kept in sync across the cluster. + +### The pxf cluster CLI + +The `pxf` and `pxf cluster` commands are a separate command-line tool for managing the PXF service across every host. Running one of its commands connects to WHPG to look up the cluster's hosts, then reaches each one over SSH to copy configuration or start and stop its local PXF service. This CLI tool doesn't take part in query execution, and it doesn't depend on the PXF service running on the host you invoke it from, so it works even if that host's PXF service is down. + +## Key concepts + +You [connect to external data](../connecting/index.md) through a PXF external table, which brings together a few pieces of configuration to tell PXF what to read and how to reach it. + +- **External table**: The WHPG table you use to read or write external data through PXF. Create a [readable or writable external table](../../../whpg/6x/admin_guide/external/external-tables/index.md) with a `pxf://` location, the same way as any other WHPG external table. +- **Connector**: PXF's implementation for reading from and writing to a specific kind of external data store, such as Hadoop, an S3-compatible object store, or a JDBC-accessible database. See [Object stores](../connecting/object-stores/index.md), [Hadoop](../connecting/hadoop/index.md), [Connecting to SQL databases over JDBC](../connecting/jdbc.md), and [Network file system](../connecting/network-file-system.md) for the connectors PXF supports. +- **Server**: A named configuration for a connector, telling PXF how to reach a specific external source, including details such as the endpoint URL and credentials. A table's `SERVER` setting determines which connection settings its PXF requests use. See [Understanding PXF servers](../connecting/index.md#understanding-pxf-servers). +- **Profile**: A named mapping of a connector to a data format, for example `s3:parquet`. A table's `PROFILE` setting determines which connector and format code handles the request. See [Understanding PXF profiles](../connecting/index.md#understanding-pxf-profiles). + +An external table's `LOCATION` clause specifies both a profile and a server: + +```sql +CREATE EXTERNAL TABLE sales (id int, name text, amount numeric) + LOCATION ('pxf://data/sales.csv?PROFILE=s3:text&SERVER=example') + FORMAT 'CSV' (delimiter=','); +``` + +## Query execution flow + +A query against a PXF external table follows the same path on every segment, from the `pxf` extension out to the external source and back. + +1. A query against a PXF external table reaches the WHPG segments as part of the query's overall execution plan. +2. On each segment, the `pxf` extension forwards the relevant portion of the request to the local PXF service, including the profile, server name, and any parameters from the `LOCATION` clause. +3. Each PXF service independently connects to the external source, such as an object store, Hadoop cluster, or external database, using the server configuration, and reads or writes its portion of the data. +4. For reads, results stream back through the `pxf` extension into the query's execution plan on that segment. For writes, the extension confirms the PXF service accepted the data before the operation completes on that segment. + +Because each segment's PXF service handles its own portion of the data in parallel, a PXF query scales with the number of segments in your cluster rather than routing through a single coordinator process. + +### Filter pushdown + +When you query a PXF external table with a `WHERE` clause, PXF can push part of that clause down to the external data source instead of transferring every row to WHPG and filtering there. WHPG controls pushdown through the [`gp_external_enable_filter_pushdown`](../../../whpg/6x/ref_guide/config_params/guc-list.md#gp_external_enable_filter_pushdown) server configuration parameter. See [Compatibility](compatibility.md) to check which operators and data types support it. PXF falls back to filtering in WHPG for any part of the clause the connector or profile doesn't push down. + +### Column projection + +PXF also applies column projection automatically. With column projection, PXF asks the external data source for only the columns your query selects, rather than transferring every column and discarding the rest in WHPG. See [Compatibility](compatibility.md) to check which connectors and profiles support it. diff --git a/docs/pxf/6x/overview/compatibility.md b/docs/pxf/6x/overview/compatibility.md new file mode 100644 index 0000000..2ac268c --- /dev/null +++ b/docs/pxf/6x/overview/compatibility.md @@ -0,0 +1,74 @@ +--- +title: PXF for WarehousePG compatibility +navTitle: Compatibility +description: Supported WarehousePG versions, platforms, and PXF versions for the WarehousePG Platform Extension Framework (PXF). +--- + +### Platform compatibility + +RHEL 7, RHEL 8, or RHEL 9. + +### System requirements + +- Java 8 or Java 11 on every host in the cluster. + +### External system compatibility + +PXF bundles a client library for each connector. These versions determine which external system versions PXF can reach. + +| Connector | Compatible version | +|---|---| +| HDFS | Hadoop 2.10.x | +| Hive | Hive 2.3.x metastore | +| HBase | HBase 1.3.x | +| Amazon S3 and other S3-compatible object stores, including MinIO | S3 API, through the AWS SDK for Java 1.12.x | +| Azure Blob Storage and Azure Data Lake Storage Gen2 | Bundled with PXF's Hadoop 2.10.x client | +| Google Cloud Storage | GCS connector 1.9.x | +| SQL databases, through JDBC | Any JDBC 4.x-compliant driver you provide. PXF bundles the PostgreSQL driver, currently 42.7.x, by default. | + +### Operations by connector + +| Connector | Read | Write | Supported formats | +|---|---|---|---| +| HDFS | Yes | Yes | Delimited text, CSV, fixed-width, Parquet, ORC, Avro, JSON, SequenceFile, and Avro in a SequenceFile | +| Object stores (S3, Azure, Google Cloud Storage) | Yes | Yes | Same formats as HDFS | +| Network file system | Yes | Yes | Delimited text, CSV, fixed-width, Parquet, ORC, Avro, and JSON | +| Hive | Yes | No | Whatever format the Hive table itself uses, delimited text, SequenceFile, RCFile, ORC, Parquet, or Avro | +| HBase | Yes | No | Not applicable, HBase's own column-family model | +| SQL databases, through JDBC | Yes | Yes | Not applicable, the external database's native column types | + +### Filter pushdown support + +| Profile | Comparisons (`<`, `>`, `<=`, `>=`, `=`, `<>`) | `LIKE` | `IS [NOT] NULL` | `IN` | `AND` / `OR` | `NOT` | +|---|---|---|---|---|---|---| +| `jdbc` | Yes | Yes⁴ | Yes | No | Yes | Yes | +| `*:parquet` | Yes¹ | No | Yes¹ | Yes¹ | Yes¹ | Yes¹ | +| `*:orc`, except `hive:orc` | Yes¹ ³ | No | Yes¹ ³ | Yes¹ ³ | Yes¹ ³ | Yes¹ ³ | +| `s3:parquet` and `s3:text` with S3 Select | Yes | No | Yes | Yes | Yes | Yes | +| `hbase` | Yes | No | Yes | No | Yes | No | +| `hive:text`, `hive` on a text-format table | Yes² | No | No | No | Yes² | No | +| `hive:rc`, `hive` on an RCFile table | Yes² | No | Yes | Yes | Yes² | Yes | +| `hive:orc`, `hive` on an ORC table | Yes² | No | Yes | Yes | Yes² | Yes | +| `hive` on a Parquet table | Yes² | No | No | Yes | Yes² | Yes | +| `hive:orc` with `VECTORIZE=true` | Yes² | No | No | No | Yes² | No | + +- ¹ PXF applies the predicate itself instead of the remote system. Doing so doesn't reduce the data transferred over the network, but it does reduce how much of that data PXF holds in memory at once while processing the result. +- ² PXF supports partition pruning based on partition keys. +- ³ PXF bases filtering on file-level, stripe-level, and row-level ORC statistics. +- ⁴ The `jdbc` profile supports `LIKE` only for `TEXT` columns. + +PXF doesn't support filter pushdown for any other profile, including `*:avro`, `*:AvroSequenceFile`, `*:SequenceFile`, `*:json`, `*:text`, `*:csv`, `*:fixedwidth`, and `*:text:multi`. + +Supported data types for pushdown include `INT2`, `INT4`, `INT8`, `CHAR`, `TEXT`, `VARCHAR`, `FLOAT`, `BOOL`, and `NUMERIC`, except with `hive` on a Parquet table. `DATE` and `TIMESTAMP` push down only with the JDBC connector, `hive:rc`, `hive:orc`, and `hive` on an RCFile or ORC table. + +### Column projection support + +| Data source | Profiles | +|---|---| +| SQL database, through JDBC | `jdbc` | +| Hive | `hive` on a text, Parquet, RCFile, or ORC table, `hive:rc`, `hive:orc` | +| HDFS | `hdfs:orc`, `hdfs:parquet` | +| Network file system | `file:orc`, `file:parquet` | +| Object stores (S3, Azure, Google Cloud Storage) | `s3:orc`, `s3:parquet`, `wasbs:orc`, `wasbs:parquet`, `abfss:orc`, `abfss:parquet`, `gs:orc`, `gs:parquet` | + +PXF may still turn projection off for a specific query if it can't serialize the query's filter, for example when a `WHERE` clause resolves to a `boolean` column rather than a comparison. diff --git a/docs/pxf/6x/overview/images/pxf-architecture.svg b/docs/pxf/6x/overview/images/pxf-architecture.svg new file mode 100644 index 0000000..2f13b42 --- /dev/null +++ b/docs/pxf/6x/overview/images/pxf-architecture.svg @@ -0,0 +1,71 @@ + + PXF architecture: the pxf cluster CLI, the pxf extension, and the PXF service on the coordinator and every WHPG segment host, connecting out to an external data source + The pxf cluster CLI, run from a shell, reaches into a WHPG cluster with a coordinator, cdw, and two segment hosts, sdw1 and sdw2, grouped separately to show that the coordinator plays a different role. Each host runs a pxf extension paired with a PXF service. Each segment host's PXF service connects independently to an external data source such as an object store, Hadoop, Hive, HBase, or a SQL database. + + + + + + + + + + + + + + + + + + pxf cluster CLI + + + + + WHPG cluster + + + cdw + + pxf extension + + + PXF service + + + + + + sdw1 + + pxf extension + + + PXF service + + + sdw2 + + pxf extension + + + PXF service + + + + + + + + External data source + + + Object store + + Hadoop (HDFS) + + Hive / HBase + + SQL database + diff --git a/docs/pxf/6x/overview/index.md b/docs/pxf/6x/overview/index.md new file mode 100644 index 0000000..7ebb34c --- /dev/null +++ b/docs/pxf/6x/overview/index.md @@ -0,0 +1,27 @@ +--- +title: Overview of PXF for WarehousePG +navTitle: Overview +description: Learn what the WarehousePG Platform Extension Framework (PXF) does and which external sources it connects to. +--- + +Data managed by your organization often lives outside your WarehousePG (WHPG) cluster, in systems like Hadoop, object stores, or other SQL databases. The WarehousePG Platform Extension Framework (PXF) gives you access to that external data through built-in connectors that map an external data source to a WHPG table definition. You read and write external data through PXF the same way you query or insert into a WHPG external table, using standard SQL. + +## Supported connectors and formats + +PXF connects to external data through [profiles](../connecting/index.md#understanding-pxf-profiles), which pair a connector, such as an object store or Hadoop, with a data format. Supported connectors include: + +- Object stores compatible with the S3 API, including Amazon S3 and MinIO +- Hadoop (HDFS) +- Hive +- HBase +- SQL databases, through the Java Database Connectivity (JDBC) connector + +Supported data formats include text, CSV, Avro, JSON, RCFile, Parquet, SequenceFile, and ORC, depending on the connector. + +PXF supports both reading external data into WHPG and writing WHPG data out to external storage, through readable and writable external tables respectively. + +## Explore PXF + +- [Architecture](architecture.md): Understand the PXF components and how a query reaches external data. +- [Compatibility](compatibility.md): Check supported WHPG versions, platforms, and PXF versions. +- [Known issues](known_issues.md): Review current limitations and workarounds. diff --git a/docs/pxf/6x/overview/known_issues.md b/docs/pxf/6x/overview/known_issues.md new file mode 100644 index 0000000..51a77de --- /dev/null +++ b/docs/pxf/6x/overview/known_issues.md @@ -0,0 +1,11 @@ +--- +title: Known issues +navTitle: Known issues +description: Learn about known issues and configuration gotchas for PXF on WarehousePG. +--- + +This release includes the following known issues and limitations. Where applicable, workarounds are included to help mitigate the impact. These issues are actively tracked and are planned for resolution in a future release. + +- If `fs.s3a.endpoint` in a server's site XML file doesn't include the URI scheme (`http://` or `https://`), PXF fails with a connection error such as `Unsupported or unrecognized SSL message`, and the PXF logs don't show additional detail. Always include the scheme in the endpoint value. See [S3-compatible object stores](../connecting/object-stores/s3.md). + +- Third-party JDBC driver JARs placed in the global `$PXF_BASE/lib` directory can cause classloader conflicts with libraries PXF already ships. Place them in the server-specific directory (`$PXF_BASE/servers//`) instead, so PXF loads them in an isolated classloader. diff --git a/docs/pxf/6x/reference/commands.md b/docs/pxf/6x/reference/commands.md new file mode 100644 index 0000000..c65a2e5 --- /dev/null +++ b/docs/pxf/6x/reference/commands.md @@ -0,0 +1,203 @@ +--- +title: PXF commands +navTitle: Commands +description: Reference for the pxf and pxf cluster commands. +--- + +PXF provides two related commands. Run `pxf ` to act on the local host only, or `pxf cluster ` to run the same action across every host in the cluster over SSH, using the coordinator's `PXF_HOME`, `PXF_BASE`, and other environment variables. Use the `pxf cluster` form for routine cluster administration, covered in [Managing the PXF cluster](../administering.md). + +## Cluster commands + +Run these from the coordinator host. + +### pxf cluster prepare + +Creates `$PXF_BASE` on every host and copies the default configuration templates from `$PXF_HOME/conf` into it. See [Initializing and starting PXF](../configuring.md#initializing-and-starting-pxf) for usage in context. + +```bash +pxf cluster prepare +``` + +### pxf cluster register + +Installs the PXF extension's control, SQL, and shared library files under `$GPHOME` on every host. Needed because `make install` only places these files under `$GPHOME` on the coordinator. See [Installing on the cluster](../installing.md#installing-on-the-cluster) for usage in context. + +```bash +pxf cluster register +``` + +### pxf cluster start + +Starts the PXF service on every host. See [Starting, stopping, and restarting PXF](../administering.md#starting-stopping-and-restarting-pxf) for usage in context. + +```bash +pxf cluster start +``` + +### pxf cluster stop + +Stops the PXF service on every host. See [Starting, stopping, and restarting PXF](../administering.md#starting-stopping-and-restarting-pxf) for usage in context. + +```bash +pxf cluster stop +``` + +### pxf cluster restart + +Stops, then starts, the PXF service on every host. See [Starting, stopping, and restarting PXF](../administering.md#starting-stopping-and-restarting-pxf) for usage in context. + +```bash +pxf cluster restart +``` + +### pxf cluster status + +Reports whether the PXF service is both running and responsive on every host. See [Monitoring PXF](../administering.md#monitoring-pxf) for usage in context. + +```bash +pxf cluster status +``` + +### pxf cluster sync + +Copies `$PXF_BASE/{conf,lib,servers}` from the coordinator to every other host. See [Applying a configuration change](../administering.md#applying-a-configuration-change) and [Relocating $PXF_BASE](../administering.md#relocating-pxf_base) for usage in context. + +```bash +pxf cluster sync [options] +``` + +| Option | Description | +|--------|-------------| +| `--delete` | Also remove files on the remote hosts that no longer exist on the coordinator. | + +**Example:** + +```bash +pxf cluster sync --delete +``` + +### pxf cluster migrate + +Migrates configuration from an older PXF installation into a new `$PXF_BASE`. + +```bash +pxf cluster migrate +``` + +### pxf cluster init + +(Deprecated) Installs the PXF extension under `$GPHOME`. Use [`pxf cluster register`](#pxf-cluster-register) instead. + +```bash +pxf cluster init +``` + +### pxf cluster reset + +(Deprecated) No operation. + +```bash +pxf cluster reset +``` + +## Local commands + +Run these on a single host to act on that host's PXF instance only. + +### pxf prepare + +Creates `$PXF_BASE` on the local host and copies the default configuration templates into it. Equivalent to [`pxf cluster prepare`](#pxf-cluster-prepare), scoped to this host. + +```bash +pxf prepare +``` + +### pxf register + +Installs the PXF extension under `$GPHOME` on the local host. Useful after a WHPG upgrade. + +```bash +pxf register +``` + +### pxf start + +Starts the local PXF service instance. + +```bash +pxf start +``` + +### pxf stop + +Stops the local PXF service instance. + +```bash +pxf stop +``` + +### pxf restart + +Restarts the local PXF service instance. + +```bash +pxf restart +``` + +### pxf status + +Reports whether the local PXF service instance is running. + +```bash +pxf status +``` + +### pxf sync + +Copies `$PXF_BASE/{conf,lib,servers}` from the local host to another host. Unlike [`pxf cluster sync`](#pxf-cluster-sync), you specify the target host explicitly. + +```bash +pxf sync [options] +``` + +| Option | Description | +|--------|-------------| +| `--delete` | Also remove files on `` that no longer exist locally. | + +**Example:** + +```bash +pxf sync sdw3 --delete +``` + +### pxf migrate + +Migrates configuration from an older PXF installation into a new `$PXF_BASE` on the local host. + +```bash +pxf migrate +``` + +### pxf version + +Shows the PXF server version. + +```bash +pxf version +``` + +### pxf init + +(Deprecated) Installs the PXF extension under `$GPHOME` on the local host. Use [`pxf register`](#pxf-register) instead. + +```bash +pxf init +``` + +### pxf reset + +(Deprecated) No operation. + +```bash +pxf reset +``` diff --git a/docs/pxf/6x/reference/configuration-files.md b/docs/pxf/6x/reference/configuration-files.md new file mode 100644 index 0000000..d058583 --- /dev/null +++ b/docs/pxf/6x/reference/configuration-files.md @@ -0,0 +1,68 @@ +--- +title: PXF configuration files +navTitle: Configuration files +description: Reference for the PXF configuration files under $PXF_BASE/conf and the properties you can set in each. +--- + +`$PXF_BASE/conf` holds the files you use to configure the PXF service itself, separate from the server configurations you use to connect to external data sources. See [Managing the PXF cluster](../administering.md) for how to apply a change to any of these files. + +## pxf-application.properties + +Configures the PXF Service application. Commonly changed properties: + +| Property | Description | Default | +|---|---|---| +| `server.address` | The address PXF listens on. | `localhost` | +| `pxf.max.threads` | The maximum number of PXF Tomcat threads. | 200 | +| `pxf.connection.timeout` | The connection timeout for read operations. | 5m | +| `pxf.connection.upload-timeout` | The connection timeout for write operations. | 5m | +| `pxf.log.level` | The log level for the PXF service. | info | + +## pxf-env.sh + +Configures the PXF Java virtual machine and runtime paths. Commonly changed properties: + +| Property | Description | Default | +|---|---|---| +| `JAVA_HOME` | The path to the Java JRE home directory. | `/usr/java/default` | +| `PXF_LOGDIR` | The PXF log directory. | `$PXF_BASE/logs` | +| `PXF_RUNDIR` | The PXF run directory. | `$PXF_BASE/run` | +| `PXF_JVM_OPTS` | The default options for the PXF JVM, including heap size. | `-Xmx2g -Xms1g` | +| `PXF_OOM_KILL` | Whether PXF terminates itself automatically on an out-of-memory error. | true | +| `PXF_OOM_DUMP_PATH` | The absolute path to the dump file PXF generates on an out-of-memory error. | No dump file (empty) | +| `PXF_LOADER_PATH` | Additional directories and JARs for PXF to class-load. | (empty) | +| `LD_LIBRARY_PATH` | Additional directories and native libraries for PXF to load. | (empty) | + +You already edited this file to set `JAVA_HOME` in [Configuring and starting PXF](../configuring.md). + +`PXF_HOST` and `PXF_PORT` aren't part of `pxf-env.sh`. They're plain environment variables you set in `~/.bashrc` on a host, and they tell the `pxf` extension where to reach the PXF service, rather than configuring the service itself. Set them only if you're running PXF on a non-default port or on a host other than the one WHPG expects. + +## pxf-log4j2.xml + +Configures PXF and third-party library logging. The root logger, which covers PXF's own code, logs at the level set by `pxf.log.level` in `pxf-application.properties`, `info` by default. Third-party libraries PXF depends on, such as `org.apache.hadoop` and `org.apache.parquet`, already have their own `` entries tuned to a quieter level, so they don't flood the log at `info`. + +Add a `` element inside `` to set a level for a specific package, for example to debug PXF's Hive connector: + +```xml + +``` + +## pxf-profiles.xml + +Defines custom PXF profiles, pairing a name with the Java classes that read or write data for it. PXF's built-in profiles are already defined for you, so you only need this file if you're adding a custom profile: + +```xml + + + my_profile + A custom profile + + com.example.MyFragmenter + com.example.MyAccessor + com.example.MyResolver + + + +``` + +`fragmenter`, `accessor`, and `resolver` are fully qualified names of Java classes implementing PXF's plugin interfaces. Reference the profile from a `LOCATION` clause with `PROFILE=my_profile`. diff --git a/docs/pxf/6x/reference/configuration-templates.md b/docs/pxf/6x/reference/configuration-templates.md new file mode 100644 index 0000000..8d873df --- /dev/null +++ b/docs/pxf/6x/reference/configuration-templates.md @@ -0,0 +1,158 @@ +--- +title: PXF configuration templates +navTitle: Configuration templates +description: Reference for the site XML templates under $PXF_HOME/templates and the properties in each. +--- + +`$PXF_HOME/templates` holds a template site XML file for each connector. Copy the template for your connector into a server directory under `$PXF_BASE/servers/`, then edit it as described on the relevant connector page, for example [Object stores](../connecting/object-stores/index.md) or [Hadoop](../connecting/hadoop/index.md). Reference `` as the `SERVER` parameter in an external table's `LOCATION` clause. + +## core-site.xml + +| Property | Default | Description | +|---|---|---| +| `fs.defaultFS` | `hdfs://0.0.0.0:8020` | The address of the HDFS NameNode. | + +## hdfs-site.xml + +| Property | Default | Description | +|---|---|---| +| `dfs.permissions.enabled` | `true` | Enforces HDFS file permission checks. | +| `dfs.block.access.token.enable` | `true` | Requires a block access token for datanode communication. | +| `dfs.support.append` | `true` | Allows appending to existing files. | +| `hadoop.proxyuser.gpadmin.hosts` | `*` | The hosts `gpadmin` can proxy from when impersonation is enabled. | +| `hadoop.proxyuser.gpadmin.groups` | `*` | The groups `gpadmin` can impersonate when impersonation is enabled. | +| `hadoop.security.authorization` | `true` | Enables service-level authorization checks. | + +## hive-site.xml + +| Property | Default | Description | +|---|---|---| +| `hive.metastore.uris` | `thrift://localhost:9083` | The URI PXF uses to reach the Hive metastore. | +| `hive.metastore.integral.jdo.pushdown` | `true` | Pushes down predicates on integral partition columns to the metastore, improving performance for tables with many partitions. | + +## hbase-site.xml + +| Property | Default | Description | +|---|---|---| +| `hbase.rootdir` | `hdfs://0.0.0.0:8020/hbase` | The location on HDFS where HBase stores its data. | +| `hbase.zookeeper.quorum` | `127.0.0.1` | The ZooKeeper ensemble HBase uses for coordination. | + +## mapred-site.xml + +| Property | Default | Description | +|---|---|---| +| `mapreduce.framework.name` | `local` | The MapReduce execution framework. | +| `mapreduce.input.fileinputformat.input.dir.recursive` | `true` | Reads files in subdirectories recursively. | + +## yarn-site.xml + +| Property | Default | Description | +|---|---|---| +| `yarn.resourcemanager.address` | `0.0.0.0:8032` | The address of the YARN ResourceManager. | + +## s3-site.xml + +For Amazon S3 itself, without the endpoint or path-style properties MinIO and other S3-compatible stores need. See `minio-site.xml` for those. + +| Property | Default | Description | +|---|---|---| +| `fs.s3a.access.key` | Required, no default | Your AWS access key ID. | +| `fs.s3a.secret.key` | Required, no default | Your AWS secret access key. | +| `fs.s3a.fast.upload` | `true` | Uses fast upload buffering for writes. | + +## minio-site.xml + +For MinIO and other S3-compatible object stores. + +| Property | Default | Description | +|---|---|---| +| `fs.s3a.endpoint` | Required, no default | The object store's endpoint URL, including the URI scheme. | +| `fs.s3a.access.key` | Required, no default | The access key for the object store. | +| `fs.s3a.secret.key` | Required, no default | The secret key for the object store. | +| `fs.s3a.fast.upload` | `true` | Uses fast upload buffering for writes. | +| `fs.s3a.path.style.access` | `true` | Uses path-style bucket addressing, required by MinIO and most non-AWS S3-compatible stores. | + +Both `s3-site.xml` and `minio-site.xml` also accept any other Hadoop S3A property, such as `fs.s3a.buffer.dir` for local buffering during upload, `fs.s3a.multipart.size` for the multipart upload threshold, or `fs.s3a.connection.maximum` for connection pooling. See the [S3A section of the Hadoop-AWS module documentation](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#S3A) for the full list of available properties. See [Connecting to an S3-compatible object store](../connecting/object-stores/s3.md) for the properties specific to each authentication method. + +## gs-site.xml + +For Google Cloud Storage. + +| Property | Default | Description | +|---|---|---| +| `google.cloud.auth.service.account.enable` | `true` | Authenticates using a Google Cloud service account. | +| `google.cloud.auth.service.account.json.keyfile` | Required, no default | The path to your Google Cloud service account JSON key file. | +| `fs.AbstractFileSystem.gs.impl` | `com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS` | The file system implementation class for Google Cloud Storage. | + +## abfss-site.xml + +For Azure Data Lake Storage Gen2. + +| Property | Default | Description | +|---|---|---| +| `fs.azure.account.auth.type` | `OAuth` | The authentication type. | +| `fs.azure.account.oauth.provider.type` | `org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider` | The OAuth token provider class. | +| `fs.azure.account.oauth2.client.endpoint` | Required, no default | Your Azure OAuth client endpoint. | +| `fs.azure.account.oauth2.client.id` | Required, no default | Your Azure OAuth client ID. | +| `fs.azure.account.oauth2.client.secret` | Required, no default | Your Azure OAuth client secret. | + +## wasbs-site.xml + +For Azure Blob Storage. + +| Property | Default | Description | +|---|---|---| +| `fs.adl.oauth2.access.token.provider.type` | `ClientCredential` | The OAuth token provider type. | +| `fs.azure.account.key..blob.core.windows.net` | Required, no default | The access key for your Azure Blob Storage account. Replace `` in the property name with your account name. | +| `fs.AbstractFileSystem.wasbs.impl` | `org.apache.hadoop.fs.azure.Wasbs` | The file system implementation class for Azure Blob Storage. | + +## jdbc-site.xml + +### Connection properties + +| Property | Default | Description | +|---|---|---| +| `jdbc.driver` | Required, no default | The JDBC driver class name, for example `org.postgresql.Driver`. | +| `jdbc.url` | Required, no default | The JDBC connection URL, for example `jdbc:postgresql://localhost/postgres`. | +| `jdbc.user` | Required, no default | The username for connecting to the external database. | +| `jdbc.password` | Required, no default | The password for connecting to the external database. | + +### Optional properties + +These properties are commented out in the template. Uncomment and set only the ones you need. + +| Property | Default | Description | +|---|---|---| +| `jdbc.pool.enabled` | `true` | Enables connection pooling. | +| `jdbc.pool.property.maximumPoolSize` | `15` | The maximum number of pooled connections to the external database. | +| `jdbc.pool.property.connectionTimeout` | `30000` | The maximum time, in milliseconds, to wait for a connection from the pool. | +| `jdbc.pool.property.idleTimeout` | `30000` | The maximum time, in milliseconds, a connection sits idle in the pool. | +| `jdbc.pool.property.minimumIdle` | `0` | The minimum number of idle connections to maintain in the pool. | +| `jdbc.pool.qualifier` | None | Creates a separate connection pool for each value of this property. Useful for per-user pools alongside `jdbc.session.property` settings that switch users after connecting. | +| `jdbc.statement.writeSize` | `100` | The number of rows batched together before writing. | +| `jdbc.statement.fetchSize` | `1000` | The number of rows fetched at a time when reading. The MySQL JDBC driver instead streams rows one at a time by default. | +| `jdbc.connection.transactionIsolation` | None | The transaction isolation level: `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ`, or `SERIALIZABLE`. Confirm your database supports the level you choose. | +| `jdbc.session.property.` | None | Sets a session-level variable in the external database before running a query. Add one property per variable, replacing `` in the property name. | +| `jdbc.connection.property.` | None | Sets a JDBC connection property. Add one property per value, replacing `` in the property name. | +| `pxf.service.user.impersonation` | `false` | Impersonates the connecting WHPG user when connecting to the external database. | +| `hadoop.security.authentication` | None | Set to `kerberos` when connecting to a Kerberized Hive metastore. | +| `jdbc.read.prepared-statement` | `false` | Uses a `PreparedStatement` instead of a `Statement` when reading. Some drivers, such as Teradata's FastExport, require this. | + +## pxf-site.xml + +For Kerberos authentication, user impersonation, predicate pushdown, and ORC and Parquet write behavior. This template applies to any server, not only Hadoop connectors. See [Authenticating with Kerberos](../connecting/hadoop/kerberos.md) for the Kerberos and impersonation properties in context. + +| Property | Default | Description | +|---|---|---| +| `pxf.service.kerberos.principal` | `gpadmin/_HOST@EXAMPLE.COM` | The Kerberos principal the PXF service uses. PXF replaces `_HOST` with the host's fully qualified domain name. | +| `pxf.service.kerberos.keytab` | `${pxf.base}/keytabs/pxf.service.keytab` | The path to the Kerberos keytab file. | +| `pxf.service.user.impersonation` | `true` | Impersonates the connecting WHPG user when accessing the external system. | +| `pxf.service.kerberos.constrained-delegation` | `false` | Uses Kerberos constrained delegation (S4U2Self/Proxy) for impersonation instead of requiring PXF to be a Hadoop proxy user. Requires additional Active Directory or IPA server configuration. | +| `pxf.service.kerberos.ticket-renew-window` | `0.8` | The fraction of a Kerberos ticket's lifespan PXF waits through before refreshing it. | +| `pxf.service.user.name` | `${user.name}` | Overrides the proxy user PXF presents to the external system. Set this on unsecured clusters with a proxy user already configured, or to make every WHPG user appear as this one user. | +| `pxf.fs.basePath` | None | The base path PXF uses when building a file URI for a `file:*` profile. Required for any server that accesses data using a `file:*` profile. | +| `pxf.ppd.hive` | `true` | Enables predicate pushdown for Hive profiles. | +| `pxf.sasl.connection.retries` | `5` | The number of retries when a NameNode refuses a SASL connection with a `GSS initiate failed` error. | +| `pxf.orc.write.decimal.overflow` | `round` | The behavior when writing a bare `NUMERIC` column, mapped internally to `DECIMAL(38, 10)`, and a value's integer digit count exceeds what that precision and scale allow: `error` fails the write, `round` rounds the value to fit, or `ignore` writes `NULL`. Define the column as `NUMERIC(, )` instead of a bare `NUMERIC` for more control over when an overflow happens. | +| `pxf.orc.write.timezone.utc` | `true` | Writes ORC timestamp values using UTC instead of the PXF JVM's local timezone. | +| `pxf.parquet.write.decimal.overflow` | `round` | The behavior when writing a bare `NUMERIC` column, mapped internally to `DECIMAL(38, 18)`, and a value's integer digit count exceeds what that precision and scale allow: `error` fails the write, `round` rounds the value to fit, or `ignore` writes `NULL`. Define the column as `NUMERIC(, )` instead of a bare `NUMERIC` for more control over when an overflow happens. | diff --git a/docs/pxf/6x/reference/index.md b/docs/pxf/6x/reference/index.md new file mode 100644 index 0000000..d872f8b --- /dev/null +++ b/docs/pxf/6x/reference/index.md @@ -0,0 +1,12 @@ +--- +title: PXF reference +navTitle: Reference +description: Reference material for PXF configuration files and commands. +--- + +Reference material for administering PXF. + +- [Commands](commands.md): The `pxf` and `pxf cluster` commands. +- [Profiles](profiles.md): The built-in profiles PXF ships for each connector. +- [Configuration files](configuration-files.md): The files under `$PXF_BASE/conf` and the properties you can set in each. +- [Configuration templates](configuration-templates.md): The site XML templates under `$PXF_HOME/templates` and the properties in each. diff --git a/docs/pxf/6x/reference/profiles.md b/docs/pxf/6x/reference/profiles.md new file mode 100644 index 0000000..939bc7f --- /dev/null +++ b/docs/pxf/6x/reference/profiles.md @@ -0,0 +1,160 @@ +--- +title: PXF profiles +navTitle: Profiles +description: Reference for the built-in profiles PXF ships for each connector. +--- + +Specify a profile as the `PROFILE` parameter in an external table's `LOCATION` clause. See [Understanding PXF profiles](../connecting/index.md#understanding-pxf-profiles) for what a profile is. + +## HDFS, object stores, and network file system + +`hdfs`, `s3`, `abfss` (Azure Data Lake Storage), `wasbs` (Azure Blob Storage), `gs` (Google Cloud Storage), and `file` (network file system) all share the same format suffixes. Combine a connector prefix with a format suffix to form a profile name, for example `s3:parquet` or `file:csv`. + +| Format | Profile suffix | Available on | Notes | +|---|---|---|---| +| Delimited text | `:text` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Plain text. | +| CSV | `:csv` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | | +| Multi-line text | `:text:multi` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Supports multi-line records. | +| Fixed-width text | `:fixedwidth` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Reads columns at fixed byte offsets. | +| Parquet | `:parquet` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Columnar, binary, with the schema embedded in the file. | +| ORC | `:orc` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Columnar, binary. | +| Avro | `:avro` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | Row-based, binary, paired with a schema. | +| JSON | `:json` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs`, `file` | | +| SequenceFile | `:SequenceFile` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs` | Hadoop's binary key-value container format. | +| Avro in a SequenceFile | `:AvroSequenceFile` | `hdfs`, `s3`, `abfss`, `wasbs`, `gs` | Avro-encoded records stored inside a SequenceFile container. | + +`file:SequenceFile` and `file:AvroSequenceFile` aren't available, since SequenceFile is a Hadoop-specific container format. See [Object stores](../connecting/object-stores/index.md), [Hadoop](../connecting/hadoop/index.md), and [Network file system](../connecting/network-file-system.md) for connecting to each of these sources. + +Only `:text` and `:csv` use `FORMAT 'TEXT'` or `FORMAT 'CSV'`. Every other format needs `FORMAT 'CUSTOM'`, with `FORMATTER='pxfwritable_import'` on a readable external table or `FORMATTER='pxfwritable_export'` on a writable one. + +## Hive + +| Profile | Description | +|---|---| +| `hive` | Reads any Hive-supported storage format through Hive's SerDe layer. | +| `hive:text` | Reads a Hive table stored as text files directly, skipping the SerDe layer. | +| `hive:rc` | Reads a Hive table stored as RCFile directly, skipping the SerDe layer. | +| `hive:orc` | Reads a Hive table stored as ORC directly, skipping the SerDe layer. Supports vectorized reads, see [Hive](../connecting/hadoop/hive.md) for the `VECTORIZE` option. | + +The `hive` profile reads through Hive's SerDe (serializer/deserializer), the class Hive itself uses to parse the table's storage format, so it works with any format Hive supports. A `hive:` profile instead skips the SerDe and reads the underlying files directly, which is faster but only available for the formats listed above. See [Hive](../connecting/hadoop/hive.md) for connecting to a Hive metastore. + +## HBase + +| Profile | Description | +|---|---| +| `hbase` | Reads an HBase table. See [HBase](../connecting/hadoop/hbase.md). | + +## JDBC + +| Profile | Description | +|---|---| +| `jdbc` | Reads and writes data through a JDBC driver. See [Connecting to SQL databases over JDBC](../connecting/jdbc.md). | + +## Examples + +The following examples show the options and syntax specific to each format, using a mix of connectors to illustrate that the same options apply regardless of which connector you pair the format with. + +### Multi-byte delimiters + +The `:text` and `:csv` profiles only support a single-byte delimiter. To read data with a multi-byte or multi-character delimiter, such as a currency symbol, use a `:csv` profile with the `pxfdelimited_import` custom formatter instead: + +```sql +CREATE EXTERNAL TABLE mbyte_delim (id int, city text, country text) + LOCATION ('pxf://data/multibyte_currency?PROFILE=s3:csv&SERVER=s3srvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfdelimited_import', DELIMITER='¤'); +``` + +`DELIMITER` is required and accepts a string up to 32 bytes. Add `QUOTE` and `ESCAPE` if your data quotes or escapes values: + +```sql +CREATE EXTERNAL TABLE mbyte_delim_quoted (id int, city text, country text) + LOCATION ('pxf://data/multibyte_quoted?PROFILE=s3:csv&SERVER=s3srvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfdelimited_import', DELIMITER='¤', QUOTE '"', ESCAPE '\'); +``` + +If your data's line ending isn't a line feed, add `NEWLINE` (`CR` or `CRLF`) to both the formatter options and the `LOCATION` clause. This formatter only supports reading data, not writing it. + +### Avro + +Avro is a compact binary format that pairs each record with a schema describing its fields. A schema looks like this: + +```json +{ + "type": "record", + "name": "User", + "fields": [ + { "name": "id", "type": "int" }, + { "name": "name", "type": "string" } + ] +} +``` + +Create a readable external table with an `:avro` profile, pointing at an existing Avro file: + +```sql +CREATE EXTERNAL TABLE pxf_avro_read (id int, name text, details text[]) + LOCATION ('pxf://data/emp_data.avro?PROFILE=gs:avro&SERVER=gcssrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); + +SELECT * FROM pxf_avro_read; +``` + +If you don't provide a `SCHEMA` option, PXF generates the Avro schema from the external table's column definitions, using each column name as the Avro field name: + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_avro_write (id int, username text, followers text[]) + LOCATION ('pxf://data/pxf_write.avro?PROFILE=gs:avro&SERVER=gcssrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); + +INSERT INTO pxf_avro_write VALUES (1, 'Mickey', ARRAY['Pluto', 'Donald', 'Mini']); +``` + +Create a separate readable external table at the same location to query the data back, since you can't query a writable external table directly. To use your own schema file instead of letting PXF generate one, add the `SCHEMA` option to the `LOCATION` clause: + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_avro_write_with_schema (id int, username text, followers text[]) + LOCATION ('pxf://data/pxf_write.avro?PROFILE=gs:avro&SERVER=gcssrvcfg&SCHEMA=/path/to/avro_schema.avsc') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); +``` + +### JSON + +PXF reads and writes JSON in two layouts. The default expects one JSON object per line, known as JSON Lines. An alternate mode expects a single root object whose value is an array of records, useful when each file already holds a complete JSON document rather than one record per line. + +Use dot notation to project nested fields into columns: + +```sql +CREATE EXTERNAL TABLE pxf_json_read (created_at text, "user.id" int, "user.location" text) + LOCATION ('pxf://data/pxf_examples/events.jsonl?PROFILE=file:json&SERVER=filesrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); + +SELECT * FROM pxf_json_read; +``` + +For a file that holds a single root object with an array of records instead of one object per line, add `IDENTIFIER`, naming the field that marks the start of each record: + +```sql +CREATE EXTERNAL TABLE pxf_json_read_single (created_at text, "user.id" int) + LOCATION ('pxf://data/pxf_examples/events.json?PROFILE=file:json&SERVER=filesrvcfg&IDENTIFIER=created_at') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import'); +``` + +PXF writes each row as its own JSON Lines record by default: + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_json_write (created_at text, id int, location text) + LOCATION ('pxf://data/pxf_examples/events_out?PROFILE=file:json&SERVER=filesrvcfg') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); + +INSERT INTO pxf_json_write VALUES ('2026-07-21', 1, 'Austin'); +``` + +Add `ROOT` to write a single root object holding an array of records instead, naming the root-level attribute: + +```sql +CREATE WRITABLE EXTERNAL TABLE pxf_json_write_single (created_at text, id int, location text) + LOCATION ('pxf://data/pxf_examples/events_root?PROFILE=file:json&SERVER=filesrvcfg&ROOT=records') + FORMAT 'CUSTOM' (FORMATTER='pxfwritable_export'); +``` + +Create a separate readable external table at the same location, with a matching `IDENTIFIER`, to query the data back, since you can't query a writable external table directly. PXF only writes scalar columns or one-dimensional arrays, so a writable external table can't include nested or column-projected fields. diff --git a/docs/pxf/6x/release_notes/6.10_rel_notes.md b/docs/pxf/6x/release_notes/6.10_rel_notes.md new file mode 100644 index 0000000..3c0e325 --- /dev/null +++ b/docs/pxf/6x/release_notes/6.10_rel_notes.md @@ -0,0 +1,18 @@ +--- +title: PXF 6.10.2 release notes +navTitle: Version 6.10.2 +description: Release notes for PXF for WarehousePG version 6.10.2. +--- + +## PXF 6.10.2 + +Released: 26 Feb 2026 + +### Enhancements + +- Added `TIMESTAMPTZ` support to the JDBC profile. +- Replaced the `adl` profile with `abfss`, for Azure Data Lake Storage Gen2 support. + +### Bug fixes + +- Fixed the `pxf.control` and `pxf_fdw.control` extension files installing to the PXF package directory instead of `$GPHOME/share/postgresql/extension`, where WHPG expects them. The package now installs both files to the correct location directly, so `pxf cluster register` is no longer required after a default installation. diff --git a/docs/pxf/6x/release_notes/index.md b/docs/pxf/6x/release_notes/index.md new file mode 100644 index 0000000..534cd24 --- /dev/null +++ b/docs/pxf/6x/release_notes/index.md @@ -0,0 +1,11 @@ +--- +title: PXF release notes +navTitle: Release notes +description: Release notes provide information on what's new in each release of PXF for WarehousePG. +--- + +The PXF documentation describes the latest version of PXF for WarehousePG, including minor releases and patches. + +| Version | Release date | +|---------|--------------| +| [6.10.2](6.10_rel_notes.md#pxf-6102) | 26 Feb 2026 | diff --git a/docs/whpg-backup/incremental.md b/docs/whpg-backup/incremental.md index e4a6889..242a20a 100644 --- a/docs/whpg-backup/incremental.md +++ b/docs/whpg-backup/incremental.md @@ -1,5 +1,7 @@ --- title: Creating incremental backups of append-optimized tables +navTitle: Creating incremental backups +description: Create and restore incremental backups of append-optimized tables with gpbackup and gprestore. --- diff --git a/docs/whpg-backup/index.md b/docs/whpg-backup/index.md index c4d90bd..af2530e 100644 --- a/docs/whpg-backup/index.md +++ b/docs/whpg-backup/index.md @@ -1,5 +1,7 @@ --- title: WarehousePG Backup and Restore +navTitle: WarehousePG Backup and Restore +description: Back up and restore WarehousePG tables in parallel with whpg-backup. --- diff --git a/docs/whpg-backup/installing.md b/docs/whpg-backup/installing.md index c728ff4..09fd8f4 100644 --- a/docs/whpg-backup/installing.md +++ b/docs/whpg-backup/installing.md @@ -1,5 +1,7 @@ --- title: Installing WarehousePG Backup and Restore +navTitle: Installing +description: Learn how to install the gpbackup and gprestore utilities on your WarehousePG cluster. --- diff --git a/docs/whpg-backup/overview/architecture.md b/docs/whpg-backup/overview/architecture.md index d4d0af2..97da3a1 100644 --- a/docs/whpg-backup/overview/architecture.md +++ b/docs/whpg-backup/overview/architecture.md @@ -1,5 +1,7 @@ --- title: Architecture of WarehousePG Backup and Restore +navTitle: Architecture +description: Learn how gpbackup and gprestore back up and restore metadata and table data across the coordinator and segment hosts. --- diff --git a/docs/whpg-backup/overview/compatibility.md b/docs/whpg-backup/overview/compatibility.md index 439690d..376b63a 100644 --- a/docs/whpg-backup/overview/compatibility.md +++ b/docs/whpg-backup/overview/compatibility.md @@ -1,5 +1,7 @@ --- title: Compatibility for WarehousePG Backup and Restore +navTitle: Compatibility +description: Compatible WarehousePG versions and supported platforms for gpbackup and gprestore. --- diff --git a/docs/whpg-backup/overview/index.md b/docs/whpg-backup/overview/index.md index acf8c4c..82e8e9c 100644 --- a/docs/whpg-backup/overview/index.md +++ b/docs/whpg-backup/overview/index.md @@ -1,5 +1,7 @@ --- title: Overview of WarehousePG Backup and Restore +navTitle: Overview +description: Learn about the architecture, compatibility, and known issues for WarehousePG Backup and Restore. --- diff --git a/docs/whpg-backup/overview/known_issues.md b/docs/whpg-backup/overview/known_issues.md index cf234ea..90f20fe 100644 --- a/docs/whpg-backup/overview/known_issues.md +++ b/docs/whpg-backup/overview/known_issues.md @@ -1,5 +1,7 @@ --- title: Known issues +navTitle: Known issues +description: Learn about known issues and limitations in WarehousePG Backup and Restore. --- diff --git a/docs/whpg-backup/reference/gpbackup.md b/docs/whpg-backup/reference/gpbackup.md index 2c19c9c..c9dcd91 100644 --- a/docs/whpg-backup/reference/gpbackup.md +++ b/docs/whpg-backup/reference/gpbackup.md @@ -1,5 +1,6 @@ --- title: gpbackup +description: Command reference for the gpbackup utility, including all options and examples. --- diff --git a/docs/whpg-backup/reference/gprestore.md b/docs/whpg-backup/reference/gprestore.md index aa273ac..9058426 100644 --- a/docs/whpg-backup/reference/gprestore.md +++ b/docs/whpg-backup/reference/gprestore.md @@ -1,5 +1,6 @@ --- title: gprestore +description: Command reference for the gprestore utility, including all options and examples. --- diff --git a/docs/whpg-backup/reference/index.md b/docs/whpg-backup/reference/index.md index adb5245..7bd3c9b 100644 --- a/docs/whpg-backup/reference/index.md +++ b/docs/whpg-backup/reference/index.md @@ -1,5 +1,7 @@ --- title: WarehousePG Backup and Restore reference +navTitle: Reference +description: The complete reference to gpbackup and gprestore commands. --- diff --git a/docs/whpg-backup/release_notes/1.31.0_rel_notes.md b/docs/whpg-backup/release_notes/1.31.0_rel_notes.md index 2b38de5..0f73cca 100644 --- a/docs/whpg-backup/release_notes/1.31.0_rel_notes.md +++ b/docs/whpg-backup/release_notes/1.31.0_rel_notes.md @@ -1,5 +1,7 @@ --- title: WarehousePG Backup and Restore 1.31.0 release notes +navTitle: Version 1.31.0 +description: Release notes for version 1.31.0 of WarehousePG Backup and Restore. --- diff --git a/docs/whpg-backup/release_notes/1.32.0_rel_notes.md b/docs/whpg-backup/release_notes/1.32.0_rel_notes.md index f2f828d..b1e709f 100644 --- a/docs/whpg-backup/release_notes/1.32.0_rel_notes.md +++ b/docs/whpg-backup/release_notes/1.32.0_rel_notes.md @@ -1,5 +1,7 @@ --- title: WarehousePG Backup and Restore 1.32.0 release notes +navTitle: Version 1.32.0 +description: Release notes for version 1.32.0 of WarehousePG Backup and Restore. --- diff --git a/docs/whpg-backup/release_notes/1.32.1_rel_notes.md b/docs/whpg-backup/release_notes/1.32.1_rel_notes.md index dc7e30e..cb2f6eb 100644 --- a/docs/whpg-backup/release_notes/1.32.1_rel_notes.md +++ b/docs/whpg-backup/release_notes/1.32.1_rel_notes.md @@ -1,5 +1,7 @@ --- title: WarehousePG Backup and Restore 1.32.1 release notes +navTitle: Version 1.32.1 +description: Release notes for version 1.32.1 of WarehousePG Backup and Restore. --- diff --git a/docs/whpg-backup/release_notes/1.33.0_rel_notes.md b/docs/whpg-backup/release_notes/1.33.0_rel_notes.md index 83208bd..4de6e2c 100644 --- a/docs/whpg-backup/release_notes/1.33.0_rel_notes.md +++ b/docs/whpg-backup/release_notes/1.33.0_rel_notes.md @@ -1,5 +1,7 @@ --- title: WarehousePG Backup and Restore 1.33.0 release notes +navTitle: Version 1.33.0 +description: Release notes for version 1.33.0 of WarehousePG Backup and Restore. --- diff --git a/docs/whpg-backup/release_notes/1.33.1_rel_notes.md b/docs/whpg-backup/release_notes/1.33.1_rel_notes.md index c9727e2..fee7fd4 100644 --- a/docs/whpg-backup/release_notes/1.33.1_rel_notes.md +++ b/docs/whpg-backup/release_notes/1.33.1_rel_notes.md @@ -1,5 +1,7 @@ --- title: WarehousePG Backup and Restore 1.33.1 release notes +navTitle: Version 1.33.1 +description: Release notes for version 1.33.1 of WarehousePG Backup and Restore. --- diff --git a/docs/whpg-backup/release_notes/index.md b/docs/whpg-backup/release_notes/index.md index 4f03326..920179e 100644 --- a/docs/whpg-backup/release_notes/index.md +++ b/docs/whpg-backup/release_notes/index.md @@ -1,5 +1,7 @@ --- title: WarehousePG Backup and Restore release notes +navTitle: Release notes +description: Release notes provide information on what's new in each release of WarehousePG Backup and Restore. --- diff --git a/docs/whpg-backup/s3-plugin.md b/docs/whpg-backup/s3-plugin.md index 6c490e6..edb382e 100644 --- a/docs/whpg-backup/s3-plugin.md +++ b/docs/whpg-backup/s3-plugin.md @@ -1,5 +1,7 @@ --- title: Using the S3 storage plugin +navTitle: Using the S3 storage plugin +description: Configure gpbackup and gprestore to write WarehousePG backups directly to Amazon S3 or an S3-compatible storage server. --- diff --git a/docs/whpg-backup/using.md b/docs/whpg-backup/using.md index c0d18e2..56c4720 100644 --- a/docs/whpg-backup/using.md +++ b/docs/whpg-backup/using.md @@ -1,5 +1,7 @@ --- title: Backing up and restoring your tables +navTitle: Backing up and restoring +description: Back up and restore WarehousePG tables with gpbackup and gprestore, including full backups, filtering, email notifications, and incremental backups. --- diff --git a/docs/whpg/6x/admin_guide/analytics/procedural_languages/pl_java.md b/docs/whpg/6x/admin_guide/analytics/procedural_languages/pl_java.md index 0499a85..874823d 100644 --- a/docs/whpg/6x/admin_guide/analytics/procedural_languages/pl_java.md +++ b/docs/whpg/6x/admin_guide/analytics/procedural_languages/pl_java.md @@ -108,7 +108,7 @@ See the *WarehousePG Reference Guide* for information about the WarehousePG serv PL/Java requires a Java runtime environment on each WarehousePG host. Ensure that the same Java environment is at the same location on all hosts: masters and segments. The command `java -version` displays the Java version. -The commands that you use to install Java depend on the host system operating system and Java version. This list describes how to install OpenJDK 8 or 11 (Java 8 JDK or Java 11 JDK) on RHEL/CentOS or Ubuntu. +The commands that you use to install Java depend on the host system operating system and Java version. This list describes how to install OpenJDK 8 or 11 (Java 8 JDK or Java 11 JDK) on RHEL/CentOS. - RHEL 7/CentOS 7 - This `yum` command installs OpenJDK 8 or 11. @@ -142,23 +142,12 @@ The commands that you use to install Java depend on the host system operating sy The second command returns some `failed to read link` errors that can be ignored. -- Ubuntu - These `apt` commands install OpenJDK 8 or 11. - - ``` - $ sudo apt update - $ sudo apt install openjdk--jdk - ``` - - For OpenJDK 8 the version is `8`, for OpenJDK 11 the version is `11`. - After installing OpenJDK on a RHEL or CentOS system, run this `update-alternatives` command to change the default Java. Enter the number that represents the OpenJDK version to use as the default. ``` $ sudo update-alternatives --config java ``` -The `update-alternatives` command is not required on Ubuntu systems. - > **Note** When configuring host systems, you can use the [gpssh](../../../ref_guide/utility_guide/reference/gpssh.md) utility to run bash shell commands on multiple remote hosts. diff --git a/docs/whpg/6x/admin_guide/analytics/procedural_languages/pl_r.md b/docs/whpg/6x/admin_guide/analytics/procedural_languages/pl_r.md index 0ff0263..111a778 100644 --- a/docs/whpg/6x/admin_guide/analytics/procedural_languages/pl_r.md +++ b/docs/whpg/6x/admin_guide/analytics/procedural_languages/pl_r.md @@ -30,8 +30,6 @@ For information about supported PL/R versions, see the *WarehousePG Release Note For RHEL and CentOS, installing the PL/R package installs R in `$GPHOME/ext/R-` and updates `$GPHOME/greenplum_path.sh` for WarehousePG to use R. -To use PL/R on Ubuntu host systems, you must install and configure R on all WarehousePG host systems before installing PL/R. - > **Note** You can use the [gpssh](../../../ref_guide/utility_guide/reference/gpssh.md) utility to run bash shell commands on multiple remote hosts. 1. To install R, run these `apt` commands on all host systems. @@ -106,7 +104,6 @@ PL/R is registered as an untrusted language. - [Remove PL/R Support for a Database](#topic7) - [Uninstall the Extension Package](#topic8) -- [Uninstall R (Ubuntu)](#topic_ifv_tsf_w3b) When you remove PL/R language support from a database, the PL/R routines that you created in the database will no longer work. @@ -130,27 +127,6 @@ If no databases have PL/R as a registered language, uninstall the WarehousePG PL On RHEL and CentOS systems, uninstalling the extension uninstalls the R software that was installed with the extension. -For Ubuntu systems, remove the `R_HOME` environment variable from `greenplum_path.sh` on all WarehousePG host systems. - -Source the file `$GPHOME/greenplum_path.sh` and restart the database. - -``` -$ gpstop -r -``` - - - -#### Uninstall R (Ubuntu) - -For Ubuntu systems, remove R from all WarehousePG host systems. These commands remove R from an Ubuntu system. - -``` -$ sudo apt remove r-base -$ sudo apt remove r-base-core -``` - -Removing `r-base` does not uninstall the R executable. Removing `r-base-core` uninstalls the R executable. - ### Examples diff --git a/docs/whpg/6x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md b/docs/whpg/6x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md index 19437f4..ef8c5f9 100644 --- a/docs/whpg/6x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md +++ b/docs/whpg/6x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md @@ -13,6 +13,6 @@ Creates a readable external table, *ext_expenses,* using the `pxf` protocol. The ``` -Refer to [Accessing External Data with PXF](../../pxf-overview.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System. +Refer to [HDFS](/pxf/6x/connecting/hadoop/hdfs.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System. **Parent topic:** [Examples for Creating External Tables](index.md) diff --git a/docs/whpg/6x/admin_guide/external/external-tables/index.md b/docs/whpg/6x/admin_guide/external/external-tables/index.md index eb38c29..01f62a2 100644 --- a/docs/whpg/6x/admin_guide/external/external-tables/index.md +++ b/docs/whpg/6x/admin_guide/external/external-tables/index.md @@ -29,7 +29,7 @@ Use one of the following protocols to access external table data sources. You ca The `pxf://` and `s3://` protocols are custom data access protocols, where the `file://`, `gpfdist://`, and `gpfdists://` protocols are implemented internally in WarehousePG. The custom and internal protocols differ in these ways: -- `pxf://` and `s3://` are custom protocols that must be registered using the `CREATE EXTENSION` command (`pxf`) or the `CREATE PROTOCOL` command (`s3`). Registering the PXF extension in a database creates the `pxf` protocol. (See [Accessing External Data with PXF](../pxf-overview.md).) To use the `s3` protocol, you must configure the database and register the `s3` protocol. (See [Configuring the s3 Protocol](s3-protocol.md#s3_prereq).) Internal protocols are always present and cannot be unregistered. +- `pxf://` and `s3://` are custom protocols that must be registered using the `CREATE EXTENSION` command (`pxf`) or the `CREATE PROTOCOL` command (`s3`). Registering the PXF extension in a database creates the `pxf` protocol. (See [PXF for WarehousePG](/pxf/6x/index.md).) To use the `s3` protocol, you must configure the database and register the `s3` protocol. (See [Configuring the s3 Protocol](s3-protocol.md#s3_prereq).) Internal protocols are always present and cannot be unregistered. - When a custom protocol is registered, a row is added to the `pg_extprotocol` catalog table to specify the handler functions that implement the protocol. The protocol's shared libraries must have been installed on all WarehousePG hosts. The internal protocols are not represented in the `pg_extprotocol` table and have no additional libraries to install. - To grant users permissions on custom protocols, you use `GRANT [SELECT | INSERT | ALL] ON PROTOCOL`. To allow (or deny) users permissions on the internal protocols, you use `CREATE ROLE` or `ALTER ROLE` to add the `CREATEEXTTABLE` (or `NOCREATEEXTTABLE`) attribute to each user's role. diff --git a/docs/whpg/6x/admin_guide/external/external-tables/pxf-protocol.md b/docs/whpg/6x/admin_guide/external/external-tables/pxf-protocol.md index 41b85f9..a95a35e 100644 --- a/docs/whpg/6x/admin_guide/external/external-tables/pxf-protocol.md +++ b/docs/whpg/6x/admin_guide/external/external-tables/pxf-protocol.md @@ -11,6 +11,6 @@ When you use the `pxf` protocol to query an external data store, you specify the You must explicitly initialize and start PXF before you can use the `pxf` protocol to read or write external data. You must also enable PXF in each database in which you want to allow users to create external tables to access external data, and grant permissions on the `pxf` protocol to those WarehousePG users. -For detailed information about configuring and using PXF and the `pxf` protocol, refer to [Accessing External Data with PXF](../pxf-overview.md). +For detailed information about configuring and using PXF and the `pxf` protocol, refer to [PXF for WarehousePG](/pxf/6x/index.md). **Parent topic:** [Defining External Tables](index.md) diff --git a/docs/whpg/6x/admin_guide/external/index.md b/docs/whpg/6x/admin_guide/external/index.md index 5efc441..6aafd45 100644 --- a/docs/whpg/6x/admin_guide/external/index.md +++ b/docs/whpg/6x/admin_guide/external/index.md @@ -3,7 +3,6 @@ title: Working with External Data navigation: - foreign - external-tables - - pxf-overview - using-gpfdist --- @@ -12,13 +11,13 @@ Both external and foreign tables provide access to data stored in data sources o An external table is a WarehousePG table backed with data that resides outside of the database. You create a readable external table to read data from the external data source and create a writable external table to write data to the external source. You can use external tables in SQL commands just as you would a regular database table. For example, you can `SELECT` (readable external table), `INSERT` (writable external table), and join external tables with other WarehousePG tables. External tables are most often used to load and unload database data. Refer to [Defining External Tables](external-tables/index.md) for more information about using external tables to access external data. -[Accessing External Data with PXF](pxf-overview.md) describes using PXF and external tables to access external data sources. +[PXF for WarehousePG](/pxf/6x/index.md) describes using PXF and external tables to access external data sources. A foreign table is a different kind of WarehousePG table backed with data that resides outside of the database. You can both read from and write to the same foreign table. You can similarly use foreign tables in SQL commands as described above for external tables. Refer to [Accessing External Data with Foreign Tables](foreign/index.md) for more information about accessing external data using foreign tables. Web-based external tables provide access to data served by an HTTP server or an operating system process. See [Creating and Using External Web Tables](external-tables/creating-and-using-web-external-tables.md) for more about web-based tables. -- **[Accessing External Data with PXF](pxf-overview.md)** +- **[PXF for WarehousePG](/pxf/6x/index.md)** Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. - **[Defining External Tables](external-tables/index.md)** diff --git a/docs/whpg/6x/admin_guide/external/pxf-overview.md b/docs/whpg/6x/admin_guide/external/pxf-overview.md deleted file mode 100644 index a096eb1..0000000 --- a/docs/whpg/6x/admin_guide/external/pxf-overview.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: Accessing External Data with PXF - ---- - -Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. - -PXF is installed with Hadoop and Object Storage connectors. These connectors enable you to read external data stored in text, Avro, JSON, RCFile, Parquet, SequenceFile, and ORC formats. You can use the JDBC connector to access an external SQL database. - -The WarehousePG Platform Extension Framework includes a C-language extension and a Java service. After you configure and initialize PXF, you start a single PXF JVM process on each WarehousePG segment host. This long- running process concurrently serves multiple query requests. - -For detailed information about the architecture of and using PXF, refer to the PXF documentation. - -**Parent topic:** [Working with External Data](index.md) - -**Parent topic:** [Loading and Unloading Data](../load/index.md) diff --git a/docs/whpg/6x/admin_guide/load/index.md b/docs/whpg/6x/admin_guide/load/index.md index 86b13f0..ebcd7dd 100644 --- a/docs/whpg/6x/admin_guide/load/index.md +++ b/docs/whpg/6x/admin_guide/load/index.md @@ -26,7 +26,7 @@ WarehousePG can read from and write to several types of external data sources, i - External web tables provide access to dynamic data. They can be backed with data from URLs accessed using the HTTP protocol or by the output of an OS script running on one or more segments. - The `gpfdist` utility is the WarehousePG parallel file distribution program. It is an HTTP server that is used with external tables to allow WarehousePG segments to load external data in parallel, from multiple file systems. You can run multiple instances of `gpfdist` on different hosts and network interfaces and access them in parallel. - The `gpload` utility automates the steps of a load task using `gpfdist` and a YAML-formatted control file. -- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [Accessing External Data with PXF](../external/pxf-overview.md). +- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md). The method you choose to load data depends on the characteristics of the source data—its location, size, format, and any transformations required. diff --git a/docs/whpg/6x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md b/docs/whpg/6x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md index 91e9f1c..f415fa9 100644 --- a/docs/whpg/6x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md +++ b/docs/whpg/6x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md @@ -8,7 +8,7 @@ navigation: Writable external tables that output data to files can use the WarehousePG parallel file server program, gpfdist, or the WarehousePG Platform Extension Framework (PXF), WarehousePG's interface to Hadoop. -Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [Accessing External Data with PXF](../../../external/pxf-overview.md) for instructions on setting up PXF for use with an external table +Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [PXF for WarehousePG](/pxf/6x/index.md) for instructions on setting up PXF for use with an external table - With a writable external table using the gpfdist protocol, the WarehousePG segments send their data to gpfdist, which writes the data to the named file. gpfdist must run on a host that the WarehousePG segments can access over the network. gpfdist points to a file location on the output host and writes data received from the WarehousePG segments to the file. To divide the output data among multiple files, list multiple gpfdist URIs in your writable external table definition. - A writable external web table sends data to an application as a stream of data. For example, unload data from WarehousePG and send it to an application that connects to another database or ETL tool to load the data elsewhere. Writable external web tables use the `EXECUTE` clause to specify a shell command, script, or application to run on the segment hosts and accept an input stream of data. See [Defining a Command-Based Writable External Web Table](../defining-a-command-based-writable-external-web-table/index.md) for more information about using `EXECUTE` commands in a writable external table definition. diff --git a/docs/whpg/6x/admin_guide/manage_access/client_auth/index.md b/docs/whpg/6x/admin_guide/manage_access/client_auth/index.md index 8e8922b..d7efdd8 100644 --- a/docs/whpg/6x/admin_guide/manage_access/client_auth/index.md +++ b/docs/whpg/6x/admin_guide/manage_access/client_auth/index.md @@ -40,107 +40,19 @@ local   database   role   authentication-method The following table describes meaning of each field. -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Table 1. pg_hba.conf Fields
FieldDescription
localMatches connection attempts using UNIX-domain sockets. Without a - record of this type, UNIX-domain socket connections are disallowed.
hostMatches connection attempts made using TCP/IP. Remote TCP/IP - connections will not be possible unless the server is started with an appropriate - value for the listen_addresses server configuration - parameter.
hostsslMatches connection attempts made using TCP/IP, but only when the - connection is made with SSL encryption. SSL must be enabled at server start time by - setting the ssl server configuration parameter.
hostnosslMatches connection attempts made over TCP/IP that do not use - SSL.
databaseSpecifies which database names this record matches. The value - all specifies that it matches all databases. Multiple database - names can be supplied by separating them with commas. A separate file containing - database names can be specified by preceding the file name with a - @.
roleSpecifies which database role names this record matches. The - value all specifies that it matches all roles. If the specified - role is a group and you want all members of that group to be included, precede the - role name with a +. Multiple role names can be supplied by - separating them with commas. A separate file containing role names can be specified - by preceding the file name with a @.
addressSpecifies the client machine addresses that this record matches. - This field can contain an IP address, an IP address range, or a host name.

An IP - address range is specified using standard numeric notation for the range's - starting address, then a slash (/) and a CIDR mask length. The - mask length indicates the number of high-order bits of the client IP address that - must match. Bits to the right of this should be zero in the given IP address. - There must not be any white space between the IP address, the /, - and the CIDR mask length.

Typical examples of an IPv4 address range - specified this way are 172.20.143.89/32 for a single host, or - 172.20.143.0/24 for a small network, or - 10.6.0.0/16 for a larger one. An IPv6 address range might look - like ::1/128 for a single host (in this case the IPv6 loopback - address) or fe80::7a31:c1ff:0000:0000/96 for a small network. - 0.0.0.0/0 represents all IPv4 addresses, and - ::0/0 represents all IPv6 addresses. To specify a single host, - use a mask length of 32 for IPv4 or 128 for IPv6. In a network address, do not - omit trailing zeroes.

An entry given in IPv4 format will match only IPv4 - connections, and an entry given in IPv6 format will match only IPv6 connections, - even if the represented address is in the IPv4-in-IPv6 range. -
Note: Entries in IPv6 format will be rejected if the host system C library does - not have support for IPv6 addresses.

If a host name is specified - (an address that is not an IP address or IP range is treated as a host name), that - name is compared with the result of a reverse name resolution of the client IP - address (for example, reverse DNS lookup, if DNS is used). Host name comparisons - are case insensitive. If there is a match, then a forward name resolution (for - example, forward DNS lookup) is performed on the host name to check whether any of - the addresses it resolves to are equal to the client IP address. If both - directions match, then the entry is considered to match.

Some host name - databases allow associating an IP address with multiple host names, but the - operating system only returns one host name when asked to resolve an IP address. - The host name that is used in pg_hba.conf must be the one that - the address-to-name resolution of the client IP address returns, otherwise the - line will not be considered a match.

When host names are specified in - pg_hba.conf, you should ensure that name resolution is - reasonably fast. It can be of advantage to set up a local name resolution cache - such as nscd. Also, you can enable the server configuration - parameter log_hostname to see the client host name instead of the - IP address in the log.

IP-address

IP-mask

These fields can be used as an alternative to the CIDR address - notation. Instead of specifying the mask length, the actual mask is specified in a - separate column. For example, 255.0.0.0 represents an IPv4 CIDR - mask length of 8, and 255.255.255.255 represents a CIDR mask length - of 32.
authentication-methodSpecifies the authentication method to use when connecting. - WarehousePG supports the authentication methods supported by PostgreSQL 9.4.
+**Table 1. pg_hba.conf Fields** + +| Field | Description | +|-------|--------------| +| `local` | Matches connection attempts using UNIX-domain sockets. Without a record of this type, UNIX-domain socket connections are disallowed. | +| `host` | Matches connection attempts made using TCP/IP. Remote TCP/IP connections aren't possible unless the server starts with an appropriate value for the `listen_addresses` server configuration parameter. | +| `hostssl` | Matches connection attempts made using TCP/IP, but only when the connection is made with SSL encryption. SSL must be enabled at server start time by setting the `ssl` server configuration parameter. | +| `hostnossl` | Matches connection attempts made over TCP/IP that don't use SSL. | +| `database` | Specifies which database names this record matches. The value `all` specifies that it matches all databases. Multiple database names can be supplied by separating them with commas. A separate file containing database names can be specified by preceding the file name with a `@`. | +| `role` | Specifies which database role names this record matches. The value `all` specifies that it matches all roles. If the specified role is a group and you want all members of that group included, precede the role name with a `+`. Multiple role names can be supplied by separating them with commas. A separate file containing role names can be specified by preceding the file name with a `@`. | +| `address` | Specifies the client machine addresses that this record matches. This field can contain an IP address, an IP address range, or a host name.

An IP address range is specified using standard numeric notation for the range's starting address, then a slash (`/`) and a CIDR mask length. The mask length indicates the number of high-order bits of the client IP address that must match. Bits to the right of this mask length must be zero in the given IP address. There must not be any white space between the IP address, the `/`, and the CIDR mask length.

Typical examples of an IPv4 address range specified this way are `172.20.143.89/32` for a single host, `172.20.143.0/24` for a small network, or `10.6.0.0/16` for a larger one. An IPv6 address range might look like `::1/128` for a single host (in this case the IPv6 loopback address) or `fe80::7a31:c1ff:0000:0000/96` for a small network. `0.0.0.0/0` represents all IPv4 addresses, and `::0/0` represents all IPv6 addresses. To specify a single host, use a mask length of 32 for IPv4 or 128 for IPv6. In a network address, don't omit trailing zeroes.

An entry given in IPv4 format matches only IPv4 connections, and an entry given in IPv6 format matches only IPv6 connections, even if the represented address is in the IPv4-in-IPv6 range. **Note** Entries in IPv6 format are rejected if the host system C library doesn't support IPv6 addresses.

If a host name is specified (an address that isn't an IP address or IP range is treated as a host name), that name is compared with the result of a reverse name resolution of the client IP address (for example, reverse DNS lookup, if DNS is used). Host name comparisons are case insensitive. If there's a match, then a forward name resolution (for example, forward DNS lookup) is performed on the host name to check whether any of the addresses it resolves to are equal to the client IP address. If both directions match, the entry is considered a match.

Some host name databases allow associating an IP address with multiple host names, but the operating system only returns one host name when asked to resolve an IP address. The host name used in `pg_hba.conf` must be the one that the address-to-name resolution of the client IP address returns, otherwise the line isn't considered a match.

When host names are specified in `pg_hba.conf`, make sure that name resolution is reasonably fast. Setting up a local name resolution cache such as `nscd` can help. You can also enable the server configuration parameter `log_hostname` to see the client host name instead of the IP address in the log. | +| `IP-address`
`IP-mask` | These fields can be used as an alternative to the CIDR address notation. Instead of specifying the mask length, the actual mask is specified in a separate column. For example, `255.0.0.0` represents an IPv4 CIDR mask length of 8, and `255.255.255.255` represents a CIDR mask length of 32. | +| `authentication-method` | Specifies the authentication method to use when connecting. WarehousePG supports the [authentication methods](https://www.postgresql.org/docs/9.4/auth-methods.html) supported by PostgreSQL 9.4. | > **Caution** For a more secure system, consider removing records for remote connections that use trust authentication from the `pg_hba.conf` file. Trust authentication grants any user who can connect to the server access to the database using any role they specify. You can safely replace trust authentication with ident authentication for local UNIX-socket connections. You can also use ident authentication for local and remote TCP clients, but the client host must be running an ident service and you must trust the integrity of that machine. @@ -292,6 +204,23 @@ gpconfig -c ssl -m on -v off Setting the parameter requires a server restart. This command restarts the system: `gpstop -ra`. + + +### Negotiating TLS Version Automatically + +WarehousePG negotiates the highest TLS protocol version that both the client and the coordinator support. When both endpoints run OpenSSL 1.1.1 or later, connections use TLS 1.3 automatically, with no additional configuration required. + +Confirm the negotiated version for a connection with the `ssl_version()` function, available after you create the `sslinfo` extension: + +``` +psql -d testdb -c 'CREATE EXTENSION sslinfo;' +psql -d testdb -c 'SELECT ssl_version();' +``` + +::: info Note +The `pg_stat_ssl` system view, which reports the negotiated TLS version and cipher for each connection, is not available in WarehousePG 6 because WarehousePG 6 is based on PostgreSQL 9.4, and the view was introduced in a later PostgreSQL release. +::: + ### Creating a Self-signed Certificate without a Passphrase for Testing Only @@ -299,7 +228,7 @@ Setting the parameter requires a server restart. This command restarts the syste To create a quick self-signed certificate for the server for testing, use the following OpenSSL command: ``` -# openssl req -new -text -out server.req +openssl req -new -text -out server.req ``` @@ -310,8 +239,8 @@ The program will generate a key that is passphrase protected, and does not accep To use this certificate with WarehousePG, remove the passphrase with the following commands: ``` -# openssl rsa -in privkey.pem -out server.key -# rm privkey.pem +openssl rsa -in privkey.pem -out server.key +rm privkey.pem ``` Enter the old passphrase when prompted to unlock the existing key. @@ -319,13 +248,13 @@ Enter the old passphrase when prompted to unlock the existing key. Then, enter the following command to turn the certificate into a self-signed certificate and to copy the key and certificate to a location where the server will look for them. ``` -# openssl req -x509 -in server.req -text -key server.key -out server.crt +openssl req -x509 -in server.req -text -key server.key -out server.crt ``` Finally, change the permissions on the key with the following command. The server will reject the file if the permissions are less restrictive than these. ``` -# chmod og-rwx server.key +chmod og-rwx server.key ``` For more details on how to create your server private key and certificate, refer to the [OpenSSL documentation](https://www.openssl.org/docs/). diff --git a/docs/whpg/6x/admin_guide/managing/compression.md b/docs/whpg/6x/admin_guide/managing/compression.md index b83b8ee..d4f6c6e 100644 --- a/docs/whpg/6x/admin_guide/managing/compression.md +++ b/docs/whpg/6x/admin_guide/managing/compression.md @@ -9,7 +9,7 @@ You can configure support for data compression with these features and utilities - Append-optimized tables support compressing table data. See [CREATE TABLE](../../ref_guide/sql_commands/CREATE_TABLE.md). - User-defined data types can be defined to compress data. See [CREATE TYPE](../../ref_guide/sql_commands/CREATE_TYPE.md). -- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](../external/pxf-overview.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md). +- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](/pxf/6x/index.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md). - Workfiles (temporary spill files that are created when running a query that requires more memory than it is allocated) can be compressed. See the server configuration parameter [gp_workfile_compression](../../ref_guide/config_params/guc-list.md). - The WarehousePG utilities gpbackup, gprestore, [gpload](../../ref_guide/utility_guide/reference/gpload.md), and [gplogfilter](../../ref_guide/utility_guide/reference/gplogfilter.md) support compression. diff --git a/docs/whpg/6x/install_guide/additional_modules/external_components/index.md b/docs/whpg/6x/install_guide/additional_modules/external_components/index.md index c262917..21d6637 100644 --- a/docs/whpg/6x/install_guide/additional_modules/external_components/index.md +++ b/docs/whpg/6x/install_guide/additional_modules/external_components/index.md @@ -6,6 +6,6 @@ description: Installation guides for tools that integrate with WarehousePG but a These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`. -- [WarehousePG Platform Extension Framework (PXF)](../../data_sci_pkgs/install_pxf.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop. -- whpg-backup - Parallel backup and restore for WarehousePG clusters. -- whpg-backup-s3-plugin - Amazon S3 support for whpg-backup. +- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop. +- [whpg-backup](/whpg-backup/index.md) - Parallel backup and restore for WarehousePG clusters. +- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Amazon S3 support for whpg-backup. diff --git a/docs/whpg/6x/install_guide/config_os.md b/docs/whpg/6x/install_guide/config_os.md index 6514e81..3a38b03 100644 --- a/docs/whpg/6x/install_guide/config_os.md +++ b/docs/whpg/6x/install_guide/config_os.md @@ -63,7 +63,7 @@ If you choose to enable SELinux in `Enforcing` mode, then WarehousePG processes ## Deactivate or Configure Firewall Software -You should also deactivate firewall software such as `iptables` (on systems such as RHEL 6.x and CentOS 6.x ), `firewalld` (on systems such as RHEL 7.x and CentOS 7.x and later), or `ufw` (on Ubuntu systems, deactivated by default). If firewall software is not deactivated, you must instead configure your software to allow required communication between WarehousePG hosts. +You should also deactivate firewall software such as `iptables` (on systems such as RHEL 6.x and CentOS 6.x) or `firewalld` (on systems such as RHEL 7.x and CentOS 7.x and later). If firewall software is not deactivated, you must instead configure your software to allow required communication between WarehousePG hosts. To deactivate `iptables`: @@ -344,7 +344,7 @@ XFS is the preferred data storage file system on Linux platforms. Use the `mount rw,nodev,noatime,nobarrier,inode64 ``` -The `nobarrier` option is not supported on RHEL 8 or Ubuntu systems or later. Use only the options: +The `nobarrier` option is not supported on RHEL 8 systems or later. Use only the options: ``` rw,nodev,noatime,inode64 @@ -425,7 +425,7 @@ The XFS options can also be set in the `/etc/fstab` file. This example entry fro Non-Volatile Memory Express (NVMe) - RHEL 7
RHEL 8
RHEL 9
Ubuntu + RHEL 7
RHEL 8
RHEL 9 none @@ -434,7 +434,7 @@ The XFS options can also be set in the `/etc/fstab` file. This example entry fro noop - RHEL 8
RHEL 9
Ubuntu + RHEL 8
RHEL 9 none @@ -443,7 +443,7 @@ The XFS options can also be set in the `/etc/fstab` file. This example entry fro deadline - RHEL 8
RHEL 9
Ubuntu + RHEL 8
RHEL 9 mq-deadline @@ -479,7 +479,7 @@ The XFS options can also be set in the `/etc/fstab` file. This example entry fro Refer to your operating system documentation for more information about the `grubby` utility. If you used the `grubby` command to configure the disk scheduler on a RHEL or CentOS 7.x system and later and it does not update the kernels, see the [Note](#grubby_note) at the end of the section. - For additional information about configuring the disk scheduler, refer to the RedHat Enterprise Linux documentation for [RHEL 7](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-storage_and_file_systems-configuration_tools#sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Configuration_tools-Setting_the_default_IO_scheduler), [RHEL 8](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/setting-the-disk-scheduler_monitoring-and-managing-system-status-and-performance), or [RHEL 9](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/monitoring_and_managing_system_status_and_performance/setting-the-disk-scheduler_monitoring-and-managing-system-status-and-performance). The Ubuntu wiki [IOSchedulers](https://wiki.ubuntu.com/Kernel/Reference/IOSchedulers) topic describes the I/O schedulers available on Ubuntu systems. + For additional information about configuring the disk scheduler, refer to the RedHat Enterprise Linux documentation for [RHEL 7](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-storage_and_file_systems-configuration_tools#sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Configuration_tools-Setting_the_default_IO_scheduler), [RHEL 8](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/setting-the-disk-scheduler_monitoring-and-managing-system-status-and-performance), or [RHEL 9](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/monitoring_and_managing_system_status_and_performance/setting-the-disk-scheduler_monitoring-and-managing-system-status-and-performance). @@ -518,12 +518,6 @@ On systems that use `grub2` such as RHEL 7.x or CentOS 7.x and later, use the sy After adding the parameter, reboot the system. -For Ubuntu systems, install the `hugepages` package and run this command as root: - -``` -# hugeadm --thp-never -``` - This cat command checks the state of THP. The output indicates that THP is deactivated. ``` @@ -537,7 +531,7 @@ For more information about Transparent Huge Pages or the `grubby` utility, see y ### IPC Object Removal -Deactivate IPC object removal for RHEL 7.2 or CentOS 7.2, or Ubuntu. The default `systemd` setting `RemoveIPC=yes` removes IPC connections when non-system user accounts log out. This causes the WarehousePG utility `gpinitsystem` to fail with semaphore errors. Perform one of the following to avoid this issue. +Deactivate IPC object removal for RHEL 7.2 or CentOS 7.2. The default `systemd` setting `RemoveIPC=yes` removes IPC connections when non-system user accounts log out. This causes the WarehousePG utility `gpinitsystem` to fail with semaphore errors. Perform one of the following to avoid this issue. - When you add the `gpadmin` operating system user account to the coordinator node in [Creating the WarehousePG Administrative User](#topic23), create the user as a system account. - Deactivate `RemoveIPC`. Set this parameter in `/etc/systemd/logind.conf` on the WarehousePG host systems. @@ -700,7 +694,7 @@ The following steps show how to set up the `gpadmin` user on a host, set a passw 1. Create the `gpadmin` group and user. - > **Note** If you are installing WarehousePG on RHEL 7.2 or CentOS 7.2 and want to deactivate IPC object removal by creating the `gpadmin` user as a system account, provide both the `-r` option (create the user as a system account) and the `-m` option (create a home directory) to the `useradd` command. On Ubuntu systems, you must use the `-m` option with the `useradd` command to create a home directory for a user. + > **Note** If you are installing WarehousePG on RHEL 7.2 or CentOS 7.2 and want to deactivate IPC object removal by creating the `gpadmin` user as a system account, provide both the `-r` option (create the user as a system account) and the `-m` option (create a home directory) to the `useradd` command. This example creates the `gpadmin` group, creates the `gpadmin` user as a system account with a home directory and as a member of the `gpadmin` group, and creates a password for the user. diff --git a/docs/whpg/6x/install_guide/data_sci_pkgs/install_pxf.md b/docs/whpg/6x/install_guide/data_sci_pkgs/install_pxf.md index 88788c1..47983f7 100644 --- a/docs/whpg/6x/install_guide/data_sci_pkgs/install_pxf.md +++ b/docs/whpg/6x/install_guide/data_sci_pkgs/install_pxf.md @@ -5,6 +5,6 @@ title: Platform Extension Framework (PXF) *Optional.* If you do not plan to use PXF, no action is necessary. -If you plan to use PXF, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for introductory PXF information. +If you plan to use PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md) for installation and usage information. **Parent topic:** [Installing Optional Extensions (WarehousePG)](index.md) diff --git a/docs/whpg/6x/install_guide/install_whpg.md b/docs/whpg/6x/install_guide/install_whpg.md index ecf96e0..b3a12d9 100644 --- a/docs/whpg/6x/install_guide/install_whpg.md +++ b/docs/whpg/6x/install_guide/install_whpg.md @@ -45,9 +45,9 @@ See [Example Ansible Playbook](ansible-example.md) for an example script that sh Follow these instructions to install WarehousePG from a pre-built binary. -> **Important** You require sudo or root user access to install from a pre-built RPM or DEB file. +> **Important** You require sudo or root user access to install from a pre-built RPM file. -1. Download and copy the WarehousePG package to the `gpadmin` user's home directory on the coordinator, standby coordinator, and every segment host machine. The distribution file name has the format `greenplum-db--.rpm` for RHEL, CentOS, and Oracle Linux systems, or `greenplum-db--.deb` for Ubuntu systems, where `` is similar to `rhel7-x86_64` (Red Hat 7 64-bit). +1. Download and copy the WarehousePG package to the `gpadmin` user's home directory on the coordinator, standby coordinator, and every segment host machine. The distribution file name has the format `greenplum-db--.rpm` for RHEL, CentOS, and Oracle Linux systems, where `` is similar to `rhel7-x86_64` (Red Hat 7 64-bit). > **Note** For Oracle Linux installations, download and install the `rhel7-x86_64`distribution files. @@ -58,13 +58,8 @@ Follow these instructions to install WarehousePG from a pre-built binary. ``` $ sudo yum install ./greenplum-db--.rpm ``` - - For Ubuntu systems, run the `apt` command: - - ``` - $ sudo apt install ./greenplum-db--.deb - ``` - The `yum` or `apt` command automatically installs software dependencies, copies the WarehousePG software files into a version-specific directory under `/usr/local`, `/usr/local/greenplum-db-`, and creates the symbolic link `/usr/local/greenplum-db` to the installation directory. + The `yum` command automatically installs software dependencies, copies the WarehousePG software files into a version-specific directory under `/usr/local`, `/usr/local/greenplum-db-`, and creates the symbolic link `/usr/local/greenplum-db` to the installation directory. 3. Change the owner and group of the installed files to `gpadmin`: @@ -83,7 +78,7 @@ Follow these instructions to install WarehousePG to a specific directory. > **Important** You require sudo or root user access to install from a pre-built RPM file. -1. Download and copy the WarehousePG package to the `gpadmin` user's home directory on the coordinator, standby coordinator, and every segment host machine. The distribution file name has the format `greenplum-db--.rpm` for RHEL and CentOS systems, or `greenplum-db--.deb` for Ubuntu systems, where `` is similar to `rhel7-x86_64` (Red Hat 7 64-bit). +1. Download and copy the WarehousePG package to the `gpadmin` user's home directory on the coordinator, standby coordinator, and every segment host machine. The distribution file name has the format `greenplum-db--.rpm` for RHEL and CentOS systems, where `` is similar to `rhel7-x86_64` (Red Hat 7 64-bit). 2. Manually install the WarehousePG dependencies to each host system: diff --git a/docs/whpg/6x/install_guide/migrate-linux.md b/docs/whpg/6x/install_guide/migrate-linux.md index 8265fd5..befc958 100644 --- a/docs/whpg/6x/install_guide/migrate-linux.md +++ b/docs/whpg/6x/install_guide/migrate-linux.md @@ -375,7 +375,7 @@ When you prepare your operating system environment for WarehousePG software inst ### XFS Mount Options -XFS is the preferred data storage file system on Linux platforms. Use the mount command with the following recommended XFS mount options. The `nobarrier` option is not supported on EL 8/9 or Ubuntu systems. Use only the options `rw,nodev,noatime,inode64`. +XFS is the preferred data storage file system on Linux platforms. Use the mount command with the following recommended XFS mount options. The `nobarrier` option is not supported on EL 8/9 systems. Use only the options `rw,nodev,noatime,inode64`. diff --git a/docs/whpg/6x/install_guide/platform-requirements.md b/docs/whpg/6x/install_guide/platform-requirements.md index c251a9d..9390e6f 100644 --- a/docs/whpg/6x/install_guide/platform-requirements.md +++ b/docs/whpg/6x/install_guide/platform-requirements.md @@ -19,7 +19,6 @@ WarehousePG 6 runs on the following operating system platforms: - Rocky Linux 8.7 or later - CentOS 64-bit 7.x - CentOS 64-bit 6.x -- Ubuntu 18.04 LTS - Oracle Linux 64-bit 7, using the Red Hat Compatible Kernel (RHCK) @@ -37,7 +36,7 @@ RHEL 7.3 and CentOS 7.3 resolves the issue. > **Caution** A kernel issue in Red Hat Enterprise Linux 8.5 and 8.6 can cause I/O freezes and synchronization problems with XFS filesystems. This issue is fixed in RHEL 8.7. See [RHEL8: xfs_buf deadlock between inode deletion and block allocation](https://access.redhat.com/solutions/6984334). -WarehousePG server supports TLS version 1.2 on RHEL/CentOS systems, and TLS version 1.3 on Ubuntu systems. +WarehousePG server supports TLS version 1.2 and TLS version 1.3 on RHEL/CentOS systems. @@ -91,36 +90,6 @@ WarehousePG 6 client software requires these operating system packages: - openssh - zlib -On Ubuntu systems, WarehousePG 6 requires the following software packages, which are installed automatically as dependencies when you install WarehousePG with the Debian package installer: - -- bash -- bzip2 -- iproute2 -- iputils-ping -- krb5-multidev -- libapr1 -- libaprutil1 -- libcurl3-gnutls -- libcurl4 -- libevent-2.1-6 -- libldap-2.4-2 -- libreadline7 or libreadline8 -- libuuid1 -- libxml2 -- libyaml-0-2 -- less -- locales -- net-tools -- openssh-client -- openssh-server -- openssl -- perl -- rsync -- sed -- tar -- zip -- zlib1g - WarehousePG 6 uses Python 2.7.18, which is included with the product installation (and not installed as a package dependency). > **Important** SSL is supported only on the WarehousePG coordinator host system. It cannot be used on the segment host systems. @@ -162,8 +131,7 @@ This table lists the versions of the WarehousePG Extensions that are compatible PL/R 3.0.3 -(CentOS) R 3.3.3

(Ubuntu) You install R 3.5.1+.

- +R 3.3.3 MADlib Machine Learning @@ -191,7 +159,7 @@ These WarehousePG extensions are installed with WarehousePG ### Data Connectors -- WarehousePG Platform Extension Framework (PXF) - PXF provides access to Hadoop, object store, and SQL external data stores. Refer to [Accessing External Data with PXF](../admin_guide/external/pxf-overview.md) in the *WarehousePG Administrator Guide* for PXF configuration and usage information. +- WarehousePG Platform Extension Framework (PXF) - PXF provides access to Hadoop, object store, and SQL external data stores. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for PXF configuration and usage information. - WarehousePG Connector for Apache Spark v1.6.2 - The WarehousePG Connector for Apache Spark supports high speed, parallel data transfer between WarehousePG and an Apache Spark cluster using Spark’s Scala API. @@ -272,7 +240,7 @@ The disk settings for cloud deployments are the same as on-premise with a few mo ``` rw,noatime,nobarrier,nodev,inode64 ``` - > **Note** The `nobarrier` option is not supported on RHEL 8 or Ubuntu nodes. + > **Note** The `nobarrier` option is not supported on RHEL 8 nodes. - Use mq-deadline instead of the deadline scheduler for the R5 series instance type in AWS - Use a swap disk per VM (32GB size works well) diff --git a/docs/whpg/6x/install_guide/upgrading.md b/docs/whpg/6x/install_guide/upgrading.md index 2b45fe2..fa95773 100644 --- a/docs/whpg/6x/install_guide/upgrading.md +++ b/docs/whpg/6x/install_guide/upgrading.md @@ -51,7 +51,7 @@ Starting from WarehousePG 6.27.4, the default installation directory is `/usr/ed 3. Copy the new WarehousePG software installation package to the `gpadmin` user's home directory on each coordinator, standby, and segment host. -4. *If you used `yum` or `apt` to install WarehousePG to the default location*, run these commands on each host to upgrade to the new software release. +4. *If you used `yum` to install WarehousePG to the default location*, run this command on each host to upgrade to the new software release. For RHEL/CentOS systems: @@ -59,13 +59,7 @@ Starting from WarehousePG 6.27.4, the default installation directory is `/usr/ed $ sudo yum upgrade ./greenplum-db--.rpm ``` - For Ubuntu systems: - - ``` - # apt install ./greenplum-db--.deb - ``` - - The `yum` or `apt` command installs the new WarehousePG software files into a version-specific directory under `/usr/edb/whpg6` and updates the symbolic link `/usr/local/greenplum-db` to point to the new installation directory. + The `yum` command installs the new WarehousePG software files into a version-specific directory under `/usr/edb/whpg6` and updates the symbolic link `/usr/local/greenplum-db` to point to the new installation directory. 5. *If you used `rpm` to install WarehousePG to a non-default location on RHEL/CentOS systems*, run `rpm` on each host to upgrade to the new software release and specify the same custom installation directory with the `--prefix` option. For example: diff --git a/docs/whpg/6x/ref_guide/modules/bundled/index.md b/docs/whpg/6x/ref_guide/modules/bundled/index.md index 80515a0..0770110 100644 --- a/docs/whpg/6x/ref_guide/modules/bundled/index.md +++ b/docs/whpg/6x/ref_guide/modules/bundled/index.md @@ -2,7 +2,6 @@ title: Bundled modules description: Modules that ship with WarehousePG and don't require a separate package install. navigation: - - auto-explain - btree_gin - citext - dblink @@ -15,51 +14,52 @@ navigation: - gp_pitr - gp_sparse_vector - gp_subtransaction_overflow - - greenplum_fdw - hstore - - ip4r - isn - ltree - orafce_ref - pageinspect - - pg_cron - pg_trgm - pgcrypto - postgres_fdw - sslinfo - tablefunc - - timestamp9 - uuid-ossp --- These modules ship with WarehousePG. Activate them in each database with `CREATE EXTENSION `, unless the individual reference page specifies a different activation method. -- [auto_explain](auto-explain.md) - Logs execution plans of slow statements automatically. +- amcheck - Provides functions for verifying the logical consistency of indexes and heap relations. - [btree_gin](btree_gin.md) - Provides GIN operator classes that implement B-tree equivalent behavior for certain data types. - [citext](citext.md) - Provides a case-insensitive, multibyte-aware text data type. - [dblink](dblink.md) - Provides connections to other WarehousePG databases. - [fuzzystrmatch](fuzzystrmatch.md) - Determines similarities and differences between strings. - [gp_array_agg](gp_array_agg.md) - Implements a parallel `array_agg()` aggregate function for WarehousePG. - [gp_check_functions](gp_check_functions.md) - Provides views to check for orphaned and missing relation files and a user-defined function to move orphaned files. +- gp_distribution_policy - Provides access to a table's distribution policy information. +- gp_internal_tools - Provides internal diagnostic functions for WarehousePG support and engineering use. - [gp_legacy_string_agg](gp_legacy_string_agg.md) - Implements a legacy, single-argument `string_agg()` aggregate function that was present in WarehousePG 5. - [gp_parallel_retrieve_cursor](gp_parallel_retrieve_cursor.md) - Provides extended cursor functionality to retrieve data, in parallel, directly from WarehousePG segments. - [gp_percentile_agg](gp_percentile_agg.md) - Improves GPORCA performance for ordered-set aggregate functions. - [gp_pitr](gp_pitr.md) - Supports implementing point-in-time recovery for WarehousePG 6. - [gp_sparse_vector](gp_sparse_vector.md) - Implements a data type that uses compressed storage of zeros to make vector computations on floating point numbers faster. - [gp_subtransaction_overflow](gp_subtransaction_overflow.md) - Provides a view and user-defined function for querying suboverflowed backends. -- [greenplum_fdw](greenplum_fdw.md) - Provides a foreign data wrapper for accessing data stored in one or more external WarehousePG clusters. - [hstore](hstore.md) - Provides a data type for storing sets of key/value pairs within a single value. -- [ip4r](ip4r.md) - Provides data types for operations on IPv4 and IPv6 IP addresses. - [isn](isn.md) - Provides support for international product numbering standards (EAN13, UPC, ISBN, ISMN, ISSN). - [ltree](ltree.md) - Provides data types for representing labels of data stored in a hierarchical tree-like structure. - [orafce](orafce_ref.md) - Provides Oracle SQL compatibility functions. - [pageinspect](pageinspect.md) - Provides functions for low-level inspection of database page contents. Available to superusers only. -- [pg_cron](pg_cron.md) - Provides a cron-based job scheduler that runs inside the database. +- pg_buffercache - Provides access to views for obtaining cluster-wide shared buffer metrics. - [pg_trgm](pg_trgm.md) - Provides functions and operators for determining the similarity of alphanumeric text based on trigram matching. - [pgcrypto](pgcrypto.md) - Provides cryptographic functions. +- [PL/Perl](../../../admin_guide/analytics/procedural_languages/pl_perl.md) - Enables writing functions and triggers in Perl. +- plperlu - Provides an untrusted version of PL/Perl that allows unrestricted operations, such as file and network access. +- [PL/pgSQL](../../../admin_guide/analytics/procedural_languages/pl_sql.md) - Enables writing functions, triggers, and procedural code in a SQL-like language. +- [PL/Python](../../../admin_guide/analytics/procedural_languages/pl_python.md) - Enables writing functions in Python. Enabled by default as `plpythonu` (Python 2). +- plpython2u - Provides an explicit Python 2 version of the untrusted PL/Python language. +- plpython3u - Provides an explicit Python 3 version of the untrusted PL/Python language. - [postgres_fdw](postgres_fdw.md) - Provides a foreign data wrapper for accessing data stored in an external PostgreSQL or WarehousePG database. - [sslinfo](sslinfo.md) - Provides information about the SSL certificate of the current client connection. - [tablefunc](tablefunc.md) - Provides various functions that return tables (multiple rows). -- [timestamp9](timestamp9.md) - Provides an efficient nanosecond-precision timestamp data type. - [uuid-ossp](uuid-ossp.md) - Provides functions to generate universally unique identifiers (UUIDs). diff --git a/docs/whpg/6x/ref_guide/modules/external_components/index.md b/docs/whpg/6x/ref_guide/modules/external_components/index.md index 0d95a2b..5dadc92 100644 --- a/docs/whpg/6x/ref_guide/modules/external_components/index.md +++ b/docs/whpg/6x/ref_guide/modules/external_components/index.md @@ -6,6 +6,6 @@ description: Separately installed tools and frameworks that integrate with Wareh These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`. -- [WarehousePG Platform Extension Framework (PXF)](../../../install_guide/data_sci_pkgs/install_pxf.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop. -- whpg-backup - Provides parallel backup and restore for WarehousePG clusters. -- whpg-backup-s3-plugin - Adds Amazon S3 support to whpg-backup. +- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop. +- [whpg-backup](/whpg-backup/index.md) - Provides parallel backup and restore for WarehousePG clusters. +- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Adds Amazon S3 support to whpg-backup. diff --git a/docs/whpg/6x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md b/docs/whpg/6x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md index 8321f7a..24ad530 100644 --- a/docs/whpg/6x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md +++ b/docs/whpg/6x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md @@ -184,7 +184,7 @@ See [Working with External Data](../../admin_guide/external/index.md) for detail - LOCATION ('protocol://\[host\[:port]]/path/file' \[, ...]) - If you use the `pxf` protocol to access an external data source, refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about the `pxf` protocol. + If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about the `pxf` protocol. If you use the `s3` protocol to read or write to S3, refer to [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md) for additional information about the `s3` protocol `LOCATION` clause syntax. @@ -240,13 +240,13 @@ See [Working with External Data](../../admin_guide/external/index.md) for detail When the `FORMAT` clause identfies delimited text (`TEXT`) or comma separated values (`CSV`) format, formatting options are similar to those available with the PostgreSQL [COPY](COPY.md) command. If the data in the file does not use the default column delimiter, escape character, null string and so on, you must specify the additional formatting options so that the data in the external file is read correctly by WarehousePG. For information about using a custom format, see "Loading and Unloading Data" in the *WarehousePG Administrator Guide*. - If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF. + If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF. - FORMAT 'CUSTOM' (formatter=formatter_specification) Specifies a custom data format. The formatter_specification specifies the function to use to format the data, followed by comma-separated parameters to the formatter function. The length of the formatter specification, the string including `Formatter=`, can be up to approximately 50K bytes. - If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF. + If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF. For general information about using a custom format, see "Loading and Unloading Data" in the *WarehousePG Administrator Guide*. diff --git a/docs/whpg/6x/ref_guide/sql_commands/CREATE_PROTOCOL.md b/docs/whpg/6x/ref_guide/sql_commands/CREATE_PROTOCOL.md index 29d6eee..30801bf 100644 --- a/docs/whpg/6x/ref_guide/sql_commands/CREATE_PROTOCOL.md +++ b/docs/whpg/6x/ref_guide/sql_commands/CREATE_PROTOCOL.md @@ -67,7 +67,7 @@ For information about creating and enabling a custom data access protocol, see " ## Notes -WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md#configuring-the-s3-protocol) for information about enabling the `S3` protocol. Refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about using the `pxf` protocol. +WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md#configuring-the-s3-protocol) for information about enabling the `S3` protocol. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using the `pxf` protocol. Any shared library that implements a data access protocol must be located in the same location on all WarehousePG segment hosts. For example, the shared library can be in a location specified by the operating system environment variable `LD_LIBRARY_PATH` on all hosts. You can also specify the location when you define the handler function. For example, when you define the `s3` protocol in the `CREATE PROTOCOL` command, you specify `$libdir/gps3ext.so` as the location of the shared object, where `$libdir` is located at `$GPHOME/lib`. diff --git a/docs/whpg/6x/security-guide/SecuringGPDB.md b/docs/whpg/6x/security-guide/SecuringGPDB.md index bc10c5a..a4ce28b 100644 --- a/docs/whpg/6x/security-guide/SecuringGPDB.md +++ b/docs/whpg/6x/security-guide/SecuringGPDB.md @@ -24,7 +24,7 @@ The essential security requirements fall into the following categories: ## Accessing a Kerberized Hadoop Cluster -You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to Configuring PXF for Secure HDFS for the procedure to perform this setup. +You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to [Authenticating with Kerberos](/pxf/6x/connecting/hadoop/kerberos.md) for the procedure to perform this setup. diff --git a/docs/whpg/7x/admin_guide/analytics/index.md b/docs/whpg/7x/admin_guide/analytics/index.md index e34cd14..49a95e4 100644 --- a/docs/whpg/7x/admin_guide/analytics/index.md +++ b/docs/whpg/7x/admin_guide/analytics/index.md @@ -65,7 +65,7 @@ The WarehousePG analytics capabilities allow you to: - Leverage existing SQL knowledge: WarehousePG can run dozens of statistical, machine learning, and graph methods, via SQL. - Train more models in less time by taking advantage of the parallelism in the MPP architecture and in-database analytics. - Access the data where it lives, therefore integrate data and analytics in one place. WarehousePG is infrastructure-agnostic and runs on bare metal, private cloud, and public cloud deployments. -- Use a multitude of data extensions. WarehousePG supports extensions for HDFS, Hive, and HBase as well as reading/writing data from/to cloud storage, including Amazon S3 objects. Review the capabilities of the WarehousePG [Platform Extension Framework (PXF)](../external/pxf-overview.md), which provides *connectors* that enable you to access data stored in sources external to your WarehousePG deployment. +- Use a multitude of data extensions. WarehousePG supports extensions for HDFS, Hive, and HBase as well as reading/writing data from/to cloud storage, including Amazon S3 objects. Review the capabilities of the WarehousePG [Platform Extension Framework (PXF)](/pxf/6x/index.md), which provides *connectors* that enable you to access data stored in sources external to your WarehousePG deployment. - Use familiar and leading BI and advanced analytics software that are ODBC/JDBC compatible, or have native integrations, including SAS, IBM Cognos, SAP Analytics Solutions, Qlik, Tableau, Apache Zeppelin, and Jupyter. - Run deep learning algorithms using popular frameworks like Keras and TensorFlow in an MPP relational database, with GPU (Graphical Processing Unit) acceleration. - Use containers capable of isolating executors from the host OS. WarehousePG PL/Container implements a trusted language execution engine which permits customized data science workloads or environments created for different end user workloads. diff --git a/docs/whpg/7x/admin_guide/analytics/procedural_languages/pl_r.md b/docs/whpg/7x/admin_guide/analytics/procedural_languages/pl_r.md index 8303b72..50ef75c 100644 --- a/docs/whpg/7x/admin_guide/analytics/procedural_languages/pl_r.md +++ b/docs/whpg/7x/admin_guide/analytics/procedural_languages/pl_r.md @@ -103,7 +103,6 @@ PL/R is registered as an untrusted language. ### Uninstalling PL/R - [Remove PL/R Support for a Database](#topic7) -- [Uninstall R (Ubuntu)](#topic_ifv_tsf_w3b) When you remove PL/R language support from a database, the PL/R routines that you created in the database will no longer work. @@ -119,19 +118,6 @@ $ psql -d testdb -c 'DROP EXTENSION plr;' The default command fails if any existing objects (such as functions) depend on the language. Specify the `CASCADE` option to also drop all dependent objects, including functions that you created with PL/R. - - -#### Uninstall R (Ubuntu) - -For Ubuntu systems, remove R from all WarehousePG host systems. These commands remove R from an Ubuntu system. - -``` -$ sudo apt remove r-base -$ sudo apt remove r-base-core -``` - -Removing `r-base` does not uninstall the R executable. Removing `r-base-core` uninstalls the R executable. - ### Examples diff --git a/docs/whpg/7x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md b/docs/whpg/7x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md index 19437f4..ef8c5f9 100644 --- a/docs/whpg/7x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md +++ b/docs/whpg/7x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md @@ -13,6 +13,6 @@ Creates a readable external table, *ext_expenses,* using the `pxf` protocol. The ``` -Refer to [Accessing External Data with PXF](../../pxf-overview.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System. +Refer to [HDFS](/pxf/6x/connecting/hadoop/hdfs.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System. **Parent topic:** [Examples for Creating External Tables](index.md) diff --git a/docs/whpg/7x/admin_guide/external/external-tables/pxf-protocol.md b/docs/whpg/7x/admin_guide/external/external-tables/pxf-protocol.md index 41b85f9..a95a35e 100644 --- a/docs/whpg/7x/admin_guide/external/external-tables/pxf-protocol.md +++ b/docs/whpg/7x/admin_guide/external/external-tables/pxf-protocol.md @@ -11,6 +11,6 @@ When you use the `pxf` protocol to query an external data store, you specify the You must explicitly initialize and start PXF before you can use the `pxf` protocol to read or write external data. You must also enable PXF in each database in which you want to allow users to create external tables to access external data, and grant permissions on the `pxf` protocol to those WarehousePG users. -For detailed information about configuring and using PXF and the `pxf` protocol, refer to [Accessing External Data with PXF](../pxf-overview.md). +For detailed information about configuring and using PXF and the `pxf` protocol, refer to [PXF for WarehousePG](/pxf/6x/index.md). **Parent topic:** [Defining External Tables](index.md) diff --git a/docs/whpg/7x/admin_guide/external/foreign/index.md b/docs/whpg/7x/admin_guide/external/foreign/index.md index 8b08cbd..9c2e479 100644 --- a/docs/whpg/7x/admin_guide/external/foreign/index.md +++ b/docs/whpg/7x/admin_guide/external/foreign/index.md @@ -9,7 +9,7 @@ WarehousePG implements portions of the SQL/MED specification, allowing you to ac You can access foreign data with help from a *foreign-data wrapper*. A foreign-data wrapper is a library that communicates with a remote data source. This library hides the source-specific connection and data access details. -The WarehousePG distribution includes the [postgres_fdw](../../../ref_guide/modules/bundled/postgres_fdw.md) foreign data wrapper. +The WarehousePG distribution includes the [postgres_fdw](../../../ref_guide/modules/bundled/postgres_fdw.md) foreign data wrapper. If you use PXF to access external data, the [pxf_fdw](/pxf/6x/foreign-data-wrapper.md) foreign data wrapper lets you query those same PXF-connected sources through standard foreign tables instead of `pxf://` external tables. If none of the existing PostgreSQL or WarehousePG foreign-data wrappers suit your needs, you can write your own as described in [Writing a Foreign Data Wrapper](devel-fdw.md). diff --git a/docs/whpg/7x/admin_guide/external/index.md b/docs/whpg/7x/admin_guide/external/index.md index 55b886e..8fb8a46 100644 --- a/docs/whpg/7x/admin_guide/external/index.md +++ b/docs/whpg/7x/admin_guide/external/index.md @@ -3,7 +3,6 @@ title: Working with External Data navigation: - foreign - external-tables - - pxf-overview - using-gpfdist redirects: - working-with-file-based-ext-tables @@ -48,4 +47,4 @@ Web-based external tables provide access to data served by an HTTP server or an ## About Accessing External Data with pxf -Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. Refer to [Accessing External Data with PXF](pxf-overview.md) for more information about using PXF. +Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for more information about using PXF. diff --git a/docs/whpg/7x/admin_guide/external/pxf-overview.md b/docs/whpg/7x/admin_guide/external/pxf-overview.md deleted file mode 100644 index a586bd9..0000000 --- a/docs/whpg/7x/admin_guide/external/pxf-overview.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: Accessing External Data with PXF - ---- - -Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. - -PXF is installed with Hadoop and Object Storage connectors. These connectors enable you to read and write external data stored in text, Avro, JSON, RCFile, Parquet, SequenceFile, and ORC formats. You can use the JDBC connector to access an external SQL database. - -The WarehousePG Platform Extension Framework includes a C-language extension and a Java service. After you configure and initialize PXF, you start a single PXF JVM process on each WarehousePG segment host. This long- running process concurrently serves multiple query requests. - -For detailed information about the architecture of and using PXF, refer to the WarehousePG Platform Extension Framework PXF documentation - -**Parent topic:** [Working with External Data](index.md) diff --git a/docs/whpg/7x/admin_guide/load/index.md b/docs/whpg/7x/admin_guide/load/index.md index 562afdb..319e17e 100644 --- a/docs/whpg/7x/admin_guide/load/index.md +++ b/docs/whpg/7x/admin_guide/load/index.md @@ -28,7 +28,7 @@ WarehousePG can read from and write to several types of external data sources, i - External web tables provide access to dynamic data. They can be backed with data from URLs accessed using the HTTP protocol or by the output of an OS script running on one or more segments. - The `gpfdist` utility is the WarehousePG parallel file distribution program. It is an HTTP server that is used with external tables to allow WarehousePG segments to load external data in parallel, from multiple file systems. You can run multiple instances of `gpfdist` on different hosts and network interfaces and access them in parallel. - The `gpload` utility automates the steps of a load task using `gpfdist` and a YAML-formatted control file. -- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [Accessing External Data with PXF](../external/pxf-overview.md). +- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md). The method you choose to load data depends on the characteristics of the source data—its location, size, format, and any transformations required. diff --git a/docs/whpg/7x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md b/docs/whpg/7x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md index 91e9f1c..f415fa9 100644 --- a/docs/whpg/7x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md +++ b/docs/whpg/7x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md @@ -8,7 +8,7 @@ navigation: Writable external tables that output data to files can use the WarehousePG parallel file server program, gpfdist, or the WarehousePG Platform Extension Framework (PXF), WarehousePG's interface to Hadoop. -Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [Accessing External Data with PXF](../../../external/pxf-overview.md) for instructions on setting up PXF for use with an external table +Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [PXF for WarehousePG](/pxf/6x/index.md) for instructions on setting up PXF for use with an external table - With a writable external table using the gpfdist protocol, the WarehousePG segments send their data to gpfdist, which writes the data to the named file. gpfdist must run on a host that the WarehousePG segments can access over the network. gpfdist points to a file location on the output host and writes data received from the WarehousePG segments to the file. To divide the output data among multiple files, list multiple gpfdist URIs in your writable external table definition. - A writable external web table sends data to an application as a stream of data. For example, unload data from WarehousePG and send it to an application that connects to another database or ETL tool to load the data elsewhere. Writable external web tables use the `EXECUTE` clause to specify a shell command, script, or application to run on the segment hosts and accept an input stream of data. See [Defining a Command-Based Writable External Web Table](../defining-a-command-based-writable-external-web-table/index.md) for more information about using `EXECUTE` commands in a writable external table definition. diff --git a/docs/whpg/7x/admin_guide/manage_access/client_auth/index.md b/docs/whpg/7x/admin_guide/manage_access/client_auth/index.md index 389139b..b8a4768 100644 --- a/docs/whpg/7x/admin_guide/manage_access/client_auth/index.md +++ b/docs/whpg/7x/admin_guide/manage_access/client_auth/index.md @@ -207,6 +207,26 @@ gpconfig -c ssl -m on -v off Setting the parameter requires a server restart. This command restarts the system: `gpstop -ra`. + + +### Requiring a Minimum TLS Version + +WarehousePG negotiates the highest TLS protocol version that both the client and the coordinator support. When both endpoints run OpenSSL 1.1.1 or later, connections use TLS 1.3 automatically, with no additional configuration required. + +To require a minimum TLS protocol version instead of relying on automatic negotiation, set the server configuration parameter `ssl_min_protocol_version` on the coordinator and standby coordinator hosts. For example, this `gpconfig` command requires TLS 1.3 for all connections: + +``` +gpconfig -c ssl_min_protocol_version -v 'TLSv1.3' +``` + +Setting the parameter requires a server restart. Run `gpstop -ra` to restart the system. + +Valid values for `ssl_min_protocol_version` are `TLSv1`, `TLSv1.1`, `TLSv1.2`, and `TLSv1.3`. A related parameter, `ssl_max_protocol_version`, sets the maximum protocol version and accepts the same values, plus an empty string to allow the latest version that both endpoints support with no upper bound. For more information about both parameters, see [ssl_min_protocol_version](../../../ref_guide/config_params/guc-list.md#ssl_min_protocol_version) and [ssl_max_protocol_version](../../../ref_guide/config_params/guc-list.md#ssl_max_protocol_version) in the *WarehousePG Reference Guide*. + +::: info Note +The `ssl_min_protocol_version` and `ssl_max_protocol_version` parameters and the `pg_stat_ssl` system view are not available in WarehousePG 6. +::: + ### Creating a Self-signed Certificate without a Passphrase for Testing Only @@ -214,7 +234,7 @@ Setting the parameter requires a server restart. This command restarts the syste To create a quick self-signed certificate for the server for testing, use the following OpenSSL command: ``` -# openssl req -new -text -out server.req +openssl req -new -text -out server.req ``` @@ -225,8 +245,8 @@ The program will generate a key that is passphrase protected, and does not accep To use this certificate with WarehousePG, remove the passphrase with the following commands: ``` -# openssl rsa -in privkey.pem -out server.key -# rm privkey.pem +openssl rsa -in privkey.pem -out server.key +rm privkey.pem ``` Enter the old passphrase when prompted to unlock the existing key. @@ -234,13 +254,13 @@ Enter the old passphrase when prompted to unlock the existing key. Then, enter the following command to turn the certificate into a self-signed certificate and to copy the key and certificate to a location where the server will look for them. ``` -# openssl req -x509 -in server.req -text -key server.key -out server.crt +openssl req -x509 -in server.req -text -key server.key -out server.crt ``` Finally, change the permissions on the key with the following command. The server will reject the file if the permissions are less restrictive than these. ``` -# chmod og-rwx server.key +chmod og-rwx server.key ``` For more details on how to create your server private key and certificate, refer to the [OpenSSL documentation](https://www.openssl.org/docs/). diff --git a/docs/whpg/7x/admin_guide/managing/compression.md b/docs/whpg/7x/admin_guide/managing/compression.md index e230b1b..b306388 100644 --- a/docs/whpg/7x/admin_guide/managing/compression.md +++ b/docs/whpg/7x/admin_guide/managing/compression.md @@ -9,7 +9,7 @@ You can configure support for data compression with these features and utilities - Append-optimized tables support compressing table data. See [CREATE TABLE](../../ref_guide/sql_commands/CREATE_TABLE.md). - User-defined data types can be defined to compress data. See [CREATE TYPE](../../ref_guide/sql_commands/CREATE_TYPE.md). -- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](../external/pxf-overview.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md). +- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](/pxf/6x/index.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md). - Workfiles (temporary spill files that are created when running a query that requires more memory than it is allocated) can be compressed. See the server configuration parameter [gp_workfile_compression](../../ref_guide/config_params/guc-list.md). - The WarehousePG utilities `gpbackup`, `gprestore`, `gpload`, `gplogfilter` support compression. diff --git a/docs/whpg/7x/admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md b/docs/whpg/7x/admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md index 0a7829c..bb926de 100644 --- a/docs/whpg/7x/admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md +++ b/docs/whpg/7x/admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md @@ -216,12 +216,6 @@ You do not need to change your version of cgroup, you can simply skip to [Config ``` grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="systemd.unified_cgroup_hierarchy=1" ``` -- Ubuntu systems: - ``` - vim /etc/default/grub - # add or modify: GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1" - update-grub - ``` If you want to switch from cgroup v2 to v1, run the following commands as root: @@ -229,12 +223,6 @@ If you want to switch from cgroup v2 to v1, run the following commands as root: ``` grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller" ``` -- Ubuntu systems: - ``` - vim /etc/default/grub - # add or modify: GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0" - update-grub - ``` After that, reboot your host in order for the changes to take effect. diff --git a/docs/whpg/7x/install_guide/additional_modules/external_components/index.md b/docs/whpg/7x/install_guide/additional_modules/external_components/index.md index 7c5d39d..1b7ab66 100644 --- a/docs/whpg/7x/install_guide/additional_modules/external_components/index.md +++ b/docs/whpg/7x/install_guide/additional_modules/external_components/index.md @@ -8,6 +8,6 @@ navigation: These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`. -- [WarehousePG Platform Extension Framework (PXF)](install_pxf.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop. -- whpg-backup - Parallel backup and restore for WarehousePG clusters. -- whpg-backup-s3-plugin - Amazon S3 support for whpg-backup. +- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop. +- [whpg-backup](/whpg-backup/index.md) - Parallel backup and restore for WarehousePG clusters. +- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Amazon S3 support for whpg-backup. diff --git a/docs/whpg/7x/install_guide/additional_modules/external_components/install_pxf.md b/docs/whpg/7x/install_guide/additional_modules/external_components/install_pxf.md index 9f0db6f..fe15024 100644 --- a/docs/whpg/7x/install_guide/additional_modules/external_components/install_pxf.md +++ b/docs/whpg/7x/install_guide/additional_modules/external_components/install_pxf.md @@ -5,6 +5,6 @@ title: WarehousePG Platform Extension Framework (PXF) *Optional.* If you do not plan to use PXF, no action is necessary. -If you plan to use PXF, refer to [Accessing External Data with PXF](../../../admin_guide/external/pxf-overview.md) for introductory PXF information. +If you plan to use PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md) for installation and usage information. **Parent topic:** [Installing Optional Extensions (WarehousePG)](index.md) diff --git a/docs/whpg/7x/install_guide/config_os.md b/docs/whpg/7x/install_guide/config_os.md index 4fc25e2..f3516fd 100644 --- a/docs/whpg/7x/install_guide/config_os.md +++ b/docs/whpg/7x/install_guide/config_os.md @@ -386,11 +386,11 @@ The XFS options can also be set in the `/etc/fstab` file. This example entry fro | Storage Device Type | OS | Recommended Scheduler Policy | | :--------------------------------- | :----------------------------------------- | :--------------------------- | - | Non-Volatile Memory Express (NVMe) | RHEL 7
RHEL 8
RHEL 9
Ubuntu | `none` | - | Solid-State Drives (SSD) | RHEL 7 | `noop` | - | Solid-State Drives (SSD) | RHEL 8
RHEL 9
Ubuntu | `none` | - | Other | RHEL 7 | `deadline` | - | Other | RHEL 8
RHEL 9
Ubuntu | `mq-deadline` | + | Non-Volatile Memory Express (NVMe) | RHEL 7
RHEL 8
RHEL 9 | `none` | + | Solid-State Drives (SSD) | RHEL 7 | `noop` | + | Solid-State Drives (SSD) | RHEL 8
RHEL 9 | `none` | + | Other | RHEL 7 | `deadline` | + | Other | RHEL 8
RHEL 9 | `mq-deadline` | To specify a scheduler until the next system reboot, run the following: @@ -461,12 +461,6 @@ On systems that use `grub2`, use the system utility `grubby`. This command adds After adding the parameter, reboot the system. -For Ubuntu systems, install the `hugepages` package and run this command as root: - -``` -# hugeadm --thp-never -``` - This cat command checks the state of THP. The output indicates that THP is deactivated. ``` @@ -646,7 +640,7 @@ The following steps show how to set up the `gpadmin` user on a host, set a passw 1. Create the `gpadmin` group and user. - > **Note** If you are installing WarehousePG on RHEL 7.2 or CentOS 7.2 and want to deactivate IPC object removal by creating the `gpadmin` user as a system account, provide both the `-r` option (create the user as a system account) and the `-m` option (create a home directory) to the `useradd` command. On Ubuntu systems, you must use the `-m` option with the `useradd` command to create a home directory for a user. + > **Note** If you are installing WarehousePG on RHEL 7.2 or CentOS 7.2 and want to deactivate IPC object removal by creating the `gpadmin` user as a system account, provide both the `-r` option (create the user as a system account) and the `-m` option (create a home directory) to the `useradd` command. This example creates the `gpadmin` group, creates the `gpadmin` user as a system account with a home directory and as a member of the `gpadmin` group, and creates a password for the user. diff --git a/docs/whpg/7x/install_guide/install_whpg.md b/docs/whpg/7x/install_guide/install_whpg.md index 07c5cb0..14de8b7 100644 --- a/docs/whpg/7x/install_guide/install_whpg.md +++ b/docs/whpg/7x/install_guide/install_whpg.md @@ -45,7 +45,7 @@ Before you begin installing WarehousePG, be sure you have completed the steps in Follow these instructions to install WarehousePG from a pre-built binary. -> **Important** You require sudo or root user access to install from a pre-built RPM or DEB file. +> **Important** You require sudo or root user access to install from a pre-built RPM file. 1. Download and copy the WarehousePG package to the `gpadmin` user's home directory on the coordinator, standby coordinator, and every segment host machine. The distribution file name has the format `greenplum-db--.rpm` for RHEL, Oracle Linux, or Rocky Linux systems. diff --git a/docs/whpg/7x/install_guide/platform-requirements.md b/docs/whpg/7x/install_guide/platform-requirements.md index a89e521..c271c06 100644 --- a/docs/whpg/7x/install_guide/platform-requirements.md +++ b/docs/whpg/7x/install_guide/platform-requirements.md @@ -19,7 +19,7 @@ WarehousePG 7 runs on the following operating system platforms: > **Caution** A kernel issue in Red Hat Enterprise Linux 8.5 and 8.6 can cause I/O freezes and synchronization problems with XFS filesystems. This issue is fixed in RHEL 8.7. See [RHEL8: xfs_buf deadlock between inode deletion and block allocation](https://access.redhat.com/solutions/6984334). -WarehousePG server supports TLS version 1.2 on RHEL/CentOS systems, and TLS version 1.3 on Ubuntu systems. +WarehousePG server supports TLS version 1.2 and TLS version 1.3 on RHEL/CentOS systems. @@ -133,7 +133,7 @@ This table lists the versions of the WarehousePG Extensions that are compatible | Component | Package Version | Additional Information | |-----------|----------------|----------------------| | [PL/Java](../admin_guide/analytics/procedural_languages/pl_java.md) | 2.0.7 | Supports Java 8 and 11. | -| [PL/R](../admin_guide/analytics/procedural_languages/pl_r.md) | 3.1.1 | (CentOS) R 3.3.3
(Ubuntu) You install R 3.5.1+. | +| [PL/R](../admin_guide/analytics/procedural_languages/pl_r.md) | 3.1.1 | R 3.3.3 | | PL/Container Image for R | 2.1.2 | R 3.6.3 | | PL/Container Images for Python | 2.1.2 | Python 2.7.12
Python 3.7 | | [MADlib Machine Learning](../admin_guide/analytics/madlib.md) | 2.1.0 | Support matrix at [MADlib FAQ](https://cwiki.apache.org/confluence/display/MADLIB/FAQ#FAQ-Q1-2WhatdatabaseplatformsdoesMADlibsupportandwhatistheupgradematrix?). | @@ -224,7 +224,7 @@ The disk settings for cloud deployments are the same as on-premise with a few mo ``` rw,noatime,nobarrier,nodev,inode64 ``` - > **Note** The `nobarrier` option is not supported on RHEL 8 or Ubuntu nodes. + > **Note** The `nobarrier` option is not supported on RHEL 8 nodes. - Use mq-deadline instead of the deadline scheduler for the R5 series instance type in AWS - Use a swap disk per VM (32GB size works well) diff --git a/docs/whpg/7x/ref_guide/config_params/guc-list.md b/docs/whpg/7x/ref_guide/config_params/guc-list.md index 7ed9a97..24d51a7 100644 --- a/docs/whpg/7x/ref_guide/config_params/guc-list.md +++ b/docs/whpg/7x/ref_guide/config_params/guc-list.md @@ -3810,6 +3810,30 @@ See the openssl manual page for a list of supported ciphers. | ----------- | --------------------------------- | ---------------------------- | | string | ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH | coordinator, system, restart | + + +## ssl_max_protocol_version + +Sets the maximum TLS protocol version that WarehousePG accepts for secure connections. An empty string, the default, allows any protocol version up to the most recent version supported by the OpenSSL library used to build WarehousePG. + +| Value Range | Default | Set Classifications | +| --------------------------------------------------- | ------- | ----------------------------- | +| TLSv1, TLSv1.1, TLSv1.2, TLSv1.3, or an empty string | | coordinator, system, restart | + + + +## ssl_min_protocol_version + +Sets the minimum TLS protocol version that WarehousePG accepts for secure connections. WarehousePG rejects connections that negotiate an earlier protocol version. See [Requiring a Minimum TLS Version](../../admin_guide/manage_access/client_auth/index.md#topic_ssl_min_version) in the *WarehousePG Administrator Guide* for an example that requires TLS 1.3. + +::: info Note +This parameter is not available in WarehousePG 6. +::: + +| Value Range | Default | Set Classifications | +| ------------------------------------- | ------- | ----------------------------- | +| TLSv1, TLSv1.1, TLSv1.2, or TLSv1.3 | TLSv1.2 | coordinator, system, restart | + ## standard_conforming_strings diff --git a/docs/whpg/7x/ref_guide/modules/bundled/index.md b/docs/whpg/7x/ref_guide/modules/bundled/index.md index e46a456..2161ac6 100644 --- a/docs/whpg/7x/ref_guide/modules/bundled/index.md +++ b/docs/whpg/7x/ref_guide/modules/bundled/index.md @@ -2,7 +2,6 @@ title: Bundled Modules description: Modules that ship with WarehousePG and don't require a separate package install. navigation: - - auto-explain - btree_gin - citext - dblink @@ -22,56 +21,43 @@ navigation: - postgres_fdw - sslinfo - tablefunc - - tsm_system_rows - - tsm_system_time - uuid-ossp --- These modules ship with WarehousePG. Activate them in each database with `CREATE EXTENSION `, unless the individual reference page specifies a different activation method. -- adminpack - Provides administrative support functions used by management tools such as pgAdmin. -- amcheck - Provides functions for verifying the logical consistency of indexes and heap relations. -- [auto_explain](auto-explain.md) - Logs execution plans of slow statements automatically. -- bloom - Implements a Bloom filter-based index access method for equality comparisons on multiple columns. - [btree_gin](btree_gin.md) - Provides GIN operator classes that implement B-tree equivalent behavior for certain data types. -- btree_gist - Provides GiST operator classes that implement B-tree equivalent behavior for a range of data types. - [citext](citext.md) - Provides a case-insensitive, multibyte-aware text data type. -- cube - Provides a data type for representing multidimensional cubes. - [dblink](dblink.md) - Provides connections to other WarehousePG databases. -- dict_int - Provides a text search dictionary template that controls indexing of integers. -- dict_xsyn - Provides a text search dictionary template for extended synonym processing. -- earthdistance - Provides two approaches to calculating great-circle distances on Earth's surface. - file_fdw - Provides a foreign-data wrapper for accessing data files in the server's file system. +- [fuzzystrmatch](fuzzystrmatch.md) - Provides functions to determine similarities and differences between strings. +- gp_debug_numsegments - Provides functions for testing and debugging cluster segment count behavior. +- gp_distribution_policy - Provides access to a table's distribution policy information. - [gp_exttable_fdw](gp_exttable_fdw.md) - Built-in foreign-data wrapper that internally converts an external table to a foreign table. +- gp_inject_fault - Provides fault injection points for testing cluster fault tolerance and recovery. +- gp_internal_tools - Provides internal diagnostic functions for WarehousePG support and engineering use. - [gp_legacy_string_agg](gp_legacy_string_agg.md) - Implements a legacy, single-argument `string_agg()` aggregate function. +- gp_replica_check - Provides functions for verifying consistency between primary and mirror segments. - [gp_sparse_vector](gp_sparse_vector.md) - Implements a data type that uses compressed storage of zeros to make vector computations on floating point numbers faster. +- [gp_toolkit](../../gp_toolkit.md) - Provides administrative views and functions for monitoring database status, such as skew, locks, and disk usage. - [hstore](hstore.md) - Provides a data type for storing sets of key/value pairs within a single value. - intarray - Provides additional functions, operators, and index support for arrays of integers with no null elements. - [isn](isn.md) - Provides data types for international product numbering standards (EAN13, UPC, ISBN, ISMN, ISSN). -- lo - Provides support for managing large objects. - [ltree](ltree.md) - Provides data types for representing labels of data stored in a hierarchical tree-like structure. - [orafce](orafce_ref.md) - Provides Oracle SQL compatibility functions. - [pageinspect](pageinspect.md) - Provides functions for low-level inspection of the contents of database pages. Available to superusers only. - [pg_buffercache](pg_buffercache.md) - Provides access to views for obtaining cluster-wide shared buffer metrics. -- pg_freespacemap - Provides a function for examining the free space map. - pg_hint_plan - Allows controlling query execution plans using hints specified in SQL comments. -- pg_prewarm - Provides functions for loading relation data into the buffer cache. - [pg_stat_statements](pg_stat_statements.md) - Tracks execution statistics of all SQL statements. Requires `shared_preload_libraries`. - [pg_trgm](pg_trgm.md) - Provides functions and operators for determining the similarity of alphanumeric text based on trigram matching. -- pg_visibility - Provides functions for examining the visibility map and page-level visibility information of a relation. - [pgcrypto](pgcrypto.md) - Provides cryptographic functions. -- pgrowlocks - Provides a function that shows row locking information for a given table. -- pgstattuple - Provides functions for obtaining tuple-level and index-level statistics. - [PL/Perl](../../../admin_guide/analytics/procedural_languages/pl_perl.md) - Enables writing functions and triggers in Perl. +- plperlu - Provides an untrusted version of PL/Perl that allows unrestricted operations, such as file and network access. - [PL/pgSQL](../../../admin_guide/analytics/procedural_languages/pl_sql.md) - Enables writing functions, triggers, and procedural code in a SQL-like language. - [PL/Python](../../../admin_guide/analytics/procedural_languages/pl_python.md) - Enables writing functions in Python. - [postgres_fdw](postgres_fdw.md) - Provides a foreign data wrapper for accessing data stored in an external Postgres database. -- seg - Provides a data type for representing line segments or floating-point intervals. - [sslinfo](sslinfo.md) - Provides information about the SSL certificate of the current client connection. - [tablefunc](tablefunc.md) - Provides various functions that return tables (multiple rows). -- [tsm_system_rows](tsm_system_rows.md) - Implements the `SYSTEM_ROWS` table sampling method. -- [tsm_system_time](tsm_system_time.md) - Implements the `SYSTEM_TIME` table sampling method. - unaccent - Provides a text search dictionary that removes accents from lexemes. - [uuid-ossp](uuid-ossp.md) - Provides functions to generate universally unique identifiers (UUIDs). -- xml2 - Provides XPath querying and XSLT functionality. diff --git a/docs/whpg/7x/ref_guide/modules/external_components/index.md b/docs/whpg/7x/ref_guide/modules/external_components/index.md index ee96d3f..aecfa4a 100644 --- a/docs/whpg/7x/ref_guide/modules/external_components/index.md +++ b/docs/whpg/7x/ref_guide/modules/external_components/index.md @@ -6,6 +6,6 @@ description: Separately installed tools and frameworks that integrate with Wareh These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`. -- [WarehousePG Platform Extension Framework (PXF)](../../../install_guide/additional_modules/external_components/install_pxf.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop. -- whpg-backup - Provides parallel backup and restore for WarehousePG clusters. -- whpg-backup-s3-plugin - Adds Amazon S3 support to whpg-backup. +- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop. +- [whpg-backup](/whpg-backup/index.md) - Provides parallel backup and restore for WarehousePG clusters. +- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Adds Amazon S3 support to whpg-backup. diff --git a/docs/whpg/7x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md b/docs/whpg/7x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md index 23703be..33e380c 100644 --- a/docs/whpg/7x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md +++ b/docs/whpg/7x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md @@ -180,7 +180,7 @@ The data type of the column. ##### LOCATION ('protocol://\[host\[:port]]/path/file' \[, ...]) -If you use the `pxf` protocol to access an external data source, refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about the `pxf` protocol. +If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about the `pxf` protocol. If you use the `s3` protocol to read or write to S3, refer to [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md) for additional information about the `s3` protocol `LOCATION` clause syntax. @@ -233,13 +233,13 @@ For writable external tables, the command specified in the `EXECUTE` clause must When the `FORMAT` clause identfies delimited text (`TEXT`) or comma separated values (`CSV`) format, formatting options are similar to those available with the PostgreSQL [COPY](COPY.md) command. If the data in the file does not use the default column delimiter, escape character, null string and so on, you must specify the additional formatting options so that the data in the external file is read correctly by WarehousePG. For information about using a custom format, see [Loading and Unloading Data](../../admin_guide/load/index.md) in the *WarehousePG Administrator Guide*. -If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF. +If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF. ##### FORMAT 'CUSTOM' (formatter=formatter_specification) Specifies a custom data format. The formatter_specification specifies the function to use to format the data, followed by comma-separated parameters to the formatter function. The length of the formatter specification, the string including `Formatter=`, can be up to approximately 50K bytes. -If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF. +If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF. For general information about using a custom format, see "Loading and Unloading Data" in the *WarehousePG Administrator Guide*. diff --git a/docs/whpg/7x/ref_guide/sql_commands/CREATE_PROTOCOL.md b/docs/whpg/7x/ref_guide/sql_commands/CREATE_PROTOCOL.md index 9a702fa..bff695a 100644 --- a/docs/whpg/7x/ref_guide/sql_commands/CREATE_PROTOCOL.md +++ b/docs/whpg/7x/ref_guide/sql_commands/CREATE_PROTOCOL.md @@ -52,7 +52,7 @@ An optional validator function that validates the URL specified in the `CREATE E ## Notes -WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md) for information about enabling the `S3` protocol. Refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about using the `pxf` protocol. +WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md) for information about enabling the `S3` protocol. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using the `pxf` protocol. Any shared library that implements a data access protocol must be located in the same location on all WarehousePG segment hosts. For example, the shared library can be in a location specified by the operating system environment variable `LD_LIBRARY_PATH` on all hosts. You can also specify the location when you define the handler function. For example, when you define the `s3` protocol in the `CREATE PROTOCOL` command, you specify `$libdir/gps3ext.so` as the location of the shared object, where `$libdir` is located at `$GPHOME/lib`. diff --git a/docs/whpg/7x/security_guide/securing_whpg.md b/docs/whpg/7x/security_guide/securing_whpg.md index dfb5906..b70d776 100644 --- a/docs/whpg/7x/security_guide/securing_whpg.md +++ b/docs/whpg/7x/security_guide/securing_whpg.md @@ -18,7 +18,7 @@ The essential security requirements fall into the following categories: ## Accessing a Kerberized Hadoop Cluster -You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to Configuring PXF for Secure HDFS for the procedure to perform this setup. +You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to [Authenticating with Kerberos](/pxf/6x/connecting/hadoop/kerberos.md) for the procedure to perform this setup.