Skip to content
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions documentation/src/docs/batch.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Batch
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Batch Connector

The Flink Connector library for Pravega makes it possible to use a Pravega Stream as a data source and data sink in a batch program. See the below sections for details.

## Table of Contents
Expand Down
6 changes: 4 additions & 2 deletions documentation/src/docs/configurations.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Configurations
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Configurations

The Flink connector library for Pravega supports the **Flink Streaming API**, **Table API** and **Batch API**, using a common configuration class.

## Table of Contents
Expand Down
4 changes: 4 additions & 0 deletions documentation/src/docs/dev-guide.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Developer's Guide
---

<!--
Copyright Pravega Authors.

Expand Down
8 changes: 7 additions & 1 deletion documentation/src/docs/getting-started.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Getting Started
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,12 +18,14 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Pravega Flink Connectors
import TOCInline from '@theme/TOCInline';

This repository implements connectors to read and write [Pravega](http://pravega.io/) Streams with [Apache Flink](http://flink.apache.org/) stream processing framework.

The connectors can be used to build end-to-end stream processing pipelines (see [Samples](https://github.com/pravega/pravega-samples)) that use Pravega as the stream storage and message bus, and Apache Flink for computation over the streams.

<TOCInline toc={toc} />

## Features & Highlights

- **Exactly-once processing guarantees** for both Reader and Writer, supporting **end-to-end exactly-once processing pipelines**
Expand Down
6 changes: 4 additions & 2 deletions documentation/src/docs/metrics.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Metrics
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Metrics

Pravega metrics are collected and exposed via Flink metrics framework when using [`FlinkPravegaReader`](streaming.md#flinkpravegareader) or [`FlinkPravegaWriter`](streaming.md#flinkpravegawriter).

## Reader Metrics
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Overview
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Apache Flink Connectors for Pravega

This documentation describes the connectors API and it's usage to read and write [Pravega](http://pravega.io/) streams with [Apache Flink](http://flink.apache.org/) stream processing framework.

Build end-to-end stream processing pipelines that use Pravega as the stream storage and message bus, and Apache Flink for computation over the streams. See the [Pravega Concepts](http://pravega.io/docs/pravega-concepts/) page for more information.
Expand Down
6 changes: 4 additions & 2 deletions documentation/src/docs/quickstart.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Quick Start
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Getting Started

## Creating a Flink Stream Processing Project

**Note**: _You can skip this step if you have a streaming project set up already._
Expand Down
6 changes: 4 additions & 2 deletions documentation/src/docs/serialization.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Serialization
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Serialization

**Serialization** refers to converting a data element in your Flink program to/from a message in a Pravega stream.

Flink defines a standard interface for data serialization to/from byte messages delivered by various connectors. The core interfaces are:
Expand Down
18 changes: 18 additions & 0 deletions documentation/src/docs/sidebars.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
module.exports = {
mainSidebar: [
'overview',
'getting-started',
'quickstart',
'dev-guide',
{
'Features': [
'streaming',
'batch',
'table-api',
],
},
'metrics',
'configurations',
'serialization',
]
};
6 changes: 4 additions & 2 deletions documentation/src/docs/streaming.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Streaming
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Streaming Connector

The Flink Connector library for Pravega provides a data source and data sink
for use with the Flink Streaming API. See the below sections for details.

Expand Down
14 changes: 8 additions & 6 deletions documentation/src/docs/table-api.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Table API
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Table Connector

The Flink connector library for Pravega provides a table source and table sink for use with the Flink Table API.
The Table API provides a unified table source API for both the Flink streaming and batch environment, and also sink for the Flink streaming environment.

Expand Down Expand Up @@ -89,9 +91,9 @@ WITH (
| security.trust-store | optional | (none) | String | Trust Store for Pravega client |
| scan.execution.type | optional | streaming | String | Execution type for scan source. Valid values are 'streaming', 'batch'. |
| scan.reader-group.name | optional | (none) | String | Pravega reader group name |
| scan.streams | required for source | (none) | List<String> | Semicolon-separated list of stream names from which the table is read. |
| scan.start-streamcuts | optional | (none) | List<String> | Semicolon-separated list of base64 encoded strings for start streamcuts, begin of the stream if not specified |
| scan.end-streamcuts | optional | (none) | List<String> | Semicolon-separated list of base64 encoded strings for end streamcuts, unbounded end if not specified |
| scan.streams | required for source | (none) | List&lt;String&gt; | Semicolon-separated list of stream names from which the table is read. |
| scan.start-streamcuts | optional | (none) | List&lt;String&gt; | Semicolon-separated list of base64 encoded strings for start streamcuts, begin of the stream if not specified |
| scan.end-streamcuts | optional | (none) | List&lt;String&gt; | Semicolon-separated list of base64 encoded strings for end streamcuts, unbounded end if not specified |
| scan.reader-group.max-outstanding-checkpoint-request | optional | 3 | Integer | Maximum outstanding checkpoint requests to Pravega |
| scan.reader-group.refresh.interval | optional | 3 s | Duration | Refresh interval for reader group |
| scan.event-read.timeout.interval | optional | 1 s | Duration | Timeout for the call to read events from Pravega |
Expand All @@ -109,7 +111,7 @@ WITH (
`scan.execution.type` can be specified as user's choice to perform batch read or streaming read.
In the streaming environment, the table source uses a [`FlinkPravegaReader`](streaming.md#flinkpravegareader) connector.
In the batch environment, the table source uses a [`FlinkPravegaInputFormat`](batch.md#flinkpravegainputformat) connector.
Please see the documentation of [Streaming Connector](streaming.md) and [Batch Connector](#batch.md) to have a better understanding on the below mentioned parameter list.
Please see the documentation of [Streaming Connector](streaming.md) and [Batch Connector](batch.md) to have a better understanding on the below mentioned parameter list.

### Specify start and end streamcut

Expand Down