API-Publisher-Configuration.md

Ed-Fi API Publisher Configuration

The Ed-Fi API Publisher provides a hierarchical organization of configuration information, as documented below.

The first layer of configuration values are provided by the publisherSettings.json file, which should reside in the same folder as the Ed-Fi API Publisher's binaries. This file contains the general Options available for altering the runtime behavior. The Options values can also be supplied (overridden) using environment variables or command-line arguments, as needed.

Command-line arguments take precedence over environment variables, which in turn take precedence over the values defined in the publisherSettings.json configuration file. To use environment variables to provide configuration values, use the "Configuration Path" from the tables below, and add an EdFi:ApiPublisher: prefix to the name of each variable. For example, to specify a named connection for the source API using an environment variable, use an environment variable name of EdFi:ApiPublisher:Connections:Source:Name.

Options

Defines general behavior of the Ed-Fi API Publisher.

Configuration Path / Command-Line Argument	Description
Options:BearerTokenRefreshMinutes --bearerTokenRefreshMinutes	Indicates how frequently the Ed-Fi API Publisher will obtain a new bearer token from the source and target API endpoints. When the API reports the lifetime of the token it issues, this value is capped at half of that lifetime, so that a failed refresh is retried before the token expires. A refresh that keeps failing ends the run rather than letting it continue against an expired token. (Default value: 28)
Options:RetryStartingDelayMilliseconds --retryStartingDelayMilliseconds	Indicates the initial delay in milliseconds used when performing an exponential "back off" delay for retries (doubling the delay between retries after each attempt). (Default value: 100)
Options:MaxRetryAttempts --maxRetryAttempts	Indicates the number of times the Ed-Fi API Publisher will attempt to resend a request against the source or target APIs before determining that the failure is permanent. (Default value: 5)
Options:MaxDegreeOfParallelismForResourceProcessing --maxDegreeOfParallelismForResourceProcessing	Indicates the total number of resources that can be processed simultaneously. (Default value: 10)
Options:MaxDegreeOfParallelismForPostResourceItem --maxDegreeOfParallelismForPostResourceItem	Indicates the total number of threads that could be simultaneously issuing POST requests against the target API for each resource being processed. (Default value: 20)
Options:MaxDegreeOfParallelismForStreamResourcePages --maxDegreeOfParallelismForStreamResourcePages	Indicates the total number of threads that could be simultaneously issuing paged GET requests against the source API for each resource being processed. (Default value: 5)
Options:MaxConcurrentSourceRequests --maxConcurrentSourceRequests	Caps the number of requests the publisher will have in flight against the source API at any one time, counted across every resource being processed rather than per resource. Use it where the source API can only serve so much at once and the parallelism settings alone leave it with more than it can handle. Use 0 to leave the source API uncapped, in which case the parallelism settings alone govern how much it is asked to serve. See Reading from a busy source API below. (Default value: 0)
Options:TooManyRequestsRetryAttempts --tooManyRequestsRetryAttempts	The number of times a source read the API rejected with 429 Too Many Requests is retried. Use -1, the default, to follow MaxRetryAttempts; 0 to report the rejection on the first response; or any positive number. This is deliberately separate from MaxRetryAttempts, which governs every other retry in the publisher, so that the 429 handling can be turned off on its own. (Default value: -1)
Options:StreamingPagesWaitDurationSeconds --streamingPagesWaitDurationSeconds	Indicates the number of seconds to wait for the streaming of any of the currently streaming resources to complete before providing an update on progress using the logger. (Default value: 10)
Options:StreamingPageSize --streamingPageSize	Indicates the number of items to include in each page when streaming resources from the source API. (Default value: 75)
Options:ProcessingBlockBoundedCapacity --processingBlockBoundedCapacity	Caps the number of items each resource-processing block will buffer so that a slow target exerts backpressure on source page streaming instead of buffering source items in memory without limit. Use 0 for an automatic capacity, -1 to disable the bounds entirely, or any positive number for an explicit capacity; any other value fails options validation. See Backpressure and the memory ceiling below for how the ceiling is derived and which settings actually control it. (Default value: 0)
Options:IncludeDescriptors --includeDescriptors	Indicates whether or not to attempt to publish descriptors. (Default value: false)
Options:ErrorPublishingBatchSize --errorPublishingBatchSize	Indicates the number of items to batch in each call to the error writer. This could be used to optimize the size of a batch write depending on the operating environment (e.g. Amazon DynamoDB allows for 25 items to be written in a BatchWriteItem operation). (Default value: 25)
Options:ToleratedItemErrorCount --toleratedItemErrorCount	The number of documents rejected by the target that may fail to publish before the run itself is reported as a failure. Use 0 (the default) to fail a run that lost any document at all, a positive number to allow that many rejected documents, or -1 for best-effort publishing, which reports success no matter how many documents the target rejected; any other value fails options validation. It never tolerates a failure to read the source: a page or an item count that could not be retrieved always fails the run, because the number of documents behind it is unknown. A tolerated run still does not advance the last change version processed. See Run outcome and exit codes below. (Default value: 0)
Options:RemediationsScriptFile --remediationsScriptFile	Indicates the file system path to a JavaScript file containing remediations for failed POST requests against the target API.
Options:UseChangeVersionPaging --useChangeVersionPaging	Indicates whether or not to use change version paging. (Default value: false)
Options:ChangeVersionPagingWindowSize --changeVersionPagingWindowSize	Indicates the change version paging window size. (Default value: 25000)
Options:EnableRateLimit --enableRateLimit	Indicates whether or not to use rate limiting. (Default value: false)
Options:RateLimitNumberExecutions --rateLimitNumberExecutions	Indicates the maximum number of executions allowed within the defined time window. (Default value: 30)
Options:RateLimitTimeSeconds --rateLimitTimeSeconds	Indicates the the time span for the rate limit in seconds. (Default value: 1)
Options:RateLimitMaxRetries --rateLimitMaxRetries	Indicates the number of times the Ed-Fi API publisher will attempt to resend a request, rejected by rate limiting, to the source or destination APIs before determining that the failure is permanent. It applies only to the publisher's own limiter, enabled with EnableRateLimit, and has no effect on a 429 Too Many Requests returned by the source API. (Default value: 10)
Options:useReversePaging --useReversePaging	Indicates whether or not to use reverse paging mode. For more information about this feature read here. (Default value: false)
Options:DisableCursorPaging --disableCursorPaging	When false, main resources are read from an ODS/API 7.3+ source with partitioned cursor paging (/partitions + pageToken/pageSize) whenever the source supports it, falling back to offset/limit otherwise. true forces offset/limit paging for every resource. Deletes and key changes always use offset/limit. Change-version paging and reverse paging also imply offset/limit. The decision is logged once per resource (using Cursor paging / using Offset paging). (Default value: false)
Options:CursorPagingPartitionCount --cursorPagingPartitionCount	Number of partitions requested per resource under cursor paging (1..200, the API maximum). When not set, MaxDegreeOfParallelismForStreamResourcePages is used (capped at 200) so each page-fetch worker walks one partition. The source may return fewer partitions than requested for small resources. (Default value: unset)
Options:ResumeLastRun --resumeLastRun	When true, the run continues where the last one left off: each cursor-paged partition resumes at the last page whose documents all reached the target, and the change window the previous run recorded is replayed rather than recomputed. Both connections must be named (--sourceName, --targetName) for a run to be resumable, since the names are what tell one publication's state from another's. State written for a different source connection, target connection, change version namespace or publisher version, by a run whose connections were unnamed, or edited into something a run cannot act on, is refused with a warning and the run starts from the beginning. Offset-paged reads, including all /deletes and /keyChanges, are read in full, and only a run publishing to an Ed-Fi API records page progress at all. (Default value: false)
Options:RunStatePath --runStatePath	Where the run state a resume needs is kept. A directory takes the default file name inside it, so a containerised run can point this at a mounted volume without naming the file; that default name carries the source and target connection names and the change version namespace (api-publisher-run-state-{source}-to-{target}.json) so publications sharing a directory do not overwrite each other. When not set, the file sits in the working directory. This has to be storage that outlives the run. A containerised run that leaves it on the container filesystem loses the file when the container goes, and the resume with it; mount a volume and point this at it. The file is created readable only by the user running the publisher where the platform supports file modes. Every run records this state, and it is removed by a run that finishes without losing a document. (Default value: unset)
Options:LastChangeVersionProcessedNamespace --lastChangeVersionProcessedNamespace	Indicates the namespace for change version tracking. If provided, this string will be prepended to the target name when reading and writing the lastChangeVersionsProcessed named connection parameter.
Options:ProcessDeletesAndKeyChangesOnFullPublish --processDeletesAndKeyChangesOnFullPublish	When true, performs delete and key change processing even when the change window starts at version 1 or below (full publish). By default, these operations are skipped in full publish scenarios because it is assumed the target is empty. (Default value: false)

Bearer token handling

The publisher refreshes its bearer token while a run is in progress, and recovers when the API rejects a request as unauthorized by obtaining a new token and sending the request again.

• When the API reports the lifetime of the token it issues, the refresh interval is capped at half of that lifetime, so that the refresh always runs ahead of the expiry and a failed refresh can be retried before the token expires. This holds however short the lifetime is: an API issuing tokens that live less than a minute is refreshed every few seconds, and a warning is logged at startup pointing that out.
• A failed refresh is retried on a short, growing delay (5 seconds, doubling, capped at 60) for as long as the current token remains usable. Publishing is unaffected while that happens, because the token in hand is still valid. Once the token has expired, or when the API reports no lifetime, up to five consecutive failures are tolerated before the run ends.
• When the API rejects a request as unauthorized, the request waits while the token is re-acquired and is then sent again. If the token endpoint is unavailable at that moment, the re-acquisition is retried on the same backoff while the request waits, so a document or page is not lost to a brief outage of the token endpoint. A request body is buffered for that replay only once the request has been rejected, and only up to 16 MB; a larger body is not replayed and the unauthorized response is reported for it.
• When the token can no longer be obtained at all, the run ends with a FATAL log entry rather than continuing to send requests that the API will reject.

Behavior change: a run that cannot authenticate now exits with a non-zero exit code. Earlier versions could log the failure, keep going, and still exit 0, so automation that treats a zero exit code as success will begin to see failures it previously did not.

Reading from a busy source API

A source API that is serving other clients as well can answer a read with 429 Too Many Requests, meaning the publisher is asking for more than the API is willing to serve at that moment. The publisher waits and sends the read again rather than recording it as a failure.

• The wait is taken from the Retry-After header the API returned, given either as a number of seconds or as the date the API will start answering again. That wait is a minimum: where the backoff for that attempt is longer, the longer of the two is used.
• Where the response carries no Retry-After, or carries one the publisher cannot read, the wait is the same exponential backoff used for other transient failures, starting at RetryStartingDelayMilliseconds and doubling with each attempt.
• The read is sent again up to TooManyRequestsRetryAttempts times, which follows MaxRetryAttempts unless it is set on its own. A read the API is still rejecting after that is reported as a failure, the same way a read that kept failing with 503 Service Unavailable is reported today, so the run finishes with a non-empty error set and a non-zero exit code.
• This applies to reads from the source API only. A 429 returned by the target API in response to a POST is reported on the first response, unchanged from earlier versions. Bearer token requests are also outside this: they are sent on their own connection rather than through the request pipeline, so a 429 on a token request is not retried.

Behavior change: this is on by default. A read that previously failed within seconds of the source rejecting it is now waited out and retried, so a run against a busy source can take noticeably longer before it either succeeds or reports the same failure. Set TooManyRequestsRetryAttempts to 0 to return to the earlier behavior of reporting the rejection immediately. Do not use MaxRetryAttempts for this: that value drives every retry policy in the publisher, so setting it to 0 would also stop a source read being retried after a 503 and stop every retry on writes to the target.

There is a ceiling on the wait, and it is the time one request has. Each wait is spent inside the read that is waiting, against the 100 seconds a request is allowed in total, which is fixed and not configurable. Ten of those seconds are held back for the replay itself, so a wait of up to roughly 90 seconds is honored. A wait the remaining time cannot cover is not started at all: the read is abandoned and the rejection is reported straight away, with a log line naming the wait the API asked for and the time that was left. This is deliberate. Waiting anyway would let the request expire part-way through the wait, and the caller would then see a cancelled request with no status rather than the 429 that actually happened. So a source asking for two minutes gets a prompt, correctly labelled failure rather than a slow, mislabelled one.

One more thing worth knowing when sizing MaxRetryAttempts: this budget is spent inside a single request and is separate from the retry policy that covers other transient failures. A read that meets a 429 and then a 503 can spend both, so the worst-case number of attempts for one page is the product of the two rather than their sum. A retried read also does not pass back through the EnableRateLimit limiter, since the retry happens below it.

Capping what the source is asked to serve at once

MaxConcurrentSourceRequests is the other half of being a well-behaved client, and the two act at different points. Waiting out a 429 is a reaction to the source API having already been asked for too much; the cap keeps it from being asked in the first place. It counts requests to the source API as a whole, so it holds however the work happens to be divided between resources, and it is independent of EnableRateLimit, which limits requests per unit of time rather than requests at one time.

Two things the cap does not cover. Bearer token requests are not counted, for the same reason they are not retried. And a request holds its place only until the source has produced the response headers; the body is streamed after that, so a source can still be sending more than this many response bodies at once.

To pick a value, start from what the publisher can currently ask for at once, which is roughly MaxDegreeOfParallelismForResourceProcessing multiplied by MaxDegreeOfParallelismForStreamResourcePages, so 50 at the default settings, plus a few more for item counts and reference lookups. Halving that is a reasonable first step. Do not set it far below that product without also bringing the two parallelism settings down: a read waits for its place inside its own request, so if the queue is deep enough that reads at the back run out of the time a request is allowed, they are recorded as failures rather than simply served later. The publisher logs a warning at startup when the cap is less than a quarter of that product.

Backpressure and the memory ceiling

When the target API is slower than the source, the publisher's internal buffers are bounded so that source page fetching pauses instead of accumulating fetched items in memory without limit. ProcessingBlockBoundedCapacity controls this behavior:

• 0 (default) — automatic: the item buffer per resource-processing block is max(StreamingPageSize, 4 × MaxDegreeOfParallelismForPostResourceItem).
• -1 — disables all the buffering bounds (the memory-growth rollback lever; the compact error bodies and other memory improvements still apply).
• Any positive number — an explicit item capacity (values below MaxDegreeOfParallelismForPostResourceItem are raised to it so POST workers are never starved).
• Any other value fails options validation with a non-zero exit.

The same option governs the error publishing queue, so a sustained error storm also exerts backpressure instead of queueing errors in memory without limit. Each stage's errors pass through a tally block capped at 2 × ErrorPublishingBatchSize (only one stage runs at a time), the shared ingestion queue is capped at another 2 × ErrorPublishingBatchSize pending errors, and up to 4 more already-formed batches can be queued for publication behind it — a total of roughly 8 × ErrorPublishingBatchSize pending errors (200 at shipped defaults), each of which can carry a full document body. All of them are unbounded under -1.

What the ceiling actually is. The capacity is per block, not a process-wide total. Page handling and item processing are joined by an item buffer of itemCapacity items: each page-fetch worker pushes items into it one at a time and pauses when it is full, so page fetching stops until the target catches up -- even when one page message expands into a whole partition of pages under cursor paging. Each live resource pipeline can therefore hold up to roughly (MaxDegreeOfParallelismForStreamResourcePages × StreamingPageSize) + itemCapacity + itemCapacity items across its buffers (one page per fetch worker being drained, the item buffer, and the processing block), and up to MaxDegreeOfParallelismForResourceProcessing resource pipelines stream concurrently. Delete and key-change pipelines chain two bounded item blocks, so their processing term is 2 × itemCapacity. This is a fixed configuration-derived ceiling that does not grow with resource size or partition size. (One resource-proportional allocation remains: the per-resource list of page descriptors, which holds lightweight offset/limit entries or one partition token each.)

Which settings actually lower the ceiling. The dominant terms are one page per fetch worker (MaxDegreeOfParallelismForStreamResourcePages × StreamingPageSize) and the two item buffers (2 × itemCapacity, which defaults to StreamingPageSize when it is the larger value). For example, at StreamingPageSize=500 and MaxDegreeOfParallelismForStreamResourcePages=10, up to 10 pages are being drained plus 1,000 buffered items -- 6,000 items per resource. To materially reduce per-resource retention, lower MaxDegreeOfParallelismForStreamResourcePages and/or StreamingPageSize (and MaxDegreeOfParallelismForResourceProcessing to reduce how many resources buffer concurrently); use ProcessingBlockBoundedCapacity to shrink or raise the item buffers, or to disable bounding entirely.

Streamed page reads. When streaming from an API source, a fetched page is no longer buffered as a whole string before item splitting — the response body is parsed in a single forward-only pass that materializes one document at a time, so the per-page memory constant inside the ceiling above is one document rather than one whole page of text (see APIPUB-134). The SQLite connector is the documented exception: as a target it stores whole pages as TEXT rows, so it deliberately buffers each page string before writing it.

Cursor paging. Against an ODS/API 7.3+ source each page message is a partition rather than a single page. Partitions are walked page by page and every page is delivered through the bounded item buffer before the next one is fetched, so the ceiling above applies unchanged; --disableCursorPaging=true restores the offset/limit path with the same bound.

Operational notes. Slower or paused source page fetching while the target catches up is the bound working as intended, not a hang — the log reports the resolved capacities (items, page messages, pending errors) at startup. Resources configured for authorization-failure retry in authorizationFailureHandling are re-streamed through a retry pipeline after their update prerequisites complete; that second pass is bounded exactly like any other pipeline, so the ceiling above applies to it as well.

Run outcome and exit codes

Behavior change: every failure previously returned a single exit code of -1 (surfaced as 255 on Linux and macOS). Failures now return the specific codes below, and a rate limit rejection, which used to be logged and abandoned, now fails the run. A caller that tests for a non-zero exit code is unaffected; a caller that tests for 255 or -1 specifically must be updated, and a scheduled job that was silently tolerating rate-limited losses will start reporting them.

A run reports its outcome through its exit code, so that an unattended or scheduled publish does not need the log to tell a complete run from one that lost documents:

Exit code	Meaning	Expected response
0	Everything the run set out to publish was published, or the documents that failed were within ToleratedItemErrorCount.	None.
1	Every resource ran to completion, but documents were rejected by the target beyond the configured tolerance.	Read the run summary and the published errors, correct the data or the target, and re-publish.
2	The run did not complete, or part of the source could not be read: a resource faulted or was cancelled, a page or an item count could not be retrieved, the errors themselves could not be published, or a finalization activity failed. What was published is unknown.	Re-run.
3	The publisher could not obtain a bearer token from the source or target API.	Check the credentials, and check the token endpoint the run reported for that connection: since the endpoint is read from the API's Discovery document rather than composed, a 404 there means the address is wrong rather than the key. The run names where that address came from.
4	The configuration or the supplied options are invalid, so no publishing was attempted.	Correct the configuration.

Every run also prints a summary table, on both the success and the failure paths, reporting per resource and stage what the source reported (Expected), how many documents were read from the source and handed to the target (Attempted), how many the target rejected (Failed), how many were abandoned without being published or rejected (Skipped), and how many the target holds as the source asked (Published). Every one of those is counted where it happens, so a run that stopped with documents in flight reports them as attempted and nothing else, and says how many below the table. Published includes the documents that needed no write because the target already satisfied them: a descriptor whose value is already present, a deleted document the target no longer has, and a key change whose item is not on the target. Those cost nothing and lose nothing, so they do not hold the change version back. Errors reading the source (a page or an item count that could not be retrieved) are reported below the table instead of against a resource, because the documents behind them were never attempted and their number is not known.

A resource listed in authorizationFailureHandling is published twice: once by the pass that reads it, and again by the authorization retry pass once its update prerequisites complete. The row reports the second pass, because it republishes every document of the resource and therefore decides what the target holds, and a line below the table gives what the first pass had reported and how much the retry recovered. A document rejected on both passes is one failed document, not two.

Counts are in documents on every connection. The SQLite target is the one that writes a page at a time rather than a document at a time, so its pages report the number of documents they carry; a page that could not be written counts as every document in it.

Best-effort publishing does not advance the last change version processed. A run that reported an error for any document never records a new last change version, even when ToleratedItemErrorCount allowed it to report success. The one exception is a document you have chosen to abandon: a resource ignored through treatForbiddenPostAsWarning is reported in the summary's Skipped column and does not hold the change version back, because abandoning it was the configured intent. The documents that failed are inside the change window that was just published, so re-publishing that window on the next run is what gives them another chance; advancing the change version past them would leave them unrecoverable without a full re-publish. The consequence to plan for is the opposite one: while documents keep failing for a reason that does not go away, an incremental run keeps re-publishing the same window, which grows as the source changes. Best-effort makes the exit code quiet, not the loss.

API Connections

Where the publisher sends its requests

The publisher reads each connection's Discovery document, the anonymous document at the connection URL, and takes from it the paths where that API serves its data management resources and its change queries. This is what lets one tool publish into an Ed-Fi ODS/API, which serves data management under data/v3, and into an Ed-Fi Data Management Service, which serves it under data, without anything in front of either rewriting the requests.

• The path stated on the connection wins, and the Discovery document is not consulted for it. The document is still read, because the publisher takes each API's version information from it, so stating a path is a way to correct what an API declares rather than a way to publish against one that does not serve a Discovery document at all.
• Otherwise the path the API declares is used, expressed relative to the connection URL, so a prefix the connection URL already carries is not repeated.
• Where neither is available, the conventional data/v3 and changeQueries/v1 are used. This is ordinary for an ODS/API with the Change Queries feature disabled, which declares no change queries path at all.
• A path the API declares on a different host is refused rather than followed, because a connection's requests carry its credentials.
• An API that qualifies its routes by tenant answers its unqualified address with placeholders rather than values, so the connection URL has to name the tenant, for example https://server/tenant1/.

The path in use for each connection is reported in the log as the run starts.

The same document states where the API serves its token endpoint, and each connection's bearer token is requested from what it declares there.

• The authentication URL stated on the connection wins, and the Discovery document is not consulted for one.
• Otherwise the oauth URL the API declares is used. Its path, host and port are taken as declared. Where it names the same host and port as the connection but a different scheme, which an API behind a proxy that terminates TLS produces, the safer of the two schemes is used and the disagreement is reported as a warning, so a declaration cannot move a token request onto plain HTTP and a connection reached over HTTP does not discard an API that offers HTTPS.
• Where neither is available, oauth/token relative to the connection URL is used.
• A declared endpoint may name a host of its own, because an API is permitted to have its identity provider served elsewhere. A token request carries the connection's key and secret, so an endpoint that names anything other than the address the connection already reaches the API at is followed only over HTTPS, and the host it names is reported in the log. To send credentials to such a host over plain HTTP, state the endpoint on the connection instead.
• An endpoint on another host is refused outright where the connection is set not to verify server certificates (IgnoreSSLErrors), because HTTPS is what identifies the host being reached and nothing is checking it. State the endpoint on the connection if it is genuinely served there.
• An endpoint that is neither the connection's own address nor HTTPS is refused rather than followed, and so is one that still carries an unresolved route placeholder, or a stated authentication URL that is not an absolute address. Each stops the run with a configuration error before anything is published, so the run exits 4. See Run outcome and exit codes.

Behavior change: paths are no longer assumed. Each connection is asked for its Discovery document once while it is being set up, and the version check and the dependency metadata read that same copy, so a run makes no more requests to an API's root than before and usually fewer. An ODS/API is unaffected in where its requests are sent, because it declares the same data/v3 the publisher previously assumed. A DMS is the deployment this does change, and an upgrade needs one step for it. Until now a DMS could only be published to through a proxy that rewrote data/v3 onto the data it actually serves. The publisher now asks for data directly, so a proxy that routes only /data/v3/* answers 404 to every request, and nothing in that failure points at routing. Either remove the rewrite, which is what this change makes possible, or keep addressing the API as before by setting Connections:Target:DataManagementUrlSegment (--targetDataManagementUrlSegment) to data/v3. An API that declares itself at a scheme or port its callers do not use, which is what an ODS/API behind a proxy that terminates TLS does when it is not told to advertise the forwarded scheme, is still published to: only the path is taken from the declaration, requests keep the address the connection states, and the disagreement is reported as a warning. A deployment that qualifies its routes by tenant and is addressed at its unqualified URL now stops with a configuration error instead of sending requests that cannot be served. The token endpoint is no longer composed either. A connection that states no authentication URL previously requested its token from oauth/token under its own connection URL, always; it now requests it from wherever the API declares. For an ODS/API or a DMS asked at the address its callers use, that is the same address and nothing changes. An API that declares an address its callers cannot reach, such as an internal hostname behind a proxy or container network, now either sends the token request there or stops the run. Setting AuthUrl on the connection restores the previous address explicitly. Separately, an authentication URL stated on a connection is now used exactly as given rather than having a trailing slash applied to it.

Metadata for source and targets API connections can be supplied to the publisher using values stored in a persistent configuration or through environment variables and/or command-line arguments, as documented below. It is strongly recommended that you use named connections with persistent configuration for repeated publishing operations (e.g. --sourceName=abcd --targetName=wxyz). You should not mix named connections with overrides supplied through environment variables or command-line arguments.

NOTE: If the Ed-Fi API Publisher is executed using explicit connection information (rather than a pre-configured named connection), the LastChangeVersionProcessed value cannot be updated automatically upon successful publishing (as there is no API connection name associated with the information). It will be the responsibility of the caller to update the value appropriately after extracting the new change version from the log output (or through some other enterprising manner). As such, for implementing a process that is intended to only publish changes from a source to a target, it is impractical to use an approach where the API connection details are provided explicitly at execution time.

To select or supply source and target connection information, the following configuration values apply:

Configuration Path	Description
Connections:Source:Name --sourceName	The name of the preconfigured connection for the source Ed-Fi ODS API.
Connections:Source:Url --sourceUrl	The URL of the source Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Source:Key --sourceKey	The API key for authenticating with the source Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Source:Secret --sourceSecret	The API secret for authenticating with the source Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Source:AuthUrl --sourceAuthUrl	(Optional) The URL of the token endpoint for the source Ed-Fi ODS API. Normally this is not needed: the endpoint is taken from the oauth URL in the API's Discovery document. Set it when the API declares an endpoint its callers cannot use, or to send this connection's credentials to a host other than the API's own over plain HTTP, which is otherwise refused. A value set here is used exactly as given, and the Discovery document is not consulted for one. Where neither is available, oauth/token relative to the connection URL is used. (Changed) Earlier versions appended a trailing slash to this value before requesting the token; it is now sent exactly as written. If your value relied on that slash, add it.
Connections:Source:Scope --sourceScope	(Optional) The EducationOrganizationId scope requested for the resulting access token. The value must be an EducationOrganizationId that is explicitly associated with the API client by the source Ed-Fi ODS API. Intended for use to allow a single API connection configuration to be used to read changes from the controlling organization's Ed-Fi ODS API, but with the operations of the Ed-Fi API Publisher authorized for a particular Education Organization.
Connections:Source:SchoolYear --sourceSchoolYear	(Optional) The SchoolYear to use with the source connection (corresponding to a year-specific ODS API deployment).
Connections:Source:DataManagementUrlSegment --sourceDataManagementUrlSegment	(Optional) The path where the API serves its data management resources, relative to the connection URL and without leading or trailing slashes (for example data, data/v3, gateway/data). Normally this is not needed: the path is taken from the API's Discovery document. Set it when the API declares a path its callers cannot use, such as one deployed behind a gateway that declares the address it sits at rather than the address its callers reach it by. Note that an API whose Discovery document cannot be read at all is not covered by this: the publisher reads that document for version information as well, and a run against an API that does not serve it ends before these paths are used. A value set here is used as given and the Discovery document is not consulted for it, except that a connection's SchoolYear is still applied to it, the same as to a declared path. Where neither is available the conventional data/v3 is used.
Connections:Source:ChangeQueriesUrlSegment --sourceChangeQueriesUrlSegment	(Optional) The path where the API serves change queries, relative to the connection URL and without leading or trailing slashes (for example changeQueries/v1). The same rules as DataManagementUrlSegment apply, and the conventional value is changeQueries/v1.
Connections:Source:Include --include	(Optional) For source API connections, the resources to publish to the target with their dependencies. The value is defined using a CSV format (comma-separated values), and should contain the partial paths to the resources (e.g. /ed-fi/students,/custom/busRoutes). For convenience when working with Ed-Fi standard resources, only the name is required (e.g. students,studentSchoolAssociations). The Ed-Fi API Publisher will also evaluate and automatically include all dependencies of the requested resources (using the dependency metadata exposed by the target API). This will ensure (barring misconfigured authorization metadata or data policies) that data can be successfully published to the target API.
Connections:Source:IncludeOnly --includeOnly	(Optional) For source API connections, the resources to publish to the target without their dependencies. The value is defined using the same format as with --include (see above). NOTE: Use caution when publishing without automatically including all dependencies.
Connections:Source:Exclude --exclude	(Optional) For source API connections, the resources (and their dependents) to NOT publish to the target. The value is defined using the same format as with --include (see above). The Ed-Fi API Publisher will also evaluate and automatically exclude all dependent resources of the excluded resources (using the dependency metadata exposed by the target API). This will ensure (barring misconfigured authorization metadata or data policies) that data can be successfully published to the target API.
Connections:Source:ExcludeOnly --excludeOnly	(Optional) For source API connections, the specific resources to skip publishing to the target (dependent resources will still be published). The value is defined using the same format as with --include (see above). NOTE: Use caution when publishing without automatically including all dependencies.
Connections:Source:IgnoreIsolation --ignoreIsolation	(Optional) A boolean value (true/false) indicating whether the source Ed-Fi ODS API data should be published without using snapshot isolation. This argument must also be provided and set to true if the source API does not support an isolated context through the use of the snapshots resource (i.e. before Ed-Fi ODS v5.2). (NOTE: This is not a flag -- the value true or false must be provided (i.e. --ignoreIsolation=true).)
Connections:Source:ProfileName --sourceProfileName	(Optional) For source API connections, the name of the API Profile to read and write data from the API. You should use similar profiles between target and source to avoid data loss.
Connections:Source:LastChangeVersionProcessed --lastChangeVersionProcessed	(Optional) Indicates the last change version successfully published from the source API, and thus the change version after which the current publishing operation should start. This value explicitly overrides any change version value obtained from a named connection.
Connections:Target:Name --targetName	The name of the pre-configured connection for the target Ed-Fi ODS API.
Connections:Target:Url --targetUrl	The URL of the target Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Target:Key --targetKey	The API key for authenticating with the target Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Target:Secret --targetSecret	The API secret for authenticating with the target Ed-Fi ODS API. Only required if named connections are not in use.
Connections:Target:AuthUrl --targetAuthUrl	(Optional) The URL of the token endpoint for the target Ed-Fi ODS API. Normally this is not needed: the endpoint is taken from the oauth URL in the API's Discovery document. Set it when the API declares an endpoint its callers cannot use, or to send this connection's credentials to a host other than the API's own over plain HTTP, which is otherwise refused. A value set here is used exactly as given, and the Discovery document is not consulted for one. Where neither is available, oauth/token relative to the connection URL is used. (Changed) Earlier versions appended a trailing slash to this value before requesting the token; it is now sent exactly as written. If your value relied on that slash, add it.
Connections:Target:Scope --targetScope	(Optional) The EducationOrganizationId scope requested for the resulting access token. The value must be an EducationOrganizationId that is explicitly associated with the API client by the target Ed-Fi ODS API. Intended for use to allow a single API connection configuration to be used to publish changes to the controlling organization's Ed-Fi ODS API, but with the operations of the Ed-Fi API Publisher authorized for a particular Education Organization.
Connections:Target:SchoolYear --targetSchoolYear	(Optional) The SchoolYear to use with the target connection (corresponding to a year-specific ODS API deployment).
Connections:Target:DataManagementUrlSegment --targetDataManagementUrlSegment	(Optional) The path where the API serves its data management resources, relative to the connection URL and without leading or trailing slashes (for example data, data/v3, gateway/data). Normally this is not needed: the path is taken from the API's Discovery document. Set it when the API declares a path its callers cannot use, such as one deployed behind a gateway that declares the address it sits at rather than the address its callers reach it by. Note that an API whose Discovery document cannot be read at all is not covered by this: the publisher reads that document for version information as well, and a run against an API that does not serve it ends before these paths are used. A value set here is used as given and the Discovery document is not consulted for it, except that a connection's SchoolYear is still applied to it, the same as to a declared path. Where neither is available the conventional data/v3 is used.
Connections:Target:ChangeQueriesUrlSegment --targetChangeQueriesUrlSegment	(Optional) The path where the API serves change queries, relative to the connection URL and without leading or trailing slashes (for example changeQueries/v1). The same rules as DataManagementUrlSegment apply, and the conventional value is changeQueries/v1.
Connections:Target:ProfileName --targetProfileName	(Optional) For target API connections, the name of the API Profile to read and write data from the API. You should use similar profiles between target and source to avoid data loss.
Connections:Target:TreatForbiddenPostAsWarning --treatForbiddenPostAsWarning	(Optional) A boolean value (true/false) indicating whether 403 Forbidden responses from POST requests against the connection (as a target) should be treated as a warning, rather than a failure. NOTE: This option can be used in scenarios where the target API may not grant the Ed-Fi API Publisher full CRUD permissions to all the dependencies of the specified resources to be written. In such a scenario, the dependent data must already exist in the target ODS or the resulting 409 Conflict responses will cause publishing failure.

Considerations in relation to key changes and deletes

Ed-Fi API Publisher will only process key changes and deletions if specific Change Window is defined. To do so use the --lastChangeVersionProcessed value and set the --useChangeVersionPaging flag to true. Another option, if you want to keep the --useChangeVersionPaging false is defining a name for the source and target, using the --sourceName and --targetName values. More information about all these values below.

When performing a full publish (change window starting at version 1 or below), delete and key change processing are skipped by default. Use --processDeletesAndKeyChangesOnFullPublish=true if you need these operations to run during a full publish.

Authorization Failure Handling

Defines metadata (as an array of JSON objects) about which resources could experience 403 Forbidden responses caused by data dependencies needed for successful authorization, and which other resources should be processed before retrying the original request. For example, while an API client may be able to create a Student, they won't be able to update the Student until that Student is enrolled in a School through the StudentSchoolAssociation. By defining the authorization-related dependency of the update operation on the StudentSchoolAssociation, the Ed-Fi API Publisher can know to retry the failed POST request after the association has been established.

NOTE: This part of the configuration can only be defined in the publisherSettings.json file.

JSON Path	Description
/authorizationFailureHandling[*]	Defines metadata for a single resource which could experience 403 Forbidden responses.
/authorizationFailureHandling[*]/path	The partial path for the resource (e.g. /ed-fi/students) for which additional 403 Forbidden processing should be performed.
/authorizationFailureHandling[*]/updatePrerequisitePaths	An array of partial paths for the resource(s) (e.g. /ed-fi/studentSchoolAssociations) that should be processed before attempting to retry the original request which resulted in an authorization failure.

The default configuration targets Data Standard 5.x and later, where the parent resource is /ed-fi/contacts. For sources on Data Standard 4.0 or earlier, replace that entry with /ed-fi/parents → /ed-fi/studentParentAssociations. An entry whose path or prerequisites do not exist in the source's dependency graph is skipped with a warning in the log (it does not fail the run).

How the retry works: a 403 Forbidden POST response for a configured resource is skipped without publishing an error, because the entire resource is re-published through a second, bounded streaming pass once all of its updatePrerequisitePaths resources have completed. Individual failed items are not queued or retried in place — the second pass covers them (which also means the resource's documents are fetched from the source and sent to the target twice). A request that is still Forbidden during the second pass is treated as a final error (or as a warning, when the target connection sets treatForbiddenPostAsWarning).

The default configuration is as follows:

{  
  "options":   
  {  
    ...
  },  
  "authorizationFailureHandling": [  
    {  
      "path": "/ed-fi/students",  
      "updatePrerequisitePaths": ["/ed-fi/studentSchoolAssociations"]  
    },  
    {  
      "path": "/ed-fi/staffs",  
      "updatePrerequisitePaths": [  
        "/ed-fi/staffEducationOrganizationEmploymentAssociations",  
        "/ed-fi/staffEducationOrganizationAssignmentAssociations"  
      ]  
    },  
    {  
      "path": "/ed-fi/contacts",  
      "updatePrerequisitePaths": ["/ed-fi/studentContactAssociations"]  
    }
  ]
}