Orchester links

Three modules carry data between two Data Orchester instances over the signed peer link: live values in either direction, and logged history towards the instance that collects it. The peer link itself, with its peer entities, secrets, permissions and the pipe that reaches an instance behind NAT, is described in Connectivity; this page describes the modules that use it.

Every type also has the settings all modules share: whether it is enabled, its name, its status and message variables and their thresholds. They are described once, in Settings every module shares.

What every link needs

• A peer entity on each side. Here, one for the far instance, with the shared secret and, when this side dials, its address. There, one for this instance, with the same secret. Both instances' licences must belong to the same organization.
• One module, one peer. Each module talks to exactly one peer: the one whose code is in the module's DataOrchester setting. Any number of modules may use the same peer.
• A running far side. An instance that is stopped answers every call as unavailable.
• Time. The peer entity's timeout, 10 000 ms unless set, bounds each call, direct or over a pipe. Connecting gives up after 5 seconds.
• Size. A far side takes at most 500 values in one push, at most 500 names in one pull, and 8 MiB in any request.

Orchester Push

Sends the values of local variables to a peer, which writes them into its own variables of the same name, or of the same name with a prefix.

Settings

A Delay or Timeout left empty takes its default.

How values are sent

• Once at the start. Every listed variable is sent shortly after the module opens.
• On a change. Changes are gathered for delay and sent together, so a change leaves within about twice delay, in one request with every other change of that moment.
• Again every timeout. A variable that has not changed is sent again once timeout seconds have passed since it was last sent, so a far side that restarted, or missed a request, catches up without waiting for the value to move.
• Retrying. When the peer could not be reached or could not answer, the same variables go again at the next pass, delay later. When the peer refused the request, they are not resent until they change or their timeout comes round.
• The prefix. With prefix ST2_, LIT_301 is sent as ST2_LIT_301.

What the far side accepts

The far side writes a value only when the peer entity it holds for this instance lists the name, as received and so with the prefix, in its writables. The names it refuses are named in its answer, and the module shows {n} of {total} variable(s) delivered. A refused name is not resent at once; it goes again with the next change or the next timeout, and is refused again until the far side lists it. Each name should be a variable defined on the far side, where the value is converted to that variable's declared type.

Messages

A counted failure is a WARNING that reads … (1 of 3) and … (2 of 3), and becomes an ERROR reading … after 3 attempts, with the thresholds set in Settings every module shares. The complete list is in Module messages.

• Not defined covers three cases: no peer entity has that code, the peer's secret does not resolve (an env: name the service's environment does not set), or the licence admits no more peers. The text then lists the peers that did load, as (known: …).
• The reason is the answer's status, followed by its message where there is one: UNAUTHORIZED for a wrong secret, a clock more than 5 minutes apart or an instance the far side does not know; FORBIDDEN for another organization; TOO_LARGE for more than 500 values; UNAVAILABLE for a far side that is stopped or cannot be reached. ERROR - Unsigned or unverifiable response. means the far side answered without a valid signature, as it does to a request over 8 MiB and while it holds off a sender after repeated failed authentications.
• A request that goes through clears the failures.

Example

A station, STATION_2, reports three readings to the control room, whose peer entity for it lists ST2_LIT_301, ST2_FIT_501 and ST2_P101_RUN in its writables:

DataOrchester   CONTROL_ROOM
Delay [ms]      250
Timeout [s]     300
Prefix          ST2_
Variables       LIT_301, FIT_501, P101_RUN

A change of LIT_301 reaches the control room as ST2_LIT_301 within about half a second, and all three are sent again every five minutes whether they changed or not. Had the control room listed only the first two, the first send after the start would leave the module warning 2 of 3 variable(s) delivered.

Orchester Pull

Asks a peer for the current values of its variables on a schedule, and writes them into local variables. The far side needs nothing but a peer entity for this instance: any peer that authenticates may read its variables.

Settings

How values arrive

• One request per tick, asking for every listed variable. The first request is made at the first tick after the module opens, not when it opens.
• Written locally under the far side's code, with the prefix in front: with prefix ST2_, the far side's LIT_301 is written into ST2_LIT_301. The value goes through the engine's write path, converted to the local variable's declared type; a value equal to the one held changes nothing.
• Missing names. A variable that the far side does not have, or that holds no value there, is left out of the answer. The local variable keeps what it held, and the module warns.
• A slow peer. A tick that falls while the previous request is still waiting for its answer is skipped.

Messages

The peer messages mean what they mean for Orchester Push. A tick whose every name arrives clears the warning.

Example

The control room samples the same station every minute:

DataOrchester   STATION_2
CRON            0 * * * * ?
Prefix          ST2_
Variables       LIT_301, FIT_501, P101_RUN

Every minute the control room writes ST2_LIT_301, ST2_FIT_501 and ST2_P101_RUN. If the station's FIT_501 has never held a value, the module warns 1 variable(s) not defined on the peer and the other two still arrive. When the station sits behind NAT and dials the control room, the requests travel down the station's pipe, and the control room's peer entity for it needs no address.

Logger Replicator

Copies the rows a local LOGGER has written to a LOGGER on a peer, so the peer holds the history too. It sends what the far side does not have yet, so a link that was down for a day is caught up once it is back, with nothing lost.

Settings

A new record also holds a code key, which the module does not use.

How rows are copied

At each tick the module:

1. Asks the target logger, in one request, for the latest timestamp it holds of each series, under the far side's name, prefix included.
2. Reads the local rows of each series from that timestamp on, the boundary included, and sends them in batches of batch rows. A series the target has never seen is sent from its first row.
3. Moves on to the next series when a series is caught up, or after 20 batches.

• Nothing is lost or doubled. The far side drops rows it already holds, so the rows at the boundary, or a batch sent again after a lost answer, are written once. A batch that fails is sent again from the same point at the next tick.
• A backlog is worked off. Rows beyond 20 batches of one series wait for the next tick, and the module warns until it has caught up, then writes Replication caught up into its info message.
• A slow peer. A tick that falls while the previous one is still running is skipped. The peer's timeout has to cover one batch: the far side writes a whole batch before it answers.
• Rows keep their timestamps, in UTC as the local logger wrote them.
• Write codes in capitals. The local logger stores each code in capital letters, and the replicator looks rows up, and names them on the far side, exactly as variables and prefix are written. A code in lower case finds no rows here, and a prefix in lower case stores rows that the far side's history functions, which look a variable up in capitals, never find.
• The prefix names the series. One target can gather the same codes from several stations, each with its own prefix. Changing the prefix starts new series on the far side, and the whole history is sent again under the new names.

A dedicated target logger

The far side resumes each series from the latest row it holds of it. If the target logger also logs a variable of the same name as it runs, for example one that a DOPUSH from the same station keeps up to date, its latest row is always a live one: the replicator sends only what follows it, and the rows written while the link was down never arrive.

Messages

• The backlog message names the batch size: Replication behind: more than 500 row(s) pending.
• An empty Target is refused by the far side, with the reason BAD_REQUEST.
• A batch the far side refuses, or that does not reach it, is written to the log and sent again at the next tick; the module's status shows the failure only until that tick ends.
• A tick that copies everything clears the errors and warnings of the ticks before it.

Example

The station copies its level, flow and pump status to the control room every five minutes:

Logger          STATION_LOG
DataOrchester   CONTROL_ROOM
Target          REPLICA
Prefix          ST2_
CRON            0 0/5 * * * ?
Batch           500
Variables       LIT_301, FIT_501, P101_RUN

On the control room, a LOGGER with the code REPLICA, enabled and with no variables of its own, receives the rows as ST2_LIT_301, ST2_FIT_501 and ST2_P101_RUN, and a formula there reads the station's level with history(REPLICA, "ST2_LIT_301", addHours(now(), -24), now()). A second station replicating into the same logger with the prefix ST3_ keeps its series apart.

Next steps

• Connectivity
• Logger
• Settings every module shares
• Module messages
• Module catalogue

This page describes Data Orchester engine 6.12.0.