Tremor Deployment Language (Troy) #36

mfelsche · 2020-12-15T14:01:15Z

text/0000-deployment-language.md

Licenser · 2020-12-18T15:15:46Z

text/0000-deployment-language.md

+There are also the following steps:
+
+* Disconnect an artefact instance from all its connections (e.g. disconnect an offramp from its pipelines)
+* Destroy an artefact instance and thus stop its operation


We might want to mention that removing instances trickles down (up?) to connected instances for automatic reclemation of resources. And if / how we do that in the future

text/0000-deployment-language.md

Licenser · 2020-12-18T16:17:15Z

text/0000-deployment-language.md

+
+If no instance id is given, as part of the flow instantiation a new instance of the artefact is created with the flow instance id filled in as instance id. So, users can omit an instance id if they want to have their artefacts be auto-created as part of an event flow.
+They will also be auto-deleted upon flow deletion. Our goal with this setup is to make it possible for users to define their tremor setup
+as one unit, the `Flow`. It can be instantiated and deleted with one command or API call.


constistency in casing:

Suggested change

as one unit, the `Flow`. It can be instantiated and deleted with one command or API call.

as one unit, the `flow`. It can be instantiated and deleted with one command or API call.

text/0000-deployment-language.md

Licenser · 2020-12-18T16:21:18Z

text/0000-deployment-language.md

+
+If the port is ommitted, based on its position a default port is chosen. If a reference is on the LHS the port `out` is chosen, on the RHS `in` is implied. This matches the normal event flow: From the `out` port of an artefact to an `in` port of another one. Other cases need to be handled explicitly.
+
+#### Connect Statement Sugar


Can we consider not having to quite the parts in the arrow sytnax? Something like:

# without ports /source/stdin -> /pipeline/pipe; /pipeline/pipe/err -> /sink/system::stderr; # with ports pipeline/pipe:err -> /sink/my_error_file:in;

Yeah, i think we should discuss a non-stringy representation of TremorURL.

text/0000-deployment-language.md

Signed-off-by: Matthias Wahl <mwahl@wayfair.com>

Co-authored-by: Heinz N. Gies <heinz@licenser.net> Signed-off-by: Matthias Wahl <mwahl@wayfair.com>

Signed-off-by: Matthias Wahl <mwahl@wayfair.com>

mfelsche · 2021-03-24T16:00:33Z

Idea from random chatting in discord:

Remove the distinction between the bindings between pipelines and offramps/onramps and connections inside a pipeline using trickle select statements. The goal here would be to use select statements to connect onramps/offramps transparently the same way we use selects for intra-pipeline dag setup. One possibility is to allow stream references in select statements (the well known in and out) as well as tremor-urls referencing onramps/offramps. This can be implemented as some heavy syntax sugar which desugars to a pipeline definition and the flow that connects the pipeline to all the other artefacts. Unsure how this behaves with connecting multiple pipelines with multiple artefacts.

Licenser · 2021-04-09T12:59:59Z

Interesting thought, one concern I would have is how will that affect clustering. We will want to have some distinction between pipelines as a nonpersistent, semi stateless, nonblocking entity and sources/sinks as a stateful, persistent, potentially blocking entity.

Treating them exactly the same way during configuration time might make it harder for users to understand the implications of a configuration. As it is now (and that clearly isn't ideal 😂) there is an extremely clear distinction between "this interacts with the outside world and will be slow" and "woosh pipeline!!!!" that would get lost if it's all selects

mfelsche marked this pull request as ready for review December 18, 2020 13:56

darach added open-review This RFC is undergoing open / public review with intent to proceed enhancement New feature or request labels Dec 18, 2020