Source Email

Purpose

Defines the specific source parameters for an Email connected endpoint.

This Asset can be used by:

Asset type	Link
Input Processors	Stream Input Message

Prerequisite

You need:

• Email Connection

Configuration

Name & Description

• Name : Name of the Asset. Spaces are not allowed in the name.

• Description : Enter a description.

Inheritance chain of this Asset — If this Asset extends another, the inheritance chain is shown here. Click to navigate to any parent Asset in the chain.

Asset Usage: If the Asset is used by other Assets, the Asset Usage box shows how many times this Asset is used and which parts are referencing it. Otherwise it is not shown. Click to expand and then click to follow, if any.

Required roles

In case you are deploying to a Cluster which is running (a) Reactive Engine Nodes which have (b) specific Roles configured, then you can restrict use of this Asset to those Nodes with matching roles. If you want this restriction, then enter the names of the Required Roles here. Otherwise, leave empty to match all Nodes (no restriction).

Throttling & Failure Handling

Throttling

These parameters control the maximum number of new stream creations per given time period.

Max. new streams — Maximum number of streams this source is allowed to open or process within the given time period.

Per — Time interval unit for the Max. new streams value.

info

Configuration values for this parameter depend on the use case scenario. Assuming your data arrives in low frequency cycles, these values are negligible. In scenarios with many objects arriving in short time frames, it is recommended to review and adapt the default values accordingly.

Backoff Failure Handling

These parameters define backoff timing intervals in case of failures. The system will progressively throttle down the processing cycle based on the configured minimum and maximum failure backoff boundaries.

Min. failure backoff — The minimum backoff time before retrying after a failure.

Unit — Time unit for the minimum backoff value.

Max. failure backoff — The maximum backoff time before retrying after a failure.

Unit — Time unit for the maximum backoff value.

Based on these values, the next processing attempt is delayed: starting at the minimum failure backoff interval, the wait time increases step by step up to the maximum failure backoff.

Reset after number of successful streams — Resets the failure backoff throttling after this many successful stream processing attempts.

Reset after time without failure streams — Resets the failure backoff throttling after this amount of time passes without any failures.

Unit — Time unit for the time-based backoff reset.

Whatever comes first — the stream count or the time threshold — resets the failure throttling after the system returns to successful stream processing.

Polling & Processing

This source does not reflect a stream, but an object-based storage source which does not signal the existence of new objects to observers. We therefore need to define how often we want to look up (poll) the source for new objects to process.

You can choose between Fixed rate polling and Cron tab style polling.

Fixed rate

Use Fixed rate if you want to poll at constant and frequent intervals.

Polling interval [sec] — The interval in seconds at which the configured source is queried for new objects.

Cron tab

Use Cron tab if you want to poll at specific scheduled times. The Cron tab expression follows the cron tab style convention. Learn more about crontab syntax at the Quartz Scheduler documentation.

You can also use the built-in Cron expression editor — click the calendar symbol on the right hand side:

Configure your expression using the editor. The Next trigger times display at the top helps you visualize when the next triggers will fire. Press OK to store the values.

Polling timeout

Polling timeout [sec] — The time in seconds to wait before a polling request is considered failed. Set this high enough to account for endpoint responsiveness under normal operation.

Stable time

Stable time [sec] — The number of seconds that file statistics must remain unchanged before the file is considered stable for processing. Configuring this value enables stability checks before processing.

Ordering

When listing objects from the source for processing, you can define the order in which they are processed:

• Alphabetically, ascending
• Alphabetically, descending
• Last modified, ascending
• Last modified, descending

Reprocessing mode

The Reprocessing mode setting controls how layline.io's Access Coordinator handles previously processed sources that are re-ingested.

• Manual access coordinator reset — Any source element processed and stored in layline.io's history requires a manual reset in the Sources Coordinator before reprocessing occurs (default mode).
• Automatic access coordinator reset — Allows automatic reprocessing of already processed and re-ingested sources as soon as the respective input source has been moved into the configured done or error directory.
• When input changed — Behaves like Manual access coordinator reset, but also checks whether the source has potentially changed — i.e., the name is identical but the content differs. If the content has changed, reprocessing starts without manual intervention.

Wait for processing clearance

When Wait for processing clearance is activated, new input sources remain unprocessed in the input directory until either:

• A manual clearance is given through Operations, or
• A JavaScript processor executes AccessCoordinator.giveClearance(source, stream, timeout?)

Email Settings

Configure the parameters for your Email endpoint:

Connection

Use the drop-down list to select an Email Connection that should support this Email configuration. If it does not exist, you need to create it first.

info

Your Email Connection needs to have the following configured scope:

• mail.readwrite

Stream name

By its nature, an Email Source does not appear with a classical stream name. At this point you define the stream name for later identification during processing within a workflow:

• Stream name : name to apply for the email workflow stream processing

You need to ensure that the name is unique. As you can see from the example above, you can use Macros here. In the above case we chose to include several constants like subject: / from: / received: etc. In addition, we included several classical email identification elements deducted from Macros to form the email stream name. Finally, the Audit trail for data processed through the configured Email source will look similar to a classical email inbox folder:

Folders

• Input folder : The directory to read emails from. The configured input folder must be accessible to the Reactive Engine trying to access the Email source. You can use ${...} macros to expand variables defined in environment variables.

• Done folder : The directory to which emails are moved when fully processed. Leaving it empty, the emails will not be moved. The configured done folder must be accessible to the Reactive Engine trying to access the Email source. You can use ${...} macros to expand variables defined in environment variables.

• Error folder : The directory to which emails are moved in case of a problem with the email during processing. Leaving it empty, the emails will not be moved. The configured error folder must be accessible to the Reactive Engine trying to access the Email source. You can use ${...} macros to expand variables defined in environment variables.

• Ignore folder : The directory to which emails are moved in case they are configured to be ignored in the Ignore emails section (see next). Leaving it empty, the emails will not be moved. The configured ignore folder must be accessible to the Reactive Engine trying to access the Email source. You can use ${...} macros to expand variables defined in environment variables.

A final configuration could look like this:

Ignore Emails

By default, all emails arriving in the configured Input folder will be processed. Though it is possible to define which emails are not relevant (Blacklist) resp. explicitly relevant (Whitelist) for processing by activating the Ignore Emails checkbox.

• From filter : Blacklist resp. Whitelist definitions to apply towards email sender values
• Subject filter : Blacklist resp. Whitelist definitions to apply towards the email subject values

Regular expression values can be configured by pushing the respective + ADD TO BLACKLIST resp. + ADD TO WHITELIST button. Logically layline.io will check any configured blacklist configuration first and evaluate any whitelist configuration afterwards.

In case any Blacklist expression is matching an incoming email, the email will be ignored. Having configured an Ignore folder, that specific email will be moved towards that folder. Otherwise, the email remains unprocessed in the Input folder.

Any incoming email matching the Whitelist expression will lead to processing that email and moving it into the Done folder or the Error folder afterwards. In case no specific folder is configured the email will remain in the Input folder.

• If you configure only Whitelist expressions, this will implicitly mark all others to be ignored

• If you configure only Blacklist expressions, this will implicitly define all others to be processed

• Configuring a combination of both (i.e. whitelist AND blacklist have at least one entry each!), it is important to understand the following:

The order of expression evaluation will always be:

1. First Step: Ignore emails matching blacklist
2. Second Step: Process emails matching whitelist
3. Third Step: Ignore emails not falling in any category

Attachments settings

By default, attachments to emails are not loaded. Though it is possible to allow attachment loading by activating the Load attachments checkbox and configure the details:

• Maximum size of all loaded attachments [kb] : limit the overall size of attachments per processed email to not overload your system capacity (value configured in kb)

• Mime type filter : filter conditions to further restrict the loading based on type of attachment and / or size:

  • Blacklist :

    • Mime type regular expression : similar as explained above, a regular expression for type of attachments that should be ignored (e.g. 'application/.*' to exclude all attachments of ' application'-related mime types)
    • Size constraints [kb] : optional parameter that could be used to exclude only those type of attachments with a size greater than the configure size value; no configured value will ignore this parameter

  • Whitelist :

    • Mime type regular expression : similar as explained above, a regular expression for type of attachments that should be included for processing (e.g. 'application/.*' to include only attachments of 'application'-related mime types)
    • Size constraints [kb] : optional parameter that could be used to include only those type of attachments with a size less or equal than the configure size value; no configured value will ignore this parameter. Please note that the configured maximum size takes precedence in the overall setup!

• If you configure only Whitelist expressions, this will implicitly mark all others to be ignored

• If you configure only Blacklist expressions, this will implicitly define all others to be processed

• Configuring a combination of both (i.e. whitelist AND blacklist have at least one entry each!), it is important to understand the following:

The order of expression evaluation will always be:

1. First Step: Ignore emails matching blacklist
2. Second Step: Process emails matching whitelist
3. Third Step: Ignore emails not falling in any category

---

Can't find what you are looking for?

Please note, that the creation of the online documentation is Work-In-Progress. It is constantly being updated. should you have questions or suggestions, please don't hesitate to contact us at support@layline.io .