Workflow Assets

Workflow assets are the building blocks of data processing pipelines in layline.io. Each asset type serves a specific purpose in the ingestion, processing, transformation, and output of data.

Use this page to identify which assets you need for your use case. Each section explains when to use that category, followed by specific assets with their purposes.

---

Connections

When to use: Connections store reusable connection parameters for external systems. Define them once, reference them across multiple assets. Use Connections when you need to connect to the same external system from multiple workflows or assets, and want to centralize credentials and configuration.

• Connection AWS — Centralized AWS credentials and region configuration. Use when connecting to any AWS service (S3, SQS, SNS, Kinesis, etc.) from multiple assets. Supports IAM roles, access keys, and session tokens.

• Connection Email — SMTP server configuration for sending and receiving emails. Use when workflows need to send or receive email notifications. Supports TLS/SSL, authentication, and multiple SMTP servers.

• Connection FTP/SFTP — FTP or SFTP server access credentials. Use for file transfers to/from remote servers. Supports passive/active FTP modes and SSH key authentication for SFTP.

• Connection Google Cloud — Google Cloud Platform service account configuration. Use when connecting to GCP services (GCS, Pub/Sub, BigQuery) from multiple assets. Supports service account keys.

• Connection Kafka — Kafka cluster bootstrap servers and authentication. Use when publishing or consuming from Kafka topics. Supports SASL/SSL authentication and multiple brokers.

• Connection MS Graph — Microsoft Graph API credentials for Office 365 integration. Use when accessing SharePoint, OneDrive, Teams, or other Microsoft 365 services. Supports OAuth2 authentication.

• Connection NFS — Network File System server mount configuration. Use when reading/writing files to NFS shares. Supports NFSv3 and NFSv4 protocols.

• Connection SMB — SMB/CIFS share credentials for Windows file shares. Use when integrating with Windows-based file servers or Samba shares. Supports domain authentication.

• Connection Virtual File System — In-memory or local file system configuration. Use for temporary file operations, testing, or when no external storage is needed.

• Connection WebDav — WebDAV server access configuration. Use when integrating with WebDAV-enabled servers (Nextcloud, ownCloud, SharePoint WebDAV). Supports digest and basic authentication.

---

Extensions

When to use: Extensions add custom functionality, macros, and scripting capabilities to the platform. Use when you need custom JavaScript functions, reusable transformation logic, or integration with external libraries that aren't covered by built-in processors.

• Extension AWS — AWS SDK integration and custom AWS service interactions. Use when you need to call AWS services not covered by built-in assets (Lambda invocation, CloudWatch custom metrics, etc.). Includes AWS SDK methods in JavaScript context.

• Extension Prometheus — Prometheus metrics export and custom metric collection. Use when you need to expose custom metrics for monitoring, alerting, or observability. Integrates with Prometheus scraping and Grafana dashboards.

---

Formats

When to use: Formats define how messages are parsed (when reading) and serialized (when writing). Use Formats when you need to handle specific data formats — choose based on your input data structure and the format your downstream systems expect.

• ASN.1 Format — Abstract Syntax Notation One parsing for telecom, cryptography, and protocols. Use when processing BER/DER encoded data, X.509 certificates, telecommunications protocols (CAMEL, MAP, CAP), or banking protocols. Supports custom ASN.1 schemas.

• Data Dictionary Format — Centralized field definitions and validation rules. Use when you have common data structures across multiple formats (e.g., the same customer fields in CSV, JSON, and XML). Defines fields once, reuse across formats.

• Formats Introduction — Overview of format concepts and when to use each type. Start here if you're unsure which format fits your data.

• Generic Format — Flexible text, binary, or delimiter-based parsing. Use for CSV, fixed-width, custom delimited, or simple text formats. Highly configurable — define delimiters, headers, field mappings, and record structures.

• Http Format — HTTP request/response message parsing. Use when processing HTTP payloads, REST API responses, or webhook data. Handles headers, query parameters, body parsing, and status codes.

• XML Format — XML document parsing with schema validation. Use for SOAP messages, configuration files, or XML-based APIs. Supports XSD validation, namespaces, XPath expressions, and custom parsing rules.

---

Flow Processors

When to use: Flow processors transform, route, and process data within the workflow pipeline between input and output. Use these when you need to modify data, make routing decisions, or apply business logic after ingestion but before output.

• AI Classifier — Automatically classify messages using machine learning models. Use when you need to categorize documents, detect sentiment, route based on content type, or apply ML-based decisioning. Supports custom-trained models.

• AI Trainer — Train machine learning models from your workflow data. Use when you need to create custom classification models based on your specific data patterns. Feeds labeled examples to improve classifier accuracy.

• Filter & Routing — Route messages based on conditions or filter unwanted data. Use when you need if/then/else routing, content-based filtering, or to split a single input stream into multiple output paths based on message content.

• Input Frame Committer — Manage transaction boundaries and batch commits. Use when you need precise control over when input offsets are committed (e.g., ensuring data is fully processed before acknowledging Kafka messages).

• Javascript — Execute custom JavaScript code for data transformation. Use when built-in processors don't meet your needs — write custom logic, call external libraries, or implement complex transformations. Full ECMAScript support.

• Mapping — Transform data structures between formats using visual mapping. Use when converting between different schemas (e.g., internal format to partner API format). Drag-and-drop field mapping with functions and lookups.

• Python — Execute custom Python code for data processing. Use when you need Python libraries (pandas, numpy, custom ML), or when your team prefers Python over JavaScript for transformation logic.

• Stream Boundary — Detect and handle logical message boundaries in streaming data. Use when processing protocols where messages aren't clearly delimited (e.g., FIX protocol, custom TCP streams) to identify where one message ends and another begins.

• Throttle — Rate-limit message processing. Use when downstream systems can't handle high throughput, when you have API rate limits, or to prevent overwhelming external services. Supports burst and sustained rate limiting.

---

Input Processors

When to use: Input processors receive and ingest data from various sources. These are the entry points to your workflow — choose based on where your data originates and how it's delivered.

• Input Frame — Read files from file systems, FTP, or object storage. Use when processing files (CSV, JSON, XML, binary) from local disks, NFS, SMB, S3, GCS, or FTP servers. Supports polling, one-time reads, and file patterns.

• Input Kafka — Consume messages from Apache Kafka topics. Use for event-driven architectures, real-time streaming, or when integrating with Kafka-based systems. Supports consumer groups, offset management, and exactly-once processing.

• Input Message — Receive individual messages via HTTP, TCP, UDP, or internal messaging. Use for API endpoints, webhook receivers, or when external systems push data to you. Supports REST, raw TCP/UDP, and message queuing.

• Input Request-Response — Handle synchronous request-response patterns. Use when building APIs that need to return a response (e.g., REST API with JSON response, SOAP service). Correlates requests with responses.

• Input Service — Trigger workflows based on service calls or scheduled events. Use for timer-based triggers, cron jobs, or when workflows need to be invoked programmatically via API.

• Input Stream — Process continuous data streams. Use for high-throughput, low-latency scenarios where data arrives as a never-ending stream (logs, metrics, IoT telemetry).

---

Output Processors

When to use: Output processors send and output data to destinations. These are the exit points of your workflow — choose based on where your processed data needs to go and in what format.

• Output Dev Null — Discard output for testing or debugging, or to simply destroy a message. Use when you want to test a workflow without actually sending data anywhere, or to measure throughput without I/O overhead.

• Output Frame — Write files to file systems, FTP, or object storage. Use when saving processed data as files (CSV, JSON, XML, binary) to local disks, NFS, SMB, S3, GCS, or FTP servers. Supports file naming patterns, rotation, and batching.

• Output Stream — Send data to streaming destinations. Use for feeding data to Kafka, Kinesis, or other streaming platforms. Supports batching, compression, and delivery guarantees.

---

Resources

When to use: Resources define reusable configuration shared across multiple assets. Use when you have common settings, credentials, or configurations that multiple workflows or assets need to reference.

• AI Model Resource — Deploy and manage machine learning models. Use when using AI Classifier or AI Trainer — defines the model file, version, and runtime parameters.

• Data Dictionary Updates — Manage updates to Data Dictionary definitions. Use when field definitions change over time or need versioning. Allows updating validation rules without modifying formats.

• Directories — Define directory paths for file operations. Use when multiple assets need to access the same directories — centralizes path configuration and access permissions.

• Environment — Store environment variables and configuration values. Use for settings that differ between dev/staging/prod, or for sensitive values that shouldn't be hardcoded in workflows.

• OAuthClient — OAuth2 client credentials for API authentication. Use when connecting to OAuth2-protected APIs (Salesforce, Microsoft 365, Google APIs, etc.). Handles token refresh automatically.

• Secret — Store sensitive data like passwords, API keys, and certificates. Use for any credential that needs secure storage. Integrated with layline.io's security storage — secrets are encrypted at rest.

• StatusDefinition — Define custom status codes and tracking fields. Use when workflows need to track custom business states beyond standard success/failure (e.g., "pending approval", "manual review required").

---

Services

When to use: Services interface with external APIs, databases, and cloud services from within your workflows. Use these when you need to call external systems during processing — choose based on the external service you're integrating with.

• AI Service — Call external AI/ML APIs (OpenAI, Azure ML, etc.). Use when you need LLM integration, third-party ML models, or AI services not covered by built-in AI processors.

• Aerospike Service — Read from and write to Aerospike databases. Use for high-performance key-value operations when Aerospike is your data store. Supports batch operations and complex queries.

• Cassandra Service — Query and update Apache Cassandra or compatible databases like AWS Keyspaces. Use when Cassandra is your primary data store. Supports CQL queries, batch operations, and asynchronous processing.

• DynamoDB Service — Access AWS DynamoDB tables. Use for serverless applications or when DynamoDB is your data store. Supports get, put, query, scan, and batch operations with automatic retries.

• Email Service — Send emails via configured connections. Use for notifications, alerts, or reports. Supports HTML/text templates, attachments, and multiple recipients.

• HTTP Service — Call REST APIs and HTTP endpoints. Use when integrating with any HTTP-based service (partner APIs, microservices, webhooks). Supports GET, POST, PUT, DELETE, custom headers, and authentication.

• Hazelcast Service — Access Hazelcast in-memory data grid. Use for distributed caching, shared state across workflows, or high-speed data access. Supports maps, queues, and distributed locks.

• JDBC Service — Execute SQL queries against relational databases. Use for PostgreSQL, MySQL, Oracle, SQL Server, or any JDBC-compatible database. Supports parameterized queries, transactions, and connection pooling.

• KVS Service — Generic key-value store operations. Use for simple get/put operations against Redis, Memcached, or custom KVS implementations. Supports TTL, atomic operations, and batch gets.

• Message Service — Send messages to message queues and topics. Use for asynchronous communication between workflows or systems. Supports JMS, AMQP, and custom message brokers.

• Proxy Service — Route HTTP requests through proxy servers. Use when external APIs require specific proxy configurations, or for traffic routing through corporate proxies.

• Queue Service — Interface with message queues (RabbitMQ, ActiveMQ, etc.). Use for reliable message delivery, load leveling, or decoupling workflow components. Supports persistent and transient queues.

• SOAP Service — Call SOAP web services. Use when integrating with legacy enterprise systems or services that only expose SOAP interfaces. Supports WSDL parsing, SOAP headers, and attachments.

• Sequence Number Service — Generate unique sequence numbers. Use when you need guaranteed-unique IDs, sequential tracking numbers, or distributed sequence generation across multiple workflow instances.

• Services Introduction — Overview of service concepts and selection guide. Start here if you're unsure which service fits your integration need.

• Teams Service — Send notifications to Microsoft Teams channels. Use for alerting, status updates, or workflow notifications. Supports adaptive cards, mentions, and channel/webhook targets.

• Timer Service — Schedule delayed or periodic operations. Use for timeout handling, scheduled retries, or time-based workflow logic. Integrates with workflow scheduling.

• UDP Service — Send UDP datagrams. Use for lightweight, high-performance messaging where delivery isn't guaranteed (metrics, logs, real-time telemetry). Supports multicast and broadcast.

• Virtual File System Service — Access a virtual file system source from within a Javascript/Python Asset.

---

Sinks

When to use: Sinks are destination endpoints for data output and delivery. Unlike Output Processors (which are workflow exit points), Sinks are reusable destination definitions that can be referenced by multiple outputs. Use Sinks when the same destination is used from multiple workflows.

• Sink EventBridge — Send events to AWS EventBridge for serverless event routing. Use when integrating with AWS serverless architectures, triggering Lambda functions, or routing events to multiple AWS services.

• Sink FTP — Upload files to FTP/SFTP servers. Use for file delivery to remote servers, partner file exchanges, or legacy system integration. Supports passive/active modes and resume.

• Sink File System — Write files to local or mounted file systems. Use for local file output, NAS storage, or when downstream systems read from shared drives.

• Sink GCS — Upload files to Google Cloud Storage. Use when target systems are GCP-based, or for cloud storage integration with Google services.

• Sink Kafka — Publish messages to Kafka topics. Use for event streaming, feeding downstream consumers, or integrating with Kafka-based architectures. Supports partitioning and exactly-once delivery.

• Sink Kinesis — Send records to AWS Kinesis streams. Use for real-time data streaming into AWS, feeding analytics pipelines, or serverless event processing.

• Sink NFS — Write files to NFS shares. Use for Unix/Linux environment integration, shared storage scenarios, or when target systems mount NFS volumes.

• Sink OneDrive — Upload files to Microsoft OneDrive. Use for Microsoft 365 integration, user file delivery, or SharePoint Online document libraries.

• Sink S3 — Upload files to AWS S3. Use for cloud storage, data lakes, or AWS-native integrations. Supports multipart upload, encryption, and storage classes.

• Sink SMB — Write files to SMB/CIFS shares. Use for Windows server integration, shared network folders, or when target systems use Windows file sharing.

• Sink SNS — Publish notifications to AWS SNS topics. Use for fan-out messaging, mobile push notifications, or triggering multiple Lambda functions from a single event.

• Sink SQS — Send messages to AWS SQS queues. Use for reliable message delivery to AWS-based consumers, decoupling systems, or load leveling.

• Sink SharePoint — Upload files to Microsoft SharePoint. Use for document management, collaboration scenarios, or Microsoft 365 integration.

• Sink Virtual File System — Write to virtual file systems.

• Sink WebDAV — Upload files to WebDAV servers. Use for WebDAV-enabled content management systems, Nextcloud, ownCloud, or SharePoint WebDAV access.

• Sink WebSocket — Send messages over WebSocket connections. Use for real-time web applications, live dashboards, or bidirectional web communication.

---

Sources

When to use: Sources are origin endpoints for data input and ingestion. Unlike Input Processors (which are workflow entry points), Sources are reusable source definitions that can be referenced by multiple inputs. Use Sources when the same origin is read from multiple workflows.

• Source Email — Poll email inboxes for incoming messages. Use for email-based workflows, processing attachments, or triggering flows from email receipts. Supports IMAP, POP3, and attachment extraction.

• Source FTP — Monitor FTP/SFTP directories for new files. Use for partner file exchanges, legacy system integration, or when external systems drop files on FTP servers. Supports polling and file patterns.

• Source File System — Monitor local or mounted directories for file changes. Use for file-based integration, watching hot folders, or processing files written by other systems. Supports recursive watching and file patterns.

• Source Google Cloud Storage — Monitor GCS buckets for new objects. Use for cloud-native file processing, data pipeline triggers, or GCP integration. Supports event-based and polling modes.

• Source Http — Create HTTP endpoints that receive data. Use for webhook receivers, REST API inputs, or when external systems POST data to you. Supports authentication and request validation.

• Source Kafka — Consume from Kafka topics. Use for event-driven architectures, real-time streaming, or integrating with existing Kafka infrastructure. Supports consumer groups and offset management.

• Source Message — Receive messages from message queues or topics. Use for JMS, AMQP, or custom message broker integration. Supports transactions and acknowledgment modes.

• Source NFS — Monitor NFS directories for files. Use for Unix/Linux file system integration, shared storage scenarios, or when source data is on NFS mounts.

• Source OneDrive — Poll OneDrive folders for new files. Use for Microsoft 365 integration, user-uploaded content, or SharePoint Online document processing.

• Source S3 — Monitor S3 buckets for new objects. Use for cloud storage triggers, data lake ingestion, or AWS-native event processing. Supports S3 event notifications and polling.

• Source SMB — Monitor SMB/CIFS shares for files. Use for Windows environment integration, shared network folders, or when source systems use Windows file sharing.

• Source SQS — Poll AWS SQS queues for messages. Use for reliable message consumption from AWS, decoupling from AWS-based producers, or serverless integration.

• Source Service — Trigger workflows via API calls or schedules. Use for programmatic workflow invocation, cron-based triggers, or service-oriented architectures.

• Source SharePoint — Monitor SharePoint document libraries. Use for document management workflows, Microsoft 365 integration, or corporate content processing.

• Source TCP — Listen for TCP connections and receive data. Use for raw TCP protocols, custom binary protocols, or high-performance streaming. Supports framing and delimiters.

• Source Timer — Trigger workflows on schedules. Use for cron jobs, periodic data exports, or time-based workflow initiation. Supports cron expressions and intervals.

• Source UDP — Listen for UDP datagrams. Use for lightweight, high-throughput data ingestion where delivery isn't guaranteed (metrics, logs, telemetry). Supports multicast.

• Source Virtual File System — Read from virtual or in-memory file systems. Use for testing, temporary data, or when source data is generated programmatically.

• Source WebDAV — Monitor WebDAV directories for files. Use for WebDAV-enabled systems, Nextcloud, ownCloud, or SharePoint WebDAV folders.

---

Workflows

When to use: Workflows orchestrate and coordinate data processing pipelines. The Workflow asset defines how inputs, processors, and outputs connect to form a complete data flow. Every processing pipeline needs at least one Workflow asset.

• Workflow — Define a complete data processing pipeline. Use to connect Sources/Inputs to Processors to Sinks/Outputs. Defines the flow of data, error handling, parallelization, and resource allocation. Every data pipeline is a Workflow.

---

Asset Categories Overview

Category	When to Use
Sources	Reading from external systems that multiple workflows need to access. Define once, reference from many inputs.
Sinks	Writing to external systems that multiple workflows need to target. Define once, reference from many outputs.
Input Processors	The entry point to your workflow — choose based on how data arrives (files, messages, streams, APIs).
Output Processors	The exit point from your workflow — choose based on where data needs to go (files, queues, streams).
Flow Processors	Transforming, routing, or enriching data between input and output. Business logic lives here.
Formats	Parsing input data or serializing output data. Match the format to your data structure.
Connections	Reusable connection credentials for external systems. Centralizes configuration and security.
Services	Calling external APIs, databases, or services during processing. Integration logic lives here.
Resources	Shared configuration, credentials, or state across multiple assets. Reduces duplication.
Extensions	Custom code and functionality beyond built-in processors. For unique requirements.
Workflows	Orchestrating the complete data pipeline. Defines how all other assets connect and interact.

---

Quick Decision Guide

Need to read files? → Source File System, Source FTP, Source S3, or Input Frame

Need to receive HTTP calls? → Source Http or Input Message

Need to stream events? → Source Kafka, Input Kafka, or Input Stream

Need to transform data? → Flow Processors (Mapping, Javascript, Python)

Need to route based on content? → Filter & Routing

Need to call an API? → HTTP Service or SOAP Service

Need to query a database? → JDBC Service, Cassandra Service, DynamoDB Service, or Aerospike Service

Need to write files? → Sink File System, Sink S3, Sink FTP, or Output Frame

Need to send messages? → Message Service, Queue Service, Sink Kafka, or Sink SQS

Need AI/ML? → AI Classifier, AI Trainer, AI Service, or AI Model Resource

Need to reuse credentials? → Connections

Need custom logic? → Javascript or Python Flow Processors, or Extensions