Skip to main content

Aerospike Service

Purpose​

Define a service to interface with an Aerospike store.

Note: Only works with Aerospike Scalar data types

Please note, that the Aerospike Service for the moment only supports Aerospike primitive data types.

Prerequisites​

None

Configuration​

Name & Description​

  • Name : Name of the Asset. Spaces are not allowed in the name.

  • Description : Enter a description.

The Asset Usage box shows how many times this Asset is used and which parts are referencing it. Click to expand and then click to follow, if any.

Required roles​

In case you are deploying to a Cluster which is running (a) Reactive Engine Nodes which have (b) specific Roles configured, then you can restrict use of this Asset to those Nodes with matching roles. If you want this restriction, then enter the names of the Required Roles here. Otherwise, leave empty to match all Nodes (no restriction).

Hosts​

This is where you define the host seed node address(es) which can be used to connect to an Aerospike cluster.

The Aerospike Service will first try to connect to the first seed node you provide. If you have provided multiple seed nodes the service iterates through the array of nodes until it successfully connects to a node. Once a connection has been established all other nodes within the Aerospike cluster will be automatically discovered by the service.

Retries & Timeouts​

For the purpose of sturdiness in communicating with the Aerospike cluster, you may define retries and timeouts. This allows the system to retry requests in case they don't go through or are acknowledged the first time. Please note that this settings applies to all communications. So in this case all reads, writes, and delete requests.

  • Max. retries : Maximum number of times the system should retry to issue request.

  • Socket timeout [ms] : Timeout in milliseconds until a request is considered failed.

  • Total timeout [ms] : Timeout in milliseconds for all retries in total combined. Here you can define that you do not want to wait any longer than x milliseconds regardless of the number of retries.

Introduction to CRUD operations on Aerospike with the Aerospike Service​

CRUD is a general term to describe Create, Read, Update (or write), and Delete operations in stores in general. Within the Aerospike Service all operations are possible, with some limitations which we will discuss.

Create​

In Aerospike, data structures exist by inserting them. There is no actual “create” operation equivalent to a relational database, for example. In that sense, data is created by the equivalent of “write” operations. Therefore, “create” operations can be performed by “write” operations from this Aerospike Service Asset.

No dynamic create

In this Aerospike Service Asset there is no support, to dynamically create structures (equivalent to DDL operations). Namespaces, Sets and Bins are statically defined in this Asset.

Read, Update and Delete Operations​

Read, update and delete Operations are all supported. Aerospike - as a key-value-store - has many degrees of freedom in regard to what is actually stored in the value part of a record. It is for example possible to create a record with no Bins, the next with one Bin, and the next with three Bins. There is no underlying schema.

layline.io, however, requires some structure in order to map Aerospike data to layline.io and vice versa. It is therefore required to work with Aerospike Namespaces, Sets, and Bins. Bins would be the considered the same as fields within a traditional RDBMS schema, and map to fields within layline.io’s message structure.

Supported Data Types in Bins​

Aerospike supports a number of different data types, not all of which are supported by this Aerospike Service Asset. Supported are:

  • Boolean
  • Byte
  • Bytes
  • Double
  • Float
  • Integer
  • Long
  • String

Other data types custom to Aerospike are currently not supported. Drop us a note in case you require them. For further information on Aerospike’s data types check their documentation.

Two ways on how to work with data in the Aerospike Service​

You have two options on how to work with data in Aerospike using this Service:

  1. Service Functions
  2. Collections

Service Functions allow you to define a schema that you either want to read, write, or delete from/to. You can for example define a Function which writes a specific Bin in a given Namespace and Set, and name the Function “UpdateThisSpecificField”.

Collections on the other hand allow you to define a schema and will automatically create respective read, write, AND delete from/to methods. You can for example define a Collection “CustomerData” which configures access to a Namespace, Set, and Bins, and the Collection will automatically provide read, write, and delete methods.

Use Functions if you want to have granular control over what you want to do (read, write, delete), and use Collections for a more convenient and general way to read, write and delete from/to a configurable schema.

There is a good chance that defining Collections will be easier and sufficient for you. You can use Functions and Collections concurrently, however.

tip

Please note that a Collection as defined here is not equivalent to Aerospike’s Collection data type.

Example

For the purpose of further explanation, let's make up an example. Let's assume we want to work customer data. For each customer Aerospike stores a value which contains

  • a phone number (MSISDN) and
  • an array of history records which contain additional information in additional fields.

The logical record structure looks like this:

{
  "MSISDN": "017212345678",
  "History": [
    {
      "ValidFrom": "2021...",
      "ValidTo": "2021...",
      "Provider": "XYZ",
      "PaymentType": "PT"
    }
  ]
}

Let's further assume that in Aerospike this is stored in a Namespace/Set/Bins setup like so:

In our example, records are stored with the MSISDN as the key.

Service Functions​

As explained above, Service Functions allow you to define a granular method to either read, write, or delete from/to a Aerospike Namespace/Set/Bins combination.

Let’s assume we would only want to read the Bin MSISDN from our example.

Create Service Function​

First create a new Function (1):

Next fill out the details:

  • Function name (1): The name of the function. Must not have whitespaces.

  • Function description (2): Something which describes the function (optional).

  • Aerospike namespace (3): The name of the Aerospike namespace.

  • Aerospike Set (4): If the records within Aerospike are grouped into sets, then enter the set name here. This is optional.

  • Function type (5): Pick either Read, Write, or Delete. In our example we pick Read.

Create MSISDN Bin​

Configure the MSISDN Bin from our example.

To create the Bin click on ADD BIN . This will create a Bin we want to read in this Function.

Enter additional values:

Bin

  • Aerospike bin name (2): Name of the Bin. Must be the name of the Bin in Aerospike.
  • Aerospike bin type (3): This is the type of how the data is stored in Aerospike. Note that Aerospike automatically deduces the data from its content.
  • Property name in parameter/result message (4): This is the property name of how the value of the Bin will be read and set within layline.io’ internal message structure. This become clear when we describe how the data is referenced when working with it.

Encoding - Internal encoding. This defines how the value from the Bin is treated within layline.io. This is important to understand: We always need to define how we can map an external value - in this case from Aerospike - to an internal structure and vice versa. It’s relatively simple for primitive data types, e.g. a Bin of type String in Aerospike is a String in layline.io. It gets more interesting with complex data types. This is where Data Dictionary definitions come into play. We will look at this in more detail when defining the History Bin a bit further below.

  • Encoding type (5): You use the encoding type to define how values and structure are decoded into layline.io structures when reading from the source, an encoded respectively when writing to the source (Aerospike). You can choose between three different encoding types:

    • Value (5): This option supports one-to-one mapping of primitive encoding types, e.g. String to String, or Boolean to Boolean. You can also map a String to an Integer for example which will map a number which is stored in a String on Aerospike into an Integer on layline.io. Be careful, however, because these implicit type conversions must work or it will create an error.

  • Data Dictionary: This option will allow you to define a complex structure by way of defining a Data Dictionary structure. We will go through an example in the next chapter.

  • Format: This option will allow you to reference a Format which you have defined previously. An example would be that the Bin holds a comma separated record structure for which you have defined a Generic Format . If you assign this Format here, then it will be applied to decode the bin content using this Format, as well as encode the structure when writing the data. This is extra helpful in the case that the content of a Bin is of a structure which cannot easily be defined using a Data Dictionary structure (see below), for example. Let’s assume the Bin contains an ASN.1 encoded structure, you can then first define a ASN.1 format and assign it here for decoding and encoding the data respectively.

Result​

We now have defined a Service Function which can read an Aerospike Namespace, Set, and one Bin MSISDN with a given key.

To understand how to then use this Function please check chapter Using the Aerospike Service

Collections​

Collections in this context describe a schema like structure. Imagine in Aerospike you have a Namespace, and Set with a number of defined Bins in the records of that Set. A Collection would describe exactly that structure. Once defined it automatically provides all functions necessary to read, write, and delete data from that structure.

Configure the Collection​

We use the UI to configure the Collection first. Click ADD COLLECTION to create a new Collection (1):

A Collection will be created. Next we enter the Collection details:

  • Name & Description (1, 2): Enter the name and a description (optional) of the Collection. The name is mandatory and must be unique within the Service. You may not have a Function by the same name. It may not contain spaces.

  • Namespace (3): The name of the Aerospike namespace.

  • Set (4): If the records within Aerospike are grouped into sets, then enter the set name here. This is optional.

In the following step we need to define the structure within the Collection. Please check above for the structure we are trying to define (two Bins).

Configure the MSISDN Bin​

This is the same as in chapter Create MSISDN Bin

Configure the Data Bin​

In our example we still need to configure the Data bin.

Defining a complex data type Because the Data bin is not a primitive type, we first need to define the complex type which it has. Go to the following chapter “Data Dictionary” to learn how it is configured; then return here to complete to configuration.

Now that we have added the History data type we can actually add the History Bin:

For our example:

Bin

  • Aerospike Bin Name (1): The name of the Aerospike Bin: History.

  • Aerospike Bin Type (2): The data is stored as a String.

  • Property name in parameter/result message (3): Use the same name History to later reference the data within layline.io.

Encoding

  • Encoding type (4): Pick Data Dictionary from the list. This is necessary to reference the data type in the next step, which we have defined in the Data Dictionary (next chapter).

  • Message property type (5): Now enter the access path to the data type: MyNamespace.History[]. Note that we have added a [] at the end of the data type to mark it as an array of this type.

  • Serialization type (6): Pick Json. This means that data read from this Aerospike Bin will be deserialized to Json format, and serialized from Json to string when written.

Automatically generated Functions​

When defining a Collection, layline.io automatically creates three different functions for reading, writing, and deleting data from/to the Collection. For the Collection CustomerData which we have created, the following functions will be created:

FunctionSignatureReturnsDescriptionExample
Readservices.<Logical Service Name>.Read<BinName>(key: String)Message or null of nothing foundRead data from Aerospikeservices.CustomerData.ReadCustomerData({Key: "017212345678"})
Writeservices.<Logical Service Name>.Write<BinName>(key: String, Bin: {...}, WritePolicy: {...})nullWrite data to Aerospikeservices.CustomerData.WriteCustomerData({Key: "017212345678", Bin: {...}, WritePolicy: {...}})
Deleteservices.<Logical Service Name>.Delete<BinName>(key: String)Message or null if nothing deletedDelete data from Aerospikeservices.CustomerData.DeleteCustomerData({Key: "017212345678"})

As you can tell, a Collection name is simply prepended by Read, Write, and Delete for the respective functionality.

tip

If you only want to have a Read function you can achieve the same by configuring a corresponding Function, instead of a Collection.

Result​

We now have defined a Collection which references an Aerospike Namespace, Set, and two Bins MSISDN and History with a simple and complex type.

Data Dictionary​

The Data Dictionary allows you to define complex data structures which can be mapped onto external data types and vice versa. This is necessary whenever an asset needs to exchange structured data with an external system — for example, when reading from or writing to a database, an HTTP API, a message queue, or any other format that carries typed fields.

Rather than hard-coding external field names and types into your Workflow, you define your own internal data types here. These internal types are then mapped to the external system's fields at the asset level. This means your Workflow scripts work with consistent, self-documenting data structures regardless of which external system the data came from.

When you need it​

Whenever you configure an asset that exchanges structured data — a JDBC Service, a DynamoDB Service, an HTTP endpoint, an MQ message, a database Resource — you use the Data Dictionary to declare the types that represent:

  • Request parameters — the data your Workflow sends to the external system
  • Result data — the data the external system returns to your Workflow
  • Intermediate structures — types that hold data during a transformation

Entity Types​

The Data Dictionary is organized as a tree of typed entities. The available entity types are:

EntityDescription
NamespaceGroups related types. Optional. If you reuse a namespace name that already exists in the Project, the two namespaces merge.
SequenceAn ordered list of typed members. Members are accessed by name, e.g. MyNamespace.Customer.Name.
EnumerationA fixed set of named integer constants.
ChoiceA type that holds exactly one of several possible member types.
ArrayA sequence of elements of a single contained type.

Defining Types — Step by Step​

The following walkthrough shows how to build a data structure using the Data Dictionary editor. The example assumes a SQL customer table with columns id, name, and address — but the same pattern applies whenever you need to declare types for any asset.

1. Declare a new type​

Click Declare Root Type in the toolbar to add a top-level entity.

Declare root type

2. Declare a namespace (optional)​

Namespaces organize related types. To add one, right-click an existing node and select Add Sibling, then set the element type to Namespace.

Declare namespace

  • Name — The name of the namespace. If a namespace with this name already exists elsewhere in the Project, their contents merge automatically. Otherwise the name must be unique and may not contain spaces.

  • Type — Pick the entity type. For a namespace, select Namespace.

  • Description — Optional free-text description.

3. Declare a Sequence under the namespace​

Right-click the namespace and choose Add Child to add a child element.

Add child to namespace

Click the arrow next to the namespace name and select Add child. Then fill in the element details:

Declare sequence

  • Name — The name of the element, e.g. Customer.

  • Type — Select Sequence as the element type. You will add individual fields (members) in the next step.

  • Extendable Sequence — When checked, layline.io can dynamically extend the sequence's member list if incoming data contains fields that are not explicitly defined. Leave unchecked if all fields are known in advance.

4. Add members to the Sequence​

With the Sequence selected, click Add Child to add individual fields:

Add sequence members

Each member maps to a column in the external data source. You can reference any member by its full path — for example, MyNamespace.Customer.Name — from your Workflow scripts.

Common Entity Fields​

These fields are available on all entity types:

FieldDescription
NameUnique identifier within the namespace. Reusing a namespace name from another part of the Project merges the two.
TypeThe entity kind: Namespace, Sequence, Enumeration, Choice, or Array
DescriptionOptional free-text description
Extendable Sequence(Sequence only) Allows the member list to be extended dynamically at runtime
Members(Sequence) Ordered list of typed fields — click Add Child to add each one
Elements(Enumeration) Named integer constants making up the enumeration

Advanced Features​

Inheritance and Override
Entities inherited from a parent format or resource appear in the tree in a distinct inherited style. These are read-only unless overridden. Click Reset to Parent on an overridden entity to restore the inherited definition.

Copy and Paste
Use the toolbar buttons to copy a complete entity subtree and paste it elsewhere in the tree. All members and nested entities travel with it.

Filter and Sort
Use the Filter field to search entities by name. The sort buttons order nodes ascending or descending alphabetically.

See Also​

Example: Using the Aerospike Service​

The Aerospike Service can be used from within a JavaScript Asset. In our example we have a simple Workflow which reads a file with communication records of customers (1), then in a next step (2) reads corresponding history data from Aerospike, and simply outputs this data to the log. There is no other purpose in this Workflow than to demonstrate how to use the Service.

In the middle of the Workflow we find a JavaScript Processor by the name of “Tap3Debug”. This Processor reads additional customer information from Aerospike using the Aerospike Service.

How is it configured?

To use the Aerospike Service in the JavaScript Processor, we first have to assign the Service within the JavaScript Processor like so:

  • Physical Service (1): The Aerospike Service which we have configured above.

  • Logical Service Name (2): The name by which we want to use the Service within JavaScript. This could be the exact same name as the Service or a name which you can choose. Must not include

Access the Service from within a Script Processor​

Now let's finally use the service within a script processor:

Reading from Aerospike​

Signature: services.<Logical Service Name>.<Read<Collection> or Functionname>({Key: key})

Example: services.CustomerData.ReadCustomerData({Key: msisdn})

let aerospikeData = null;
try {
    // Service defined as synchronous. Therefore no promise:
    // Servcie access defined as synchronous. Therefore no promise syntax here
    aerospikeData = services.CustomerData.ReadCustomerData(
        {Key: msisdn}
    );
    // services: fixed internal term to access linked services
    // CustomerData: The logical name of the service which we have given to it
    // ReadCustomerData: Collection function to read the customer data with the given Key
} catch (error) {
    // handle error
}

// Output the MSISDN data
processor.logInfo('MSISDN: ' + aerospikeData.data.Bin.MSISDN);
processor.logInfo('History: ' + aerospikeData.data.Bin.History.PaymentType);
Insert/Update to Aerospike​

Signature: services.<Logical Service Name>.<Write<Collection> or Functionname>({Key: key, Bin: {bin1:value, bin2: value, ...}})

Example: services.CustomerData.WriteCustomerData({Key: msisdn, Bin: {MSISDN: msisdn, History: msisdnData.history}})

Properties:

  • Key: is the key of the record to be written.
  • Bin: is an object which contains the Bins to be written.
  • WritePolicy: This is where you can define some Aerospike specific write policies.
    • Generation [number]: This is the generation of the record. If the record has been updated since the last read, then the write will fail. If not defined, then write will always succeed.
    • Expiration [number]: This is the time in seconds after which the record will be deleted from Aerospike. If not defined, then the record will not expire.
    • GenerationPolicy: This defines the generation policy. If not defined, then the generation policy is NONE. If set to EXPECT_GEN_EQUAL, then the write will only succeed if the generation of the record is equal to the generation defined in the Generation property. If set to EXPECT_GEN_GT, then write will only succeed if the generation of the record is greater than the generation defined in the Generation property.
    • RecordExistsAction: This defines what to do if the record already exists. If not defined, then the record will not be written. If set to UPDATE, then the record will be updated. If set to UPDATE_ONLY, then the record will only be updated if it already exists. If set to REPLACE, then the record will be replaced. If set to REPLACE_ONLY, then the record will only be replaced if it already exists. If set to CREATE_ONLY, then the record will only be created if it does not exist.
try {
    services.CustomerData.WriteCustomerData(
        {
            Key: msisdn,
            Bin: {
                MSISDN: msisdn,
                History: msisdnData.history
            },
            WritePolicy: {
                Expiration: 3600
            }
        }
    )
} catch (error) {
    // handle error
}
Deleting from Aerospike​

Signature: services.<Logical Service Name>.<Delete<Collection> or Functionname>({Key: key})

Example: services.CustomerData.DeleteCustomerData({Key: msisdn})

try {
    services.CustomerData.DeleteCustomerData(
        {Key: msisdn}
    )
} catch (error) {
    // handle error
}

Service Testing​

layline.io provides a test facility for testing your Services before you deploy them. In this way, you save time and effort by testing your Services without having to deploy and activate a whole Project with Workflows.

Once you have configured your Service(s), you can test them: Within your Asset Configuration tab (1), switch to the Test tab (2) to test your Service.

Test Facility Toolbar​

The toolbar provides the following options:

The Testing tab provides two major views:

  1. Testcase configuration: This is where you define the testcases to be executed.
  2. Testcase execution: This is where you can execute the testcases and see the results.

You switch between these two views by clicking on the leftmost icon in the toolbar (1).

Let's start with the Testcase configuration view.

Testcase Configuration​

The concept of the Testing is to define a set of Testcases which can be executed in a batch or individually. For this purpose, you can define multiple Testcases and configure them individually. I.e. each Testcase groups a number of indidivual tests which can be executed individually or in a batch.

Adding a Testcase​

Click Add Testcase in the toolbar to add a new testcase:

A new Testcase is added. It is automatically named New<Service Asset Name>Test (3) and added to the list of Testcases (2).

  • Service test name (3): You can change the name of the Testcase here.
  • Service test description (4): You can add a description to the Testcase here.

Test Case Setup​

Basics​

In this section you define the individual tests to be executed for this Testcase.

To start, click # END in the toolbar:

A new test is added to the list of tests (1), and the test is opened for configuration (2).

Next we fill in the details:

  • Test name (3): You can change the name of the Test here.

  • Test description (4): You can add a description to the Test here.

  • Service function to test (5): Select the Service function to test here.

    This list contains all Service functions which are defined in the Service Asset. Pick the one you want to test.

    Once a Service function is selected, the system will automatically create a skeleton to fill in the respective parameters for the selected Service function.

Service Function Input Parameters​

  • Service Function Input Parameters (6): Fill in the respective parameters for the selected Service function.

    In our example we have a function GetAlertsForSite which takes two parameters baseurl and riskId. If we click on Add member in the skeleton table the system will allow you to select the respective parameter from the list of available parameters:

    Once you have selected the parameter, the system will automatically add the respective parameter name. You then add the respective value for the parameter:

Service Function Evaluation Parameters​

To automatically evaluate the result, you can add a script which analyzes the results.

Testcase Execution​

Once you have configured your Testcases, you can execute them.

There are two ways on how to trigger execution:

  • Option 1: Select Run selected test in the toolbar (1) to execute the currently selected Testcase.

    Executing a test this way will switch the tab to the Testcase execution view, execute the test and show the results.

  • Option 2: Switch to the Testcase execution view by clicking on the leftmost icon in the toolbar (1) select the test to execute, and then hit the play button next to the test.

Each option will take us to the Testcase execution view:

In this view you can find the Testcase (1) and the Tests (2) we have created. If we had created additional tests for this Testcase, they would be listed here as well.

Question marks indicate that the test has not yet been executed.

We can now either execute all tests, or run them individually:

  • Run all Tests (1): Click this button to execute all tests.

  • Run Testcase (2): Click this button to a Testcase with all its underlying individual tests.

  • Run individual Test (3): Click this button next to a test to execute this individual test.

Once a test has been executed, the question mark will be replaced by a green check mark or a red cross depending on whether the test was successful or not.

The right hand-panel will show the results of the test execution respectively:

In case of errors, the system will show the error message for further investigation.

Can't find what you are looking for?

Please note, that the creation of the online documentation is Work-In-Progress. It is constantly being updated. should you have questions or suggestions, please don't hesitate to contact us at support@layline.io .