Skip to main content

Create Your Data Flow

When testing or validating data flows, we frequently encounter the need to generate specific datasets. To accomplish this, we can leverage both user data input and extra data file. By combining these two features and coordinating input and output files, we can create interconnected data generation processes that seamlessly integrate with our desired data flow.

The following is an example of creating data to simulate a customer email update scenario.

Environment​

Assume your application stores customer data in a PostgreSQL database table like the one below:

Customer Table

Your application also exposes two Restful endpoints to update customer data:

  1. POST /customers/auth: This endpoint authenticates the customer and returns a token.
Auth Request
{
  "user": "username - text",
  "password": "password - text"
}
  1. PUT /customers/password: This endpoint updates the customer's email.
Email Update Request
{
  "id": "customer id in database - integer",
  "email": "current email - text",
  "newEmail": "new email - text"
}

Stepped Data Generation​

Step 1: Generate Database Records​

First we can generate customer database records using the following rules:

  1. username is formed by join firstname and lastname with a dot.
  2. password for all test records is "passw0rd" and the password in the database is hashed using bcrypt.
Schema File
name: "extra_output_1"
entities:
  - name: "customer"
    properties:
      - name: "id"
        type: "sequence"
        index:
          id: 0
      - name: "email"
        type: "text"
      - name: "first_name"
        type: "text"
      - name: "last_name"
        type: "text"
      - name: "username"
        type: "text"
      - name: "name"
        type: "text"
      - name: "raw_password"
        type: "text"
      - name: "password"
        type: "text"

Above files can be downloaded from the following links:

Two files will be generated in this step:

  1. customer_0.sql - SQL file to insert customer records into the database.
  2. extra.csv - CSV file to be used in the next step.

Sample snippet of the extra.csv file:

Sample extra.csv snippet

Step 2: Generate Email Update Requests​

Apart from using the extra.csv from the step 1 as the Data CSV File, we can generate email update requests for test customer records by applying configurations files below:

Schema File
name: "extra_output_2"
entities:
  - name: "my_data"
    properties:
      - name: "customer_id"
        type: "sequence"
      - name: "password"
        type: "text"
      - name: "username"
        type: "text"
      - name: "email"
        type: "text"
      - name: "group"
        type: "number"
  - name: "updated_customer"
    properties:
      - name: "id"
        type: "sequence"
        reference:
          entity: "my_data"
          property: "customer_id"
          linked: "customer_id"
      - name: "password"
        type: "text"
        reference:
          entity: "my_data"
          property: "password"
          linked: "customer_id"
      - name: "username"
        type: "text"
        reference:
          entity: "my_data"
          property: "username"
          linked: "customer_id"
      - name: "email"
        type: "text"
        reference:
          entity: "my_data"
          property: "email"
          linked: "customer_id"
      - name: "new_email"
        type: "text"

Downloadable version of the files:

Other than the regular data files, an extra json file will be generated in this step:

  1. extra.json - JSON file to be used in the test scenario.

Payload of the final json file looks like the following:

Sample output json payload
{
  "updateEmailRequests" : [ {
    "newEmail" : "mxyx.u1stw6a@pea1zeuz.inc",
    "id" : 1,
    "email" : "caldo.uve0eww@4bdmr5g.net"
  }, {
    "newEmail" : "hxusf.l2hqoi@wxpzfm.info",
    "id" : 2,
    "email" : "vzc3mci.yijm@g75.info"
  }, {
    "newEmail" : "lzevix.upw@2d3vmn.net",
    "id" : 3,
    "email" : "xtsb.8e7rhnt@3dskei.io"
  }, {
    "newEmail" : "zlixdcg.edwpe@8nef.com",
    "id" : 4,
    "email" : "hrffo.naxymd@5tiz.app"
  }, {
    "newEmail" : "owve.z8i7@pvlvsiw.org",
    "id" : 5,
    "email" : "vtzlawkg.kjnspgs@gw21.io"
  }, {
    "newEmail" : "4ytps8f.ivhren@axldzz7r.org",
    "id" : 6,
    "email" : "sdh.mz082xk@2upy9.org"
  }, {
    "newEmail" : "koquyzo.ded3@3ae.com",
    "id" : 7,
    "email" : "4igph85.twh@zslfhiid.org"
  }, {
    "newEmail" : "dsq.nsgt@spom.inc",
    "id" : 8,
    "email" : "nslg00m6.om4a5vq@e8gk.net"
  }],
  "loginRequests" : [{
    "password" : "passw0rd",
    "user" : "herta.krajcik"
  }, {
    "password" : "passw0rd",
    "user" : "rolando.kozey"
  }, {
    "password" : "passw0rd",
    "user" : "jerold.bayer"
  }, {
    "password" : "passw0rd",
    "user" : "eric.rosenbaum"
  }, {
    "password" : "passw0rd",
    "user" : "gary.parisian"
  }, {
    "password" : "passw0rd",
    "user" : "jan.labadie"
  }, {
    "password" : "passw0rd",
    "user" : "elbert.moen"
  }, {
    "password" : "passw0rd",
    "user" : "ramiro.lang"
  }]
}