Data Fabric MacGyver Becoming a Data Fabric MacGyver

Section One: Creating our second MDL table

In section one we are going to validate the data and then create our second MDL table which we we will use to route access_combined data into.

Validating the problem

First part of this lab is to check the data coming into Splunk is actually hitting the DDAS index.

  1. From your Splunk Cloud console. Click the Splunk cloud logo top left-hand side.
  2. Next click on Search & Reporting app on the left-hand menu.
  3. Adjust your timerange picker to last 15 minutes
  4. Run the following search to view our web access logs
index="access_combined"

See example output below.

image_tag

Now that we have confirmed that. Let's create a MDL raw storage destination to route our data to.

Creating a new Machine Data Lake Dataset

  1. Click the Splunk Cloud logo again on the top left-hand side.
  2. Click on Data Management app on the left-hand menu.
  3. Click the Datasets option on the left-hand menu.
From here we have can see all of the datasets from previous labs. Let's create our second MDL catalog.
  1. Click the blue Create dataset button

Like before there are lots of options here. Select the Machine Data Lake (MDL) option. See example below

image_tag

  1. Click Next when ready.
  2. Similar to creating other datasets, we need to enter important information for our MDL dataset. Fill in the details as required below.

See example filled out MDL dataset definition:

image_tag

  1. Click next when ready.

Like before we want to add access to our sc_admin user to access the MDL raw table we are creating.

  1. On the configure permissions screen, click the dropdown and select the sc_admin role.
  2. Next to the role. Click the + button next to sc_admin. See example below
  3. From the drop down select the Streaming promote, static promote, query and view option.

Once selected your screen should look like this:

image_tag

  1. Click Next when ready.

Last time we create our MDL catalog we configured a sourcetype of Linux_secure so any data that came in with Linux_secure was automatically routed to the catalog. This time we are doing to send data directly to MDL catalog for access_combined via the HEC token. Because of this we don't need to define it here in the MDL setup.

  1. Leave define data landing empty and Click next.
  2. Review the summary screen and click Create Dataset when ready.

Once created you should now see your new MDL dataset in the list. See example below:

image_tag

Now we need to get data going into it. But before we do that, let's confirm it is indeed bringing in no data at all

Validate Data is NOT flowing into MDL

  1. From your Splunk Cloud console. Click the catalog icon near the top right hand side as shown in previous step. See example below:

image_tag

We should be able to see our new access_combined_mdl RAW table in the list. See example below:

image_tag

  1. Select the new access_combined_mdl table from the list.

Now we should see a summary of our new MDL table. See example summary below:

image_tag

  1. Click the search button to open up the search page for the MDL table.

You will see a promotion warning appear. See example below:

image_tag

  1. Click the blue search button on the warning screen to proceed.

By default the search will already appear as a SPL2 query with a limit of 10 results. You should see No results yet found. See example below:

image_tag

This has just confirmed we don't have any data coming in to the MDL. Currently its a giant container waiting for data.

Let's now route data into it via a different method than what we did before.

Route Data into MDL via HEC

  1. Click the cog icon near the top right-hand side.
  2. Scroll down and select Data inputs under the DATA section.
  3. Select HTTP Event Collect link.
Your lab environment may look different to mine. So keep that in mind with screen shots!

You will see some HEC token here in the list. There should be one for our access_combined data.

  1. Select the access_combined HEC token link as shown below:

image_tag

You will see a few options here on the screen. Here we want to adjust the index where the data is being routed too.

HEC allows you to assign multiple indexes to a single HEC token. This is basically saying which indexes does this HEC token have permission to send data too.
  1. In the section which says available indexes, find and select our new ~.federated_access_combined_mdl index.

This should move it from the available indexes to the Selected indexes section.

  1. Click the access_combined index on the Selected indexes.

This should now do the reverse and move the original access_combined DDAS (tsidx) index off.

You may see the default index section change when you de-select the access_combined index. This is because there is now only one index in the selected indexes so the default changed to that.

See example screen of HEC token configuration changes.

image_tag

  1. Click Save when ready.

Ok, that should now be routing data to the MDL catalog for Access_Combined and no longer routing data to the original access-combined index. Let's test it.

Validating our data routing to MDL catalog and no longer going to access_combined DDAS index

Let's first validate data is no longer going to DDAS.

  1. Go to the Splunk search an reporting app by clicking the Splunk Cloud icon top left-hand side.
  2. Click the search & reporting app on the left-hand menu.
  3. Run the following search below which will show us last 5 minutes of searches on access_combined.
index=access_combined earliest=-5m

We should see now results. See example below:

image_tag

The original data up until we stopped it coming in will still be in the index. Something to keep in mind. You could test this by adjusting that earliest=-5m to something like earliest=-1h

Next let's validate data is going into our new MDL catalog for access_combined.

  1. Select the catalog icon again from the near top right-hand side.
  2. Again, select the access_combined_mdl table
  3. Click the search button again, followed by the search button again on the warning screen.

If all went to plan you should now see data coming into the catalog we created!

The data may take around 100 minutes to appear in the data catalog

See example of data below:

image_tag

Congratulations, that wraps up Lab 5! Now onto our last lab which we look at configuring a Dynamic Data Self Storage archive index using Amazon S3.

Click next to start the next lab on with lab.