Data Fabric MacGyver Becoming a Data Fabric MacGyver

Section One: Creating our first MDL table

In section one we are going to validate the data and then create our first MDL table which we can configure to intercept logs coming into DDAS as required.

Validating the problem

First part of this lab is to check the data coming into Splunk is actually hitting the DDAS index.

  1. From your Splunk Cloud console. Click the Splunk cloud logo top left-hand side.
  2. Next click on Search & Reporting app on the left-hand menu.
  3. Adjust your timerange picker to last 15 minutes
  4. Run the following search to view our web access logs
index="linux_secure"

See example output below.

image_tag

Now that we have confirmed that. Let's create a MDL raw storage destination to route our data to.

Creating a new Machine Data Lake Dataset

  1. Click the Splunk Cloud logo again on the top left-hand side.
  2. Click on Data Management app on the left-hand menu.
  3. Click the Datasets option on the left-hand menu.
From here we have already been in previous labs that there is a default_mdl_raw dataset already. This was created as part of the lab. But for this lab we want to create a new one to route our linux_secure data too.
  1. Click the blue Create dataset button

Like before there are lots of options here. Select the Machine Data Lake (MDL) option. See example below

image_tag

  1. Click Next when ready.
  2. Similar to creating other datasets, we need to enter important information for our MDL dataset. Fill in the details as required below.

See example filled out MDL dataset definition:

image_tag

We covered off quite a few options there, we don't have time to go through which options we chose for what reason but all the information you need can be found HERE
  1. Click next when ready.

On the next screen you can get granular around who has access to perform actions on the dataset. For simplicity we are just going to allow access to Admin for this lab but you can find more information on setting user permissions on MDL HERE

  1. On the configure permissions screen, click the dropdown and select the sc_admin role.
  2. Next to the role. Click the + button next to sc_admin. See example below
  3. From the drop down select the Streaming promote, static promote, query and view option. See example below

Once selected your screen should look like this:

image_tag

  1. Click Next when ready.

This screen looks just like a ingest processor pipeline wizard. Well it probably is. But anyway. This is where we can select how the MDL will process the data coming in and select which data lands in the machine data lake raw table we are creating.

  1. Click the partition by sourcetype option.
  2. Copy and paste the below into the Value field
linux_secure
  1. Click Apply when ready

Your process_data section should look like this:

image_tag

  1. Click next when ready
  2. Review the summary screen and click Create Dataset when ready.

Once created you should now see your new MDL dataset in the list. See example below:

image_tag

Nice work! Now we can check if data is flowing into it!

Validate Data is flowing into MDL

  1. From your Splunk Cloud console. Click Cog icon near the top right-hand side.
  2. From there select Catalog option. This should be under the section called Knowledge.

This is our new Catalog service inside of Splunk. From here you can see all of the different indexes and machine data lake catalogs. We should be able to see our new linux_secure_mdl RAW table in the list. See example below:

image_tag

  1. Select the new linux_secure_mdl table from the list.

Now we should see a summary of our new MDL table. See example summary below:

image_tag

Clicking view hamburger (three dots) next to search will take you to the section where you can get more information on the configured table and edit it as well. But let's stay focused for now.
  1. Click the search button to open up the search page for the MDL table.

You will see a promotion warning appear. See example below:

image_tag

  1. Click the blue search button on the warning screen to proceed.
Search results may take up to 10 minutes to start appearing.

By default the search will already appear as a SPL2 query with a limit of 10 results. You should see some results from our recently created MDL table. See example below:

image_tag

As we have configured the MDL table to route all data matching linux_secure sourcetype, this will mean that we are only sending data to MDL raw now in our new table.

Let's verify we no longer are seeing linux_secure coming into our DDAS storage.

  1. From a new tab open up the Splunk search bar again. You can do this by clicking the Splunk cloud logo on the top left-hand side and selecting the Search & Reporting app.
  2. Run the following SPL search against our original linux_secure index.
index=linux_secure earliest=-2m
You will notice we used the earliest=-2m command again to quickly check no new data is coming in.

See example of no results in our search now

image_tag

We have now configured the MDL table to take all of the linux secure data and we don't have to worry about it going to DDAS anymore. But what if we wanted to promote a set of that data back to DDAS? Well that is the next part of the lab.

Click next to continue on with lab.