Data Fabric MacGyver Becoming a Data Fabric MacGyver

In order to understand the benefit of VPC flow logs we need to configure some. VPC flow logs can be configured against many services. Even against an entire VPC, this provides an ability to be broad or granular. In this first part of lesson two we will set up VPC flow logs against a pre-deployed VPC in our workshop. We will then deploy an EC2 inside that VPC which when accessed will generate logs.

Setup VPC Flow Logs

  1. From your AWS console search for and select VPC

Select VPC Service

  1. This is the VPC dashboard. From here there are many things to see and do but for this lesson we want to explore the VPC. From the left-hand menu, select Your VPCs.
There are many ways to configure VPC flow logs from the AWS Console. Below is the simplest for our workshop
  1. From the VPCs page you will see the VPC created as part of the workshop. If it's not selected, select it now by ticking the tick box for it. See example below

Select Your VPC

  1. This will open up (if not open already) a panel down below showing you key information about your VPC. Let's explore the resource map by clicking the Resource Map tab.

From here you will see the network layout of the VPC. This is a handy tool to check how things are configured for your VPC in a single visual representation. See example below

Select Resource map in the top section

  1. Now it's time to create a VPC Flow log. Select the Flow logs tab. You will note currently there are no flow logs. See example below:

Select Flow logs in the top section

  1. On the right-hand side, click the Create flow log button.

Select Orange Create Flow Log Button

  1. Leaving this tab open we need to go and copy the S3 ARN as we will need it to create a flow log. To do this search for and open in a new tab S3.

Navigate to the S3 Service in the AWS console

  1. Select the logging s3 bucket for the workshop (same one used in lesson one), should be starting with splunk-fs-s3-logs-{account-id}-{region}
  2. Select the properties tab and copy the Amazon Resource Name (ARN) for the bucket. See example below. Note: you can use the two blue boxes to copy it.

Use the Copy button on the Logging Bucket S3 ARN

  1. Now we have copied S3 arn we will use that in the flow log setup page. Go back to the other tab for your flow logs now.
  2. From the Create flow log page we will fill in a few details as outlined below. See example at end.

Create Flow Log Settings with the detail as provided you will need to fill out several sections

  1. Scroll down to where it says Log file format and select parquet
  2. Tick the box for Hive-compatible S3 prefix
  3. Select Every 1 hour (60 minutes). under partition logs by time. See example below for options selected:

Partition by hour setting Flow Logs

  1. Leave all remaining settings and click Create flow log
  2. You should see a green rectangle saying Successfully created flow log for vpc-xxx.

Success Flow Log Created banner is green across the top of the screen

  1. Re-Select Your VPC's (left hand menu) then the Flow logs tab. You should now see a new flow log!
  2. Click the link for the Destination Name (your s3 bucket). This should open up your S3 bucket in a new tab.
  3. Navigate the S3 folders for AWSLogs. You should now (or very soon) notice a new folder in the Apache Hive format. eg aws-account-id={account-id}. If you select into the folder you will note we have NO VPC Flow logs so far.

So let's create an EC2 instance now and make sure it's in that VPC, so we can generate some VPC Flow logs!

Creating an EC2 Instance

If you are part of an instructor led lab we will cover off some more details about Apache Hive format which uses key=value pair folder structure. This method is generally the easiest way to optimise your searches. More on that later. AWS has some good documentation on this here.
  1. From your AWS console, search for and select ec2.

Select EC2 Service from the AWS console

  1. You will note there are no instances currently in your environment. Click the Launch instance button in the middle left-hand side of the screen.
This next section is broken down into multiple steps for creating an ec2. Pay close attention to the steps and screenshots.
  1. EC2 basic details. Fill in the below. See screenshot with example.

Enter the EC2 Service Name and Machine Type into the fields

  1. Time to create an EC2 key pair and select it. From the Key pair section. Do the following:
We most likely won't use the keypair as VPC Flow logs will generate without it but, select the correct format for you in case we want to login!

Create EC2 Key Pair with RSA and .ppk

Key pair cont... Make sure your new keypair is now selected in the drop-down.

Select the new EC2 Key Pair that you have just created

When you create the key pair you should have seen this download to your computer. Make sure you pay attention to where it went, you may need it later!
  1. You can leave all remaining defaults and click the Launch Instance button to the right hand side.

EC2 Summary of the settings you have just configured

  1. If all things went well you should see a green box saying, Success.

EC2 Instance Hyperlink is shown in a green banner

Verifying VPC Flow Logs

It can take up to 10 minutes for the first set of VPC flow logs to arrive in an S3 bucket. Time to stretch your legs! Or if your like me, go back to the S3 tab you had opened before (if not from the steps above go back to our S3 folder where the VPC flow logs should be). Press the refresh button till you see some logs. It should look like the image below when done.

VPC Flow Logs Folder In S3 under Object and vpcflowlogs/ folder
  1. Now we see the logs, let's look at the format of a VPC Flow Log. Select the first flow log from the list using the tick box and click the download button. See example below.

Select the VPC from the AWS console and navigate to the where the files are located at the smallest folder

As we have chosen the parquet format for this lab you will not be able to view this file as you would a normal text based file. Parquet is a compress binary columnar format. This means we will have to open it up in something that can help us read it. There are many ways to do this but we have chosen as easy one for you.
  1. Go to the following website here: parquetreader.com and drag your parquet downloaded file into upload section to open it up. See example below

A Text editor showing an example VPC Flow Log that has been created as part of the workshop setup

Have a look at the VPC Flow log format. It has headings explaining each column type with key items such as account ID, interface ID etc.

This wraps up the first section of Lesson Two. Keep that file VPC flow log file handy as we will look at it in a little more detail in the next section.

Congratulations! That ends the first section of lesson two; configuring and exploring VPC Flow Logs. You should now have deployed your first VPC Flow log and see it land in your S3 bucket. Let's now more onto section two; Amazon Athena and Data Definition Language (DDL).

Click next to continue.