TechGirlKB

Performance | Scalability | WordPress | Linux | Insights

Posts

NPM: No user exists for uid 1000

I thought I would write a quick post about this issue, as I’ve encountered it several times. Note, this user could be any ID, not just 1000 or 1001, etc. — it all depends on what user has launched your build container from your deployment software.

The issue: When performing a build step with npm in a Docker container, it throws this error on git checkouts:

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://[email protected]/repo.git
npm ERR! 
npm ERR! No user exists for uid 1000
npm ERR! fatal: Could not read from remote repository.
npm ERR! 
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! 
npm ERR! exited with error code: 128
 
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-05-08T19_50_34_229Z-debug.log

What was equally frustrating was that in testing the same command in the same container locally (instead of inside our deployment tools), it had no issues.

What causes this issue?

The crux of the issue is this: When NPM/node is trying to checkout from git, it uses the permissions of the node_modules or package.json configurations to determine which user should be used to pull git packages.

When you’re mounting the Docker container to your build/deploy tool, the user owning the files there might not exist in your container. And it also might not be the user that you want to be checking out the files either! By default, like it or not, Docker logs you into the container as “root” user.

So to summarize:

  • Docker logs you in as root to perform the npm build commands
  • The files you’ve mounted into the container might be owned by a user that only exists on the deployment server and not inside the Docker container
  • NPM defaults to use the owner of the node_modules files to choose which user it should use to perform git/ssh commands
  • This results in the error that the user does not exist

The fix

The fix in this case is just to perform a “chown” of the files you’ve mounted from the deployment server prior to running your npm build commands.

For example, given the scenario that I’ve mounted my files to /source on the container, and my build files are now inside /source/frontend:

$ chown -R root:root /source/frontend/; cd /source/frontend && npm run build

You can replace the path and the npm command with whatever your npm build script is for your own environment. The important part is the change of the ownership at the beginning of the command.

Have you had issues with this error in NPM? Have experiences you want to share? Feel free to leave a comment, or contact me.

Automate Patching Using AWS Systems Manager (SSM)

AWS offers a plethora of useful tools, but as a DevOps Engineer, the Systems Manager has been a godsend. Systems Manager works by installing the SSM Agent on the instances you wish to manage. Through this agent and using a set of IAM capabilities, the agent can perform management tasks on your server inventory.

I love Systems Manager because it removes a lot of overhead for me in my daily work. Here’s an example: Recently we upgraded the operating system on our servers, and noticed that as a result, the timing mechanisms on our servers were off. There was a relatively quick fix, but it had to be deployed across all our (100+) servers, which would have taken hours. Instead, I wrote one script Document in Systems Manager, and (after safely testing it on one or two dev servers first), deployed the change to all servers in one fell swoop.

One of the great things about Systems Manager is that you can create your own Documents (either Automation documents, or RunCommand documents) to execute your own scripts. Or, to trigger existing scripts within your own script. Using this capability, we were able to create a script to schedule maintenance windows, perform patching, and notify important groups along the way. I thought I would share what we learned to help anyone else going through this process.

About SSM Documents

Documents is the term AWS uses to describe scripts, written in either JSON or YAML, which can be executed against your server inventory. Some are short and simple. Others are more complex, with hundreds of lines of code to be executed. Documents typically fall into one of two groups: RunCommand, or Automation. The difference lies mainly in the formatting. Automation Documents are typically workflows with several steps triggering other Documents to perform routine tasks. For example, one Automation we use assigns IAM roles that allow the server to write to CloudWatch logs (an Association). RunCommand Documents, by contrast, are mainly scripts or commands to run.

AWS has a number of Documents pre-made for you to use, namely, AWS-RunPatchBaseline which we will use later on. You can create your own documents as well, in JSON or YAML formats. I highly recommend using the pre-made documents as an example for syntax.

To run a RunCommand Document, you can trigger it to run from the RunCommand section. If your Document is an Automation Document, you can trigger it from the Automation section. You can register either Automation or RunCommand Documents within Maintenance Windows to run at a scheduled time if desired.

About AWS-RunPatchBaseline

The AWS-RunPatchBaseline Document is an especially useful document. This document uses the baseline for patching you have selected for your servers (under the Patch Manager section). For example, if you use CentOS Linux servers, you can use the pre-defined CentOS patch baseline to receive CentOS patches.

You can organize your servers running the SSM agent into various patching groups, to identify which servers should be patched together. In our case, the option to Create Patch Groups wasn’t functioning properly. But to manually configure the Patch Groups, you can add Tags to the server instances you wish to patch. The Key should be “Patch Group” and the Value can be whatever name you wish to give the patch group. For example, we originally divided patch groups into Develop, Staging, Production. But you can choose whichever groups make sense for your organization.

Automating Patching

Running the AWS-RunPatchBaseline command during your designated maintenance windows, on lower-level testing environments like Develop and Staging is one thing. Patching production is another thing entirely. The AWS-RunPatchBaseline command installs patches and updates to packages, but in order for the device to actually register the changes and show “In Compliance” (on the Compliance page), a Reboot has to take place.

While rebooting the server usually takes a minute or less time to complete, it can be disruptive to end users and support users. With that in mind, we should consider scheduling the patching with proper maintenance windows in our status pages, and in monitoring tools (Pingdom, etc.) since we expect downtime. Notifying Slack as patching is scheduled, beginning, and ending is a good idea as well.

While the solution below is certainly not the most refined and is pretty basic bash scripting, it does the trick for many patching needs. I’ll be sure to include “gotchas” we encountered as I go as well!

Getting started

First, in the Documents section you will need to select “Create command or session” to start a new RunCommand document. Select a name, and the type of target (for most, this will be AWS::EC2::Instance). I’ll be showing how to format your script in JSON, but YAML is ok too.

To start, there are a few points of order to get out of the way at the beginning of your script. This includes things like defining any additional parameters for your script and its environment.

{
   "schemaVersion": "2.2",
   "description": "Command Document Example JSON Template",
   "mainSteps": [
     {
       "action": "aws:runShellScript",
       "name": "ApplyMaintenanceWindow",
       "inputs": {
         "runCommand": [

This part defines the schema version to be used (as of this writing, 2.2 is the latest). You can enter a description of this command if desired. After “description” you will need to include any additional parameters that should be selected when running the command, if any. In an earlier version of our Document, I had “appName” and “appEnv” parameters added, but eventually took these out, as these could be determined dynamically from our tagging structure.

If you choose to add Parameters, here is an example of how you can define appName and appEnv:

{
   "schemaVersion": "2.2",
   "description": "Command Document Example JSON Template",
   "parameters": {
     "appName": {
       "type": "String",
       "description": "(Required) Specify the app name",
       "allowedValues": [
         "App1",
         "App2",
         "App3"
       ]
     },
     "appEnv": {
       "type": "String",
       "description": "(Required) Specify the app environment",
       "default": "Develop",
       "allowedValues": [
         "Develop",
         "Staging",
         "Production"
       ]
     }
   },
   "mainSteps": [...

Gather tags

If you gather tags from your EC2 instances in other methods (such as writing them to a file for parsing) you can use this as a way to determine which instance you are running the command on as well. You can either do this entirely separate from patching (such as upon server build, or deploy of code). Or, you can do this at runtime as well. On AWS, you can use cURL to gather instance metadata, and use this to your benefit:

#!/usr/bin/env bash
# Requires aws cli to be installed and proper ec2 roles to read ec2 tags
# Requires sudo permissions to right to /etc/
INSTANCE_ID=`curl http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null`
REGION=`curl http://169.254.169.254/latest/meta-data/placement/availability-zone 2>/dev/null | sed 's/.$//'`
aws ec2 describe-tags --region $REGION --filter "Name=resource-id,Values=$INSTANCE_ID" --output=text | sed -r 's/TAGS\t(.)\t.\t.\t(.)/\1="\2"/' > /etc/ec2-tags

This script will write the tags to the /etc/ec2-tags path on your server, which can be read later.

Back in our Document, we can use that /etc/ec2-tags output (or you can use it within your Document as well if you like).

"mainSteps": [
     {
       "action": "aws:runShellScript",
       "name": "ApplyMaintenanceWindow",
       "inputs": {
         "runCommand": [
           "#!/bin/bash",
           "export APPENV=$(grep \"Environment\" /etc/ec2-tags | cut -d'\"' -f2)",
           "export APPNAME=$(grep \"AppName\" /etc/ec2-tags | cut -d'\"' -f2)",
           "export INSTANCE_ID=$(grep \"Name\" /etc/ec2-tags | cut -d'\"' -f2)",
...

Here we are defining the value of some variables that we will be using later in the script, to notify our maintenance windows/monitoring tools: the app environment, app name, and the instance ID (given by AWS).

Scheduling time frames

Now that we have identified which applications and environments are being patched, we need to identify timing. We should define what time the patching will begin, and end, for our notification and monitoring tools.

In my example, I want to schedule and notify for the maintenance 24 hours in advance. The tools I’ll be using are Pingdom and StatusPage, but as long as your monitoring tool has an API this is fairly easy to automate as well!

First you have to determine the time format your API accepts. For Pingdom, they accept UNIX timestamps (which we can get with a simple “date +%s” command). StatusPage.io by contrast uses a different and more unique format, so we have it formatted it as it expects to be read.

Quick Tip: We found that no matter the timezone specified in the StatusPage.io dashboard, it always interpreted the timezone as UTC. So to match the correct time (PST), we had to add 8 hours (or 7 hours during daylight savings time). 

     "export STATUSPAGE_TIME_START=$(date +%C%y-%m-%dT%TZ -d +1day+8hours)",       
     "export STATUSPAGE_TIME_END=$(date +%C%y-%m-%dT%TZ -d +1day+8hours+15minutes)",       
     "export PINGDOM_TIME_START=$(date +%s -d +1day)", 
     "export PINGDOM_TIME_END=$(date +%s -d +1day+15minutes)",
...

Set the StatusPage and Pingdom IDs

Now that we have our time frames settled, we also need to tell our script what ID to use for Pingdom, and what ID to use for StatusPage.io. These values are required when making API calls to these services, and will likely be different based on the application name and environment. To do this we can set variables based on our APPNAME and APPENV vars we set earlier:

      "if [[ $APPNAME == \"APP1\" && $APPENV == \"Production\" ]]; then",
      "  export PINGDOM_ID=\"123456\"",
      "  export STATUSPAGE_ID=\"abc123def456\"",
      "elif [[ $APPNAME == \"APP1\" && $APPENV == \"Demo\" ]]; then",
      "  export PINGDOM_ID=\"NONE\"",
      "  export STATUSPAGE_ID=\"456def123abc\"",
      "elif [[ $APPNAME == \"App2\" && $APPENV == \"Production\" ]]; then",
      "  export PINGDOM_ID=\"789012,345678\"",
      "  export STATUSPAGE_ID=\"ghi456jkl789\"",
      "elif [[ $APPNAME == \"APP2\" && $APPENV == \"Demo\" ]]; then",
      "  export PINGDOM_ID=\"012345\"",
      "  export STATUSPAGE_ID=\"jkl678mno901\"",
      "else",
      "  export PINGDOM_ID=\"NONE\"",
      "  export STATUSPAGE_ID=\"NONE\"",
      "fi",
...

Some things to note about the above block:

  • We’re setting the values to “NONE” if there isn’t a Pingdom or StatusPage ID for the specific app and environment – this is also the default value if the app name and environment name don’t match the combinations presented, in our “else” clause.
  • If there are multiple Pingdom checks that should be paused during this patching window, you can enter the two IDs separated by a comma (no space) – see “App2 Production” for an example.

Making the API calls

Now that we have all the proper context we can make the API calls. There will be a few sets of API calls:

  1. First we will notify Slack/Microsoft Teams that patching will begin in 24hrs, and schedule the maintenance windows in Pingdom and Statuspage.
  2. Next we will give an hour warning to Slack/Microsoft Teams that the patching will begin soon.
  3. At the 24 hour mark we will send a notice to Slack/Microsoft Teams that the patching is beginning – Pingdom and Statuspage windows will automatically begin.
  4. We will run the AWS-RunPatchBaseline command, which involves a reboot of the system.
  5. After patching completes, we will notify Slack/Microsoft Teams that the patching has completed, and list the packages updated.

Patching Scheduled Notifications

Now for the fun part: making the API calls. You will need to replace the OAuth (StatusPage), Bearer (Pingdom), and webhook URL (Microsoft Teams) based on the API tokens and webhooks for your own environments.

      "# Add StatusPage notification of scheduled maintenance",
      "if [[ $STATUSPAGE_ID != \"NONE\" ]]; then",
      "  curl -X POST 'https://api.statuspage.io/v1/pages/********/incidents' -H 'Authorization:OAuth ********-****-****-****-********' -d \"incident[name]=Scheduled Maintenance for $APPNAME - $APPENV\" -d \"incident[status]=scheduled\" -d \"incident[impact_override]=maintenance\" -d \"incident[scheduled_for]=$STATUSPAGE_TIME_START\" -d \"incident[scheduled_until]=$STATUSPAGE_TIME_END\" -d \"incident[scheduled_auto_in_progress]=true\" -d \"incident[scheduled_auto_completed]=true\"-d \"incident[components][component_id]=under_maintenance\" -d \"incident[component_ids]=$STATUSPAGE_ID\"",
      "fi",
      "",
      "# Add Pingdom Maintenance window",
      "if [[ $PINGDOM_ID != \"NONE\" ]]; then",
      "  curl -X POST 'https://api.pingdom.com/api/3.1/maintenance' -H 'Content-Type: application/json' -H 'Authorization:Bearer ******_******************_*****************' -d '{\"description\": \"Maintenance for '$APPNAME' - '$APPENV'\", \"from\": '$PINGDOM_TIME_START', \"to\": '$PINGDOM_TIME_END', \"uptimeids\": \"'$PINGDOM_ID'\" }'",
      "fi",
      "",
      "# Notify Teams room of upcoming maintenance/patching",
      "curl -X POST 'https://outlook.office.com/webhook/*******-***-****-****-***********-****-****-****-************/IncomingWebhook/***********************/********-****-****-****-**********' -H 'Content-Type: application/json' -d '{\"@type\": \"MessageCard\", \"@context\": \"http://schema.org/extensions\", \"themeColor\": \"ECB22E\", \"summary\": \"AWS Patching Window Notifications\", \"sections\": [{\"activityTitle\": \"Patching Scheduled\", \"activitySubtitle\": \"Automated patching scheduled to begin in 24 hours\", \"facts\": [{\"name\": \"App Name\", \"value\": \"'$APPNAME' - '$APPENV'\"}, {\"name\": \"Instance Name\", \"value\": \"'$INSTANCE_ID'\"}], \"markdown\": true}]}'",
      ""
...

Note that the last API call (Microsoft Teams) can be replaced with a Slack one if needed as well:

      "# Notify Slack room of upcoming maintenance/patching",
      "curl -X POST 'https://hooks.slack.com/services/********/********/****************' -H 'Content-type: application/json' -d '{\"attachments\": [{ \"mrkdwn_in\": [\"text\"], \"title\": \"Patching Scheduled\", \"text\": \"Automated patching scheduled to begin in 24 hours\", \"color\": \"#ECB22E\", \"fields\": [{ \"title\": \"App Name\", \"value\": \"'$APPNAME' - '$APPENV'\", \"short\": \"false\"}, { \"title\": \"Instance Name\", \"value\": \"'$INSTANCE_ID'\", \"short\": \"false\"}] }] }'",
      ""
...

After sending our API calls we’ll want to sleep for 23 hours until it’s time for our next notification (1 hour warning):

{
  "name": "Sleep23Hours",
  "action": "aws:runShellScript",
  "inputs": {
    "timeoutSeconds": "82810",
    "runCommand": [
      "#!/bin/bash",
      "sleep 23h"
    ]
  }
},
...

Quick Tip: We noticed that the longest “timeoutSeconds” will allow is 24 hours (86400 seconds). If you set it higher than that value, it defaults to 10 minutes (600 seconds).

One-hour Notifications

  "name": "NotifySlack1Hour",
  "action": "aws:runShellScript",
  "inputs": {
    "runCommand": [
      "#!/bin/bash",
      "export APPENV=$(grep \"Environment\" /etc/ec2-tags | cut -d'\"' -f2)",
      "export APPNAME=$(grep \"AppName\" /etc/ec2-tags | cut -d'\"' -f2)"
      "export INSTANCE_ID=$(grep \"Name\" /etc/ec2-tags | cut -d'\"' -f2)",
      "# Notify room of upcoming maintenance/patching",
      "curl -X POST 'https://hooks.slack.com/services/********/********/****************' -H 'Content-type: application/json' -d '{\"attachments\": [{ \"mrkdwn_in\": [\"text\"], \"title\": \"Upcoming Patching\", \"text\": \"Automated patching scheduled to begin in 1 hour\", \"color\": \"#ECB22E\", \"fields\": [{ \"title\": \"App Name\", \"value\": \"'$APPNAME' - '$APPENV'\", \"short\": \"false\"}, { \"title\": \"Instance Name\", \"value\": \"'$INSTANCE_ID'\", \"short\": \"false\"}] }] }'"
    ]
  }
},

Or for Microsoft Teams:

      "# Notify Teams room of upcoming maintenance/patching",
      "curl -X POST 'https://outlook.office.com/webhook/********-****-****-****-****************-****-****-****-************/IncomingWebhook/********************/******************' -H 'Content-Type: application/json' -d '{\"@type\": \"MessageCard\", \"@context\": \"http://schema.org/extensions\", \"themeColor\": \"ECB22E\", \"summary\": \"AWS Patching Window Notifications\", \"sections\": [{\"activityTitle\": \"Upcoming Patching\", \"activitySubtitle\": \"Automated patching scheduled to begin in 1 hour\", \"facts\": [{\"name\": \"App Name\", \"value\": \"'$APPNAME' - '$APPENV'\"}, {\"name\": \"Instance Name\", \"value\": \"'$INSTANCE_ID'\"}], \"markdown\": true}]}'",
      ""
    ]
  }
},

Now we sleep for one more hour before patching:

{
  "name": "SleepAgainUntilMaintenance",
  "action": "aws:runShellScript",
  "inputs": {
    "timeoutSeconds": "3610",
    "runCommand": [
      "#!/bin/bash",
      "sleep 1h"
    ]
  }
},

Patching Beginning

{
  "name": "NotifySlackMaintenanceBeginning",
  "action": "aws:runShellScript",
  "inputs": {
    "runCommand": [
      "#!/bin/bash",
      "export APPENV=$(grep \"Environment\" /etc/ec2-tags | cut -d'\"' -f2)",
      "export APPNAME=$(grep \"AppName\" /etc/ec2-tags | cut -d'\"' -f2)",
      "export INSTANCE_ID=$(grep \"Name\" /etc/ec2-tags | cut -d'\"' -f2)",
      "# Notify Teams room of upcoming maintenance/patching",
      "curl -X POST 'https://outlook.office.com/webhook/********-****-****-****-****************-****-****-****-************/IncomingWebhook/********************/******************' -H 'Content-Type: application/json' -d '{\"@type\": \"MessageCard\", \"@context\": \"http://schema.org/extensions\", \"themeColor\": \"ECB22E\", \"summary\": \"AWS Patching Window Notifications\", \"sections\": [{\"activityTitle\": \"Patching Beginning\", \"activitySubtitle\": \"Automated patching has started\", \"facts\": [{\"name\": \"App Name\", \"value\": \"'$APPNAME' - '$APPENV'\"}, {\"name\": \"Instance Name\", \"value\": \"'$INSTANCE_ID'\"}], \"markdown\": true}]}'",
      ""
    ]
  }
},

Or for Slack:

 "# Notify Slack room of upcoming maintenance/patching",
      "curl -X POST 'https://hooks.slack.com/services/********/********/****************' -H 'Content-type: application/json' -d '{\"attachments\": [{ \"mrkdwn_in\": [\"text\"], \"title\": \"Patching Beginning\", \"text\": \"Automated patching has started\", \"color\": \"#ECB22E\", \"fields\": [{ \"title\": \"App Name\", \"value\": \"'$APPNAME' - '$APPENV'\", \"short\": \"false\"}, { \"title\": \"Instance Name\", \"value\": \"'$INSTANCE_ID'\", \"short\": \"false\"}] }] }'"

Followed by the actual patching task, calling upon the existing AWS-RunPatchBaseline document:

{
  "name": "installMissingUpdates",
  "action": "aws:runDocument",
  "maxAttempts": 1,
  "onFailure": "Continue",
  "inputs": {
    "documentType": "SSMDocument",
    "documentPath": "AWS-RunPatchBaseline",
    "documentParameters": {
      "Operation": "Install",
      "RebootOption": "RebootIfNeeded"
    }
  }
},

Post-Patching Notification

Now that the patching has occurred we can send some important data to our chat software (Slack or Microsoft Teams) detailing what was updated:

{
       "action": "aws:runShellScript",
       "name": "NotifyPatchingComplete",
       "inputs": {
         "runCommand": [
           "#!/bin/bash",
           "export APPENV=$(grep \"Environment\" /etc/ec2-tags | cut -d'\"' -f2)",
           "export APPNAME=$(grep \"AppName\" /etc/ec2-tags | cut -d'\"' -f2)",
           "export INSTANCE_ID=$(grep \"Name\" /etc/ec2-tags | cut -d'\"' -f2)",
           "export INSTALL_DATE=$(date \"+%d %b %C%y %I\")",
           "export PACKAGES=\"$(rpm -qa --last | grep \"${INSTALL_DATE}\" | cut -d' ' -f1)\"",
           "",
           "# Notify Slack room that patching is complete",
           "curl -X POST 'https://hooks.slack.com/services/********/********/****************' -H 'Content-type: application/json' -d '{\"attachments\": [{ \"mrkdwn_in\": [\"text\"], \"title\": \"Patching Complete\", \"text\": \"Automated patching is now complete\", \"color\": \"#2EB67D\", \"fields\": [{ \"title\": \"App Name\", \"value\": \"'$APPNAME' - '$APPENV'\", \"short\": \"false\"}, { \"title\": \"Instance Name\", \"value\": \"'$INSTANCE_ID'\", \"short\": \"false\"}, { \"title\": \"Packages Updated\", \"value\": \"'\"${PACKAGES}\"'\", \"short\": \"false\" }] }] }'"
         ]
       }
     }
   ]
 }

Or for Teams:

      "# Notify Teams room that patching is complete",
      "curl -X POST 'https://outlook.office.com/webhook/********-****-****-****-****************-****-****-****-************/IncomingWebhook/********************/******************' -H 'Content-Type: application/json' -d '{\"@type\": \"MessageCard\", \"@context\": \"http://schema.org/extensions\", \"themeColor\": \"2EB67D\", \"summary\": \"AWS Patching Window Notifications\", \"sections\": [{\"activityTitle\": \"Patching Complete\", \"activitySubtitle\": \"Automated patching is now complete\", \"facts\": [{\"name\": \"App Name\", \"value\": \"'$APPNAME' - '$APPENV'\"}, {\"name\": \"Instance Name\", \"value\": \"'$INSTANCE_ID'\"}, {\"name\": \"Packages Updated\", \"value\": \"'\"${PACKAGES}\"'\"}], \"markdown\": true}]}'",
      ""

Note how we’re getting the packages updated from our rpm -qa --last command, and using grep to find the packages updated in the last hour – this generates the list of $PACKAGES that our command receives and sends to Slack/Teams.

Quick tip: Because the list of packages is separated by a new line, we have to wrap it in single and double quotes (e.g. ‘”${PACKAGES}”‘) so that it’s interpreted properly.

Maintenance Windows

Now that we’ve completed our script, we’ll want to ensure it runs on a specified patching schedule. For my team, we patch different environments different weeks, and build patching windows based upon that timeframe. In AWS Systems Manager, you can use several building blocks to ensure the right groups get patched together, at the right times:

  • Resource Groups: Use these to group like-instances and resources together. For example, I use resource groups to group instances with the same tags (e.g. Environment=Develop and AppName=App1).
  • Target Groups: When you create a Maintenance Window, you can create Target Groups based on Resource Groups, Tags, or specifying literal Instance IDs. I recommend Resource Groups, as (at least for me) the Tags have been hit or miss in identifying the right resources.
  • Registered Tasks: The last step in creating a Maintenance Window is to register tasks to that window. In my case I registered tasks for each Target Group I created, because the system wouldn’t let me tag more than a few Target Groups in one task registration.

Some gotchas about your maintenance windows:

  • Only 5 maintenance windows can be executing at once. This was a problem for us since we had originally set a maintenance window for each app/env, and it amounted to far more than 5. Since our script is 24hrs long, that’s a fair amount of time to be restricted.
  • The cron/rate expressions are a little different. It’s not quite the same as a standard cron expression you’d use in crontab. For example, to run my window at 12:30am on the 4th Friday of every month, I used the following expression: cron(0 30 0 ? * FRI#4 *) – standing for second, minute, hour, day-of-month, month, day-of-week, year. So if I wanted 12:30:25am on the 1st and 15th of August, I’d use cron(25 30 0 1,15 8 * *).
  • The cron rates for Maintenance Windows and Associations are different. What is accepted for Associations vs Maintenance windows in terms of cron rate are defined in the AWS docs here: https://docs.aws.amazon.com/systems-manager/latest/userguide/reference-cron-and-rate-expressions.html

Conclusion

This script was a bit tricky to pull together with all the right elements and gotchas, especially since some are not documented much (if at all) in the AWS docs thus far. However, the end results have been absolutely worth the trouble! Systems Manager is a tool that AWS has been putting a lot of work into of late, and the improvements are wonderful — they’re just not fully documented quite yet. I hope my learning helps you in your own journey to automated patching! This has made our patching and compliance cycles painless, visible to the right parties, and auditable. Have any tips, tricks, or experiences you’d like to share? Leave a comment, or contact me.

Adding Nginx HSTS Headers on AWS Load Balancer

On AWS, if you use a load balancer (ELB or ALB) you may wonder how to properly add security headers. In our situation we already redirect all HTTP requests to HTTPS. However, our security organization wanted to explicitly see HSTS headers. In this article I will explain how to do this when using AWS load balancers in your ecosystem.

What is HSTS?

HSTS stands for HTTP Strict-Transport-Security. It’s a type of header that can be added to instruct browsers that your website should only be accessed over HTTPS going forward. Personally, I thought it might be possible to add these headers at the load balancer level. While AWS did add some new options for ALBs (like perform action based on cookie presence), setting a header was not one of them.

I think this StackOverflow answer explains the reasoning well. Essentially, the header needs to be set at the source: your web server. You also shouldn’t set this header for HTTP requests (as it won’t do anything), so you will need to ensure that your ALB is communicating with your EC2 instance via HTTPS (port 443).

Configuring HSTS

HSTS is a header that can be configured in your web server configuration files. In our case, we’ll be setting it up with our web server, Nginx. You’ll need to use the “add_header” directive in the ssl/443 block of the web server config (for me, /etc/nginx/sites-available/sitename.conf — though depending on your setup, the directory structure may be different). 

## The nginx config
server {
  listen 80;
  server_name localhost;
  return 301 https://$host$request_uri;
}
server {
    listen   443 ssl;
    server_name  localhost;

# insert HSTS headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

Once you’ve added your header, close the file and do nginx -t to test the config for any errors. If all checks out, do service nginx restart or systemctl nginx restart to apply the change.

About HSTS options

You’ll see in the section above, the Strict-Transport-Security header has a few options or flags appended. Let’s dive into what they mean:

  • max-age=31536000: This directive tells the browser how long to hold onto this setting, in seconds. 31536000 seconds is 365 days, or one year. That means your browser will remember to only load the website over HTTPS for a year from the first time you accessed it. The minimum max-age is 18 weeks (10886400 seconds), though .
  • includeSubDomains: This setting tells the browser to remember the settings for both the root domain (e.g. https://yoursite.com) and for subdomains (e.g. https://www.yoursite.com).
  • preload: This setting is used when your website is registered as “preload” with Chrome at https://hstspreload.org/. This tells all Chrome users to only access your website over HTTPS.

Verify HSTS

If HSTS is working properly, you’ll see it when curling for headers. Here’s what it should look like:

$ curl -ILk https://yourdomain.com
HTTP/2 200
date: Thu, 16 Jan 2020 19:25:15 GMT
content-type: text/html; charset=UTF-8
server: nginx
x-frame-options: Deny
last-modified: Thu, 16 Jan 2020 19:25:15 GMT
cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
strict-transport-security: max-age=31536000; includeSubDomains; preload

If you prefer, you can also use the Inspect Element tools in your browser when you visit your domain. Open the Inspect Element console by right-clicking in your browser window, then click Network. Navigate to your website, and click the entry for your domain in the dropdown when it loads. You should see headers, including one for “strict-transport-security” in the list.

If you’re not seeing the header, check these things:

  • Did you restart nginx? If yes, did it show an error? If it did show an error, check the config file for any obvious syntax errors.
  • Did you make the change on the right server? I made this mistake at first too, which took me down a rabbit hole with no answers trying to find why AWS was stripping my HSTS headers. They weren’t, I was just making a very preventable mistake.
  • Is your ALB/ELB sending traffic to your EC2 instance over HTTPS/port 443? HSTS headers can’t be sent over HTTP/port 80, so you need to make sure that the load balancer is communicating over HTTPS.

Have you had issues enabling HSTS headers on AWS? Have experiences you want to share? Feel free to leave a comment, or contact me.

SSH: You Don’t Exist!

If you’ve ever been told you don’t exist by a software package, you…

  • Might be a DevOps Engineer
  • Might also start questioning your life decisions

Over the holidays we have been in the process of adding a new git repository as an NPM package to our build step for a number of projects. This can be problematic in a pipeline, since for git you also have to manage things like SSH keys, and in a pipeline most often the environment may be obscured to where you can’t really add them.

This situation caused a number of errors in our build step that we (incorrectly) assumed were caused by a bad underlying server. Turns out… we just played ourselves.

The authenticity of host ‘(your git host)’ can’t be established.

The first of this comedy of errors came from this wonderful prompt. You’ve probably seen it before, the first time you connect to a new git host. Usually you can just say “Yes, git overlords, I accept your almighty fingerprint” and we all move on with our lives. But in a container in our build step, it’s not an interactive shell. So instead, it just hangs there forever until someone wonders why that deploy never happened, and checks on it.

After only 94 deploy attempts in an effort to figure this out, we finally realized two things:

  • The npm install was taking place in a cached build step (that our deploy system conveniently placed at the very bottom of the configuration page instead of, you know, before the build steps).
  • All our attempts to fix the issue were being placed in the actual build step which takes place after the npm install and were therefore fruitless.

Anyways, once we figured that simple piece of wisdom out, we were able to resolve it by adding this line before the npm install:

mkdir -p -m 0600 ~/.ssh && ssh-keyscan <your git host> > ~/.ssh/known_hosts

Could not create leading directories

The next error we encountered was something that was probably changed on the underlying server but we can’t be certain. All of a sudden, public git packages started giving an error because git couldn’t write to a cache/tmp directory — the intermediary directories didn’t exist first. 

[?25hnpm ERR! code 128
npm ERR! Command failed: git clone --depth=1 -q -b v1.4.1 git://github.com/hubspot/messenger.git /root/.npm/_cacache/tmp/git-clone-2e2bbd46
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-2e2bbd46': Permission denied

The issue in this case was that the user wasn’t able to create this new directory for the git clone, because the parent directory(ies) didn’t exist, or the user didn’t have permissions to write to them. As this wasn’t an issue before, we believe the issue was that the directory permissions for executables on the underlying server. Ultimately what fixed it was changing the prefix for npm to somewhere that both exists and is writeable:

npm config set prefix /usr/local

You don’t exist, go away!

And finally, this supremely unhelpful error. In doing some research, this is actually an SSH error. It occurs when you’re trying to SSH as a user ID that doesn’t exist. So like, I guess it makes sense in that exact situation. But, our user is “root” and it definitely exists. If it didn’t, this whole environment would probably collapse in on itself. 

[?25hnpm ERR! Error while executing:output npm ERR! /usr/bin/git ls-remote -h -t <your git host>
npm ERR!
npm ERR! You don't exist, go away!
npm ERR! fatal: The remote end hung up unexpectedly
npm ERR!
npm ERR! exited with error code: 128

This error presented itself when trying to install a private git repository as a NPM package for the first time (for this particular app and container).

After about 59 tries to figure out what exactly was wrong with the user, container, and anything else in the environment, we finally noticed something different in this project’s package.json file — it was doing the npm install with the “global” -g flag. Thinking back to the last issue, I decided to try to change the prefix (which I had already tried, and it didn’t help), but this time with the -g flag as well. 

npm config set -g prefix /usr/local

Like magic, it worked.

Conclusion

Build steps can be a frustrating troubleshooting environment. When you don’t have access to the server itself, it can be cumbersome and loud to try to find the cause of errors. And, those errors don’t always present themselves in the same way. Most of these errors did not occur when testing from the same container on local. And, many of these errors produced little to no results in doing a google search. I hope this article helps some weary DevOps souls out there! Feel free to comment with other weird build step issues you’ve encountered as well, or contact me.

How to Use AWS SSM Parameter Store with Ansible

If you’re like many frustrated Dev/Ops teams out there, you may be tired of using Ansible’s vault function to encrypt secrets, and looking for other options. AWS offers a couple options for storing secrets:

  • The more obviously-named Secrets Manager tool
  • The less-obvious Systems Manager Parameter Store

As it turns out, Ansible has lookup plugins for both AWS tools. In this guide we’ll explore both, and why (spoiler alert!) our team decided to go with Systems Manager Parameter Store in the end.

AWS Secrets Manager vs. Parameter Store

There are a few key differences between the different secure variable storage systems.

Secrets Manager

Firstly, Secrets Manager has different secret types, mostly geared toward storing, encrypting, (and regularly rotating) database credentials. Here’s a quick look at the options.

The options shown in the “Store a new secret” pane are:

  • Credentials for RDS database
  • Credentials for Redshift cluster
  • Credentials for DocumentDB database
  • Credentials for other database
  • Other type of secrets (e.g. API key)

As you can see, most of these options are specific to database credentials. There is, however, an option to store another secret in “Key” and “Value” format. This is the option our team was planning to use for most secure variables.

In this pane, you can add a simple key-value pair. On the next screen you can add an identifying name and any tags you wish to the key, followed by a pane where you can select automatic rotation for the key if you choose.

There’s a lot to like about Secrets Manager, in particular the key rotation — if you haven’t been obscuring your secure variables in your repos in the past, it allows for easy, hands-off rotation of these keys on a regular basis. This reduces risk in case of employee turnover or security breaches. And with encryption via KMS, you can limit access to whatever IAM users and roles actually need read/write access.

Secrets Manager stores secrets for $0.40/secret per month, and $0.05 per 10,000 API calls.

Systems Manager Parameter Store

By comparison, AWS Systems Manager offers a Parameter Store which is a simple key-value pair storage option. It also offers encryption via AWS KMS, which allows the same security and simplicity of permissions management. Systems Manager is used by first installing the ssm-agent on your EC2 servers. Once it is installed, it can do things like:

  • Patch Management
  • Role/Identity Association
  • Scheduled commands
  • Run commands on a subset of servers at once
  • Organize resources into Resource Groups based on Tags
  • Show compliance with Patching and Access/Permissions policies
  • Store secure, encrypted variables in Parameter Store.

When it comes to storing parameters, the setup pane asks for a key name (which must be unique), and a value. You can store parameters as a basic String, a StringList, or a SecureString.

Parameter “value” strings can be up to 4096 characters to fit into the “Standard” pricing tier (free, and up to 10,000 parameters can be stored at this tier), or up to 8KB for the “Advanced” tier. “Advanced” tier secrets are priced at $0.05/advanced parameter per month.

If you choose the Advanced tier, expiration policies can be set on the parameters stored as well. Just like with Secrets Manager, additional tags can be added, and the values can be encrypted with the KMS key of your choice, making access control for your secrets more simple.

To recap, Parameter Store may offer more simplistic key-value pair storage, but is much less expensive (even at the Advanced tier). Secrets Manager offers several different storage types, most of which center around database credentials, but does offer a more simple key-value pair option too. Of the two, only Secrets Manager offers rotation, but the Advanced tier for Parameter Store does offer automatic expiration of parameters.

Ansible and Secret Management

With two options for secret management within AWS, it was difficult to know which to choose. We started with Secrets Manager, as Ansible offers both an aws_secret module, and an aws_secret lookup plugin.

aws_secret lookups

In our case, we were less interested in storing new secrets, and more interested in looking up the key and retrieving the value, for use in templates. That being the case, we chose to use the aws_secret lookup plugin. The example given in the documentation is:

- name: Create RDS instance with aws_secret lookup for password param
  rds:
    command: create
    instance_name: app-db
    db_engine: MySQL
    size: 10
    instance_type: db.m1.small
    username: dbadmin
    password: "{{ lookup('aws_secret', 'DbSecret') }}"
    tags:
      Environment: staging

Looks simple enough, right? Simply use the ‘aws_secret’ reference, and the name of the secret. Unfortunately it was not as simple for us.

Firstly, we found that adding the region to the command was necessary, like so:

"{{ lookup('aws_secret', 'my_api_key', region='us-west-1') }}"

That worked well enough in our vars_files to get through the deploy, provided the server running the ansible command had the proper IAM permissions. But, to my dismay, I found that this lookup didn’t return the “value” of the key-value pair, but rather a json string with BOTH the key and the value (shown below).

[{ 'my_api_key', 'my_api_key_value' }]

Unfortunately the only way I could get it to return just the “value” of the simple key-value style Secret was to add additional parsing in a script. So, as of now anyways, it looks like the Ansible aws_secret lookup plugin is limited to database secrets usage.

aws_ssm lookups

Enter Parameter Store. Since the Ansible aws_secret functions didn’t work as I had hoped, I tried the Systems Manager Parameter Store option instead. As with Secrets Manager, Ansible also has Parameter Store functionality in the form of the aws_ssm_parameter_store_module and the aws_ssm lookup plugin. And again, since we’re wanting to just read the value of secrets, we don’t need to mess with the module — just the lookup plugin. Ansible provides the following examples (although there are more use cases shown in the documentation):

- name: lookup ssm parameter store in the current region
  debug: msg="{{ lookup('aws_ssm', 'Hello' ) }}"

- name: lookup ssm parameter store in nominated region
  debug: msg="{{ lookup('aws_ssm', 'Hello', region='us-east-2' ) }}"

- name: lookup ssm parameter store without decrypted
  debug: msg="{{ lookup('aws_ssm', 'Hello', decrypt=False ) }}"

- name: lookup ssm parameter store in nominated aws profile
  debug: msg="{{ lookup('aws_ssm', 'Hello', aws_profile='myprofile' ) }}"

The examples given show easily enough how to use aws_ssm lookups within a playbook, but it can also be used in your vars_files like so:

environment: "{{ lookup('aws_ssm', 'env', region='us-west-2') }}"
app_name: "{{ lookup('aws_ssm', 'app_name', region='us-west-2') }}"
branch: "{{ lookup('aws_ssm', 'branch', region='us-west-2') }}"

Providing your instance is setup with the proper IAM permissions to read SSM parameters and read access to the KMS key used to encrypt them (if SecureString was selected), your variables should populate into templates without having to store them in an vaulted file or vaulting/encrypting individual strings.

Automating Parameter Addition

If your project (like ours) has a lot of vars to store, you may find it very tiresome to add all the keys one by one into the Systems Manager panel in AWS. As a DevOps engineer, it made me cringe thinking of having to add the variables by hand. So, I made a script that uses the AWS CLI to upload parameters.

A couple notes:

  • This script assumes you have an AWS CLI config file setup at ~/.aws/config, with multiple AWS account profiles. The one referenced is called “aws-main” — replace this with your own profile, or remove the line if you only have one profile.
  • The script adds a prefix of “dev_” to each variable, and a tag specifying the “Environment” as “Develop.” Tags and prefixes are not required, so feel free to tweak or replace as needed.
#!/usr/local/bin/bash -xe
declare -A vars

vars[env]=develop
vars[debug]=true
vars[key]="key_value"
#(more vars listed here...)

for i in "${!vars[@]}"
do
  aws ssm put-parameter \
  --profile "aws-main" \
  --name "dev_${i}" \
  --type "SecureString" \
  --value "${vars[$i]}" \
  --key-id "alias/dev-kms-key" \
  --tags Key=Environment,Value=Develop Key=Product,Value=Example \
  --region "us-west-2"
done

The bash script above declares an array “vars,” of which there are keys (env, debug, key) and values (develop, true, key_value). The loop uses the key as the iterator, and sets the value as the “value” in the SSM parameter.

There are still some manual steps in which I change values as needed per environment, and change tags/prefixes to reflect new environments. But this script helped cut the time to add parameters in 1/4 or more! Definitely a win in my book.

Conclusions

After some trial and error, here’s a recap of what we learned:

  • Secrets Manager is a more robust solution that offers rotation of secrets/keys. However, it is more expensive and charges for API calls.
  • If you’re looking to just populate the values of secrets for your variables in Ansible, SSM Parameter Store will work better for your needs.
  • Ansible’s aws_secret lookup works best for database Secrets.
  • Make sure you add an AWS region to your lookup
  • Shorten the time required to add Parameters using the AWS CLI and a bash loop.

Have any success or failure stories to share with either Secrets Manager or Parameter Store? Share in the comments, or contact me.

How to Create CIS-Compliant Partitions on AWS

If you use CIS (Center for Internet Security) ruleset in your security scans, you may need to create a partitioning scheme in your AMI that matches the recommended CIS rules. On AWS this becomes slightly harder if you use block storage (EBS). In this guide I’ll show how to create a partitioning scheme that complies with CIS rules.

Prerequisites:

  • AWS account
  • CentOS 7 operating system

CIS Partition Rules

On CentOS 7, there are several rules for partitions which both logically separate webserver-related files from things like logs, and limit execution of files (like scripts, or git clones, for example) in directories accessible by anyone (such as /tmp, /dev/shm, and /var/tmp).

The rules are as follows:

Below I’ll explain how to create a partition scheme that works for all the above rules.

Build your server

Start by building a server from your standard CentOS 7 AMI (Amazon Machine Image – if you don’t have one yet, there are some available on the Amazon Marketplace).

Sign in to your Amazon AWS dashboard and select EC2 from the Services menu.

In your EC2 (Elastic Compute Cloud dashboard), select the “Launch Instance” menu and go through the steps to launch a server with your CentOS 7 AMI. For ease of use I recommend using a t2-sized instance. While your server is launching, navigate to the “Volumes” section under the Elastic Block Store section:

Click “Create Volume” and create a basic volume in the same Availability Zone as your server.

After the volume is created, select it in the list of EBS volumes and select “Attach volume” from the dropdown menu. Select your newly-created instance from the list, and make sure the volume is added as /dev/sdf. *

*This is important – if you were to select “/dev/sda1” instead, it would try to attach as the boot volume, and we already have one of those attached to the instance. Also note, these will not be the names of the /dev/ devices on the server itself, but we’ll get to that later.

Partitioning

Now that your server is built, login via SSH and use sudo -i to escalate to the root user. Now let’s check which storage block devices are available:

# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
 xvda     259:0    0   20G  0 disk
 └─xvda1 259:1    0   20G  0 part /
 xvdf     259:2    0   20G  0 disk

If you chose t2 instance sizes in AWS, you likely have devices “xvda” and “xvdf,” where “xvdf” is the volume we manually added to the instance. If you chose t3 instances you’ll likely see device names like nvme0n1 instead. These devices are listed under dev on your instance, for reference.

Now we’ll partition the volume we added using parted.

# parted /dev/xvdf 
(parted) p 
Model: Xen Virtual Block Device (xvd) 
Disk /dev/xvdf: 18432MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt 
Disk Flags: 

Number  Start  End  Size  File system  Name  Flags 

(parted) mklabel gpt 
(parted) mkpart vartmp ext4 2MB 5% 
(parted) mkpart swap linux-swap 5% 10% 
(parted) mkpart home ext4 10% 15% 
(parted) mkpart usr ext4 15% 45% 
(parted) mkpart varlogaudit ext4 45% 55% 
(parted) mkpart varlog ext4 55% 65% 
(parted) mkpart var ext4 65% 100% 
(parted) unit GiB 
(parted) p
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdf: 18.0GiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt 
Disk Flags: 

Number  Start    End      Size     File system     Name         Flags
  1      0.00GiB  1.00GiB  1.00GiB  ext4            vartmp
  2      1.00GiB  2.00GiB  1.00GiB  linux-swap(v1)  swap
  3      2.00GiB  4.00GiB  2.00GiB  ext4            home
  4      4.00GiB  9.00GiB  5.00GiB  ext4            usr
  5      9.00GiB  11.0GiB  2.00GiB  ext4            varlogaudit
  6      11.0GiB  12.4GiB  1.40GiB  ext4            varlog
  7      12.4GiB  20.0GiB  7.60GiB  ext4            var

(parted) align-check optimal 1
1 aligned
(parted) align-check optimal 2
2 aligned
(parted) align-check optimal 3
3 aligned
(parted) align-check optimal 4
4 aligned
(parted) align-check optimal 5
5 aligned
(parted) align-check optimal 6
6 aligned
(parted) align-check optimal 7
7 aligned
(parted) quit 
Information: You may need to update /etc/fstab

Now when you run lsblk you’ll see the 7 partitions we created:

# lsblk 
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT 
xvda    202:0    0   20G  0 disk 
└─xvda1 202:1    0   20G  0 part / 
xvdf    202:80   0   18G  0 disk 
├─xvdf1 202:81   0  3.6G  0 part 
├─xvdf2 202:82   0  922M  0 part 
├─xvdf3 202:83   0  922M  0 part 
├─xvdf4 202:84   0  4.5G  0 part 
├─xvdf5 202:85   0  921M  0 part 
├─xvdf6 202:86   0  1.8G  0 part 
└─xvdf7 202:87   0  5.4G  0 part

After you’ve run through the steps above, you’ll have created the partitions, but now we need to mount them and copy the correct directories to the proper places.

First, let’s make the partitions filesystems using mkfs. We’ll need to do this for every partition except the one for swap! Note that we’re leaving out partition ID 2 in our loop below, which was the swap partition. After creating the filesystems, we’ll use mkswap to format our swap partition. Note also that you may need to change the “xvdf” parts to match the name of your secondary filesystem if it’s not xvdf. 

# for I in 1 3 4 5 6 7; do mkfs.ext4 /dev/xvdf${I}; done
# mkswap /dev/xvdf2

Next, we’ll mount each filesystem. Start by creating directories (to which we will sync files from their respective places in existing the filesystem). Again, if your filesystem is not “xvdf” please update the commands accordingly before running.

# mkdir -p /mnt/vartmp /mnt/home /mnt/usr /mnt/varlogaudit /mnt/varlog /mnt/var
# mount /dev/xvdf1 /mnt/vartmp
# mount /dev/xvdf3 /mnt/home 
# mount /dev/xvdf4 /mnt/usr 
# mount /dev/xvdf5 /mnt/varlogaudit 
# mount /dev/xvdf6 /mnt/varlog 
# mount /dev/xvdf7 /mnt/var

Now, we’ll sync the files from their existing places, to the places we’re going to be separating into different filesystems. Note, for the tricky ones that are all in the same paths (/var, /var/tmp, /var/log, and /var/log/audit), we have to exclude the separated directories from the sync and create them as empty folders with the default 755 directory permissions.

# rsync -av /var/tmp/ /mnt/vartmp/ 
# rsync -av /home/ /mnt/home/ 
# rsync -av /usr/ /mnt/usr/ 
# rsync -av /var/log/audit/ /mnt/varlogaudit/ 
# rsync -av --exclude=audit /var/log/ /mnt/varlog/ 
# rsync -av --exclude=log --exclude=tmp /var/ /mnt/var/ 
# mkdir /mnt/var/log 
# mkdir /mnt/var/tmp
# mkdir /mnt/var/log/audit 
# mkdir /mnt/varlog/audit 
# chmod 755 /mnt/var/log
# chmod 755 /mnt/var/tmp 
# chmod 755 /mnt/var/log/audit 
# chmod 755 /mnt/varlog/audit

Last, to create the /tmp partition in the proper way, we need to take some additional steps:

# systemctl unmask tmp.mount  
# systemctl enable tmp.mount 
# vi /etc/systemd/system/local-fs.target.wants/tmp.mount

Inside the /etc/systemd/system/local-fs.target.wants/tmp.mount file, edit the /tmp mount to the following options:

[Mount]  
What=tmpfs  
Where=/tmp  
Type=tmpfs  
Options=mode=1777,strictatime,noexec,nodev,nosuid

Now that the files are in the proper mounted directories, we can edit the /etc/fstab file to tell the server where to mount the files upon reboot. To do this, first, we’ll need to get the UUIDs of the partitions we’ve created:

# blkid 
/dev/xvda1: UUID="f41e390f-835b-4223-a9bb-9b45984ddf8d" TYPE="xfs" /dev/xvdf1: UUID="dbf88dd8-32b2-4cc6-aed5-aff27041b5f0" TYPE="ext4" PARTLABEL="vartmp" PARTUUID="5bf3e3a1-320d-407d-8f23-6a22e49abae4" 
/dev/xvdf2: UUID="238e1e7d-f843-4dbd-b738-8898d6cbb90d" TYPE="swap" PARTLABEL="swap" PARTUUID="2facca1c-838a-4ec7-b101-e27ba1ed3240" 
/dev/xvdf3: UUID="ac9d140e-0117-4e3c-b5ea-53bb384b9e3c" TYPE="ext4" PARTLABEL="home" PARTUUID="e75893d8-61b8-4a49-bd61-b03012599040" 
/dev/xvdf4: UUID="a16400bd-32d4-4f90-b736-e36d0f98f5d8" TYPE="ext4" PARTLABEL="usr" PARTUUID="3083ee67-f318-4d8e-8fdf-96f7f06a0bef" /dev/xvdf5: UUID="c4415c95-8cd2-4f1e-b404-8eac4652d865" TYPE="ext4" PARTLABEL="varlogaudit" PARTUUID="37ed0fd9-8586-4e7b-b42e-397fcbf0a05c" 
/dev/xvdf6: UUID="a29905e6-2311-4038-b6fa-d1a8d4eea8e9" TYPE="ext4" PARTLABEL="varlog" PARTUUID="762e310e-c849-48f4-9cab-a534f2fad590" 
/dev/xvdf7: UUID="ac026296-4ad9-4632-8319-6406b20f02cd" TYPE="ext4" PARTLABEL="var" PARTUUID="201df56e-daaa-4d0d-a79e-daf30c3bb114"

In your /etc/fstab file, enter (something like) the following, replacing the UUIDs in this example with the ones in your blkid output. Be sure to scroll all the way over to see the full contents of the snippet below!

#
# /etc/fstab
# Created by anaconda on Mon Jan 28 20:51:49 2019
# 
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#

UUID=f41e390f-835b-4223-a9bb-9b45984ddf8d /                       xfs     defaults        0 0
UUID=ac9d140e-0117-4e3c-b5ea-53bb384b9e3c /home                   ext4    defaults,noatime,acl,user_xattr,nodev,nosuid    0 2
UUID=a16400bd-32d4-4f90-b736-e36d0f98f5d8 /usr                    ext4    defaults,noatime,nodev,errors=remount-ro        0 2
UUID=c4415c95-8cd2-4f1e-b404-8eac4652d865 /var/log/audit          ext4    defaults,noatime,nodev,nosuid                   0 2
UUID=a29905e6-2311-4038-b6fa-d1a8d4eea8e9 /var/log                ext4    defaults,noatime,nodev,nosuid                   0 2
UUID=ac026296-4ad9-4632-8319-6406b20f02cd /var                    ext4    defaults,noatime,nodev,nosuid                   0 2
UUID=238e1e7d-f843-4dbd-b738-8898d6cbb90d swap                    swap    defaults                                        0 0
UUID=dbf88dd8-32b2-4cc6-aed5-aff27041b5f0 /var/tmp                ext4    defaults,noatime,nodev,nosuid,noexec            0 0
tmpfs                                     /dev/shm                tmpfs   defaults,nodev,nosuid,noexec                    0 0
tmpfs                                     /tmp                    tmpfs   defaults,noatime,nodev,noexec,nosuid,size=256m  0 0

If you were to type df -h at this moment, you’d likely have output like the following, since we mounted the /mnt folders:

# df -h 
Filesystem      Size  Used Avail Use% Mounted on 
/dev/xvda1       20G  2.4G   18G  12% / 
devtmpfs        1.9G     0  1.9G   0% /dev 
tmpfs           1.9G     0  1.9G   0% /dev/shm 
tmpfs           1.9G   17M  1.9G   1% /run 
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup 
tmpfs           379M     0  379M   0% /run/user/1000 
/dev/xvdf1      3.5G   15M  3.3G   1% /mnt/vartmp 
/dev/xvdf3      892M   81M  750M  10% /mnt/home 
/dev/xvdf4      4.4G  1.7G  2.5G  41% /mnt/usr 
/dev/xvdf5      891M  3.5M  826M   1% /mnt/varlogaudit 
/dev/xvdf6      1.8G   30M  1.7G   2% /mnt/varlog 
/dev/xvdf7      5.2G  407M  4.6G   9% /mnt/var

But, after a reboot, we’ll see those folders mounted as /var, /var/tmp, /var/log, and so on. One more important thing: If you are using selinux, you will need to restore the default file and directory contexts — this prevents you from being locked out of SSH after a reboot! 

# touch /.autorelabel;reboot

Wait a few minutes, and then SSH in to your instance once more. Post-reboot, you should see your folders mounted like the following:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
 devtmpfs        1.9G  4.0K  1.9G   1% /dev
 tmpfs           1.9G     0  1.9G   0% /dev/shm
 tmpfs           1.9G   25M  1.9G   2% /run
 tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
 /dev/xvda1   20G  5.7G   15G  29% /
 /dev/xvdf4  4.8G  2.6G  2.0G  57% /usr
 /dev/xvdf7  7.4G  577M  6.4G   9% /var
 /dev/xvdf3  2.0G  946M  889M  52% /home
 /dev/xvdf1  991M  2.6M  922M   1% /var/tmp
 /dev/xvdf6  1.4G  211M  1.1G  17% /var/log
 /dev/xvdf5  2.0G  536M  1.3G  30% /var/log/audit
 tmpfs           256M  300K  256M   1% /tmp
 tmpfs           389M     0  389M   0% /run/user/1002
 tmpfs           389M     0  389M   0% /run/user/1000

Voila! You’ve successfully created partitions that are compliant with CIS rules. From here you can select your instance in the EC2 dashboard, click “Actions” > “Stop,” and then “Actions” > “Image” > “Create Image” to create your new AMI using these partitions for use going forward!

Please note, I’ve done my best to include information for other situations, but these instructions may not apply to everyone or every template you may use on AWS or CentOS 7. Thanks again, and I hope this guide helps!