Kickbox Frownyface

Accurate Atomic Counters with Amazon DynamoDB

2021-08-14T12:43:07+00:00

Amazon DynamoDB is a fully managed key-value database at AWS. It’s great in a lot of ways: it’s serverless, it scales out incredibly wide, and it’s incredibly cheap for what it is. Good use cases for Dynamo are player preferences, game status, and low-cost global leaderboards.

But if you’re using Dynamo for an atomic counter under heavy load, that absolutey has to be accurate, it can be tricky to get right. In this post we’ll give an easy solution for incrementing a counter without losing updates.

To show this, we’ll imagine a common use case: adding one to a “likes” counter on an image, say. In this example, we have a DynamoDB table images which has a single primary key of id, an integer. Each item in this table has, among other things, a likes field which is supposed to store the number of likes that images has.

How can we implement an atomic increment to that likes column for any image id?

The Easy But (Sometimes) Wrong Way

Google DynamoDB atomic counters and you’ll immediately get to AWS’s documentation for atomic counters. The advice is to use ADD where you simply add one to the likes field of an images item.

In Python it looks like this:

import boto3
dynamodb = boto3.client('dynamodb')

def add_like(id):
    """Add one like to image `id`"""
    dynamodb.update_item(
        TableName='images',
        Key={'id': {'N': id}},
        UpdateExpression='ADD likes :num',
        ExpressionAttributeValues={':num': {'N': '1'}}
    )

So all that means is, “Add one to likes where id = id”. Simple! And we like simple over here at Kickbox Frownyface.

However, there’s a problem here: this update isn’t actually atomic, in that if you have two Lambda functions (say) calling this function at exactly the same time, one of the ADD functions is effectively dropped.

For example let’s say that the likes for this adorable kitten picture is at 5 and both Bob and Alice click “like” at pretty much the same time.

If you are unlucky, you may end up with likes just set to 6 instead of 7, because both Alice and Bob were adding one to 5 at precisely the same time. (If you don’t believe me, try it out – fan out 1000 or so Lambdas and you’re pretty likely to end up with likes less than the expected 1000.)

Is that bad? Well, that really depends. If you’re OK with the occasional dropped update, then leave well enough alone and move on – you have bigger fish to fry!

But if your counters just must be correct, let’s make a Conditional Write.

Adding a Conditional Expression

What we want to do is change the function from “just add one” to “just add one to what I think it is right now”. In our example, when Bob clicks, the function will run “Add one to likes so long as the current number of likes is five” and when Alice clicks, the function will also run “Add one to likes so long as the current number of likes is five.” If both calls happen simultaneously, one will go through, and the other will be rejected because the conditional is no longer satisfied.

The code is pretty similar, we just do a fetch then a conditional update:

import boto3
dynamodb = boto3.client('dynamodb')

def add_like(id):
    """Add one like to image `id`"""
    existing = dynamodb.get_item(
            TableName='images',
            Key={'id': {'N': id}}
        )
    dynamodb.update_item(
        TableName=self.table_name,
        Key={'id': {'N': id}},
        UpdateExpression='ADD likes :num',
        ConditionExpression='likes = :current',
        ExpressionAttributeValues={':num': {'N': '1'},
            ':current': {'N': existing['Item']['likes']['N']
        }
    )

This code is fetching the current record for image id and storing that in existing. Then it does an update on that image, adding 1 to likes so long as the current likes value is equal to the value we just fetched.

Now, if Alice and Bob both add their likes at precisely the same time, one of them will go through and the other will have a ConditionalCheckFailedException thrown.

Are we done? Not quite! We still have the number of likes for that image set to six! Either Alice’s or Bob’s like wasn’t recorded. But now at least we know it was dropped, so let’s retry the failed one with code like this:

def add_like_safely(id, tries=10):
    attempts, success = 0, False
    while attempts <= tries and not success:
        try:
            attempts = attempts + 1
            results = add_like(id)
            success = True
       except ClientError as e:
            if e.response['Error']['Code'] == 'ConditionalCheckFailedException': 
                time.sleep(0.01)
            else:
                raise
    return success

This way, if both Alice and Bob call add_like_safely at precisely the same time, one of them will get through first… but the exception will be caught and the other will try again after a short delay to give the data time to propagate out.

Automatically Building and Deploying Jekyll on AWS with CodePipeline

2021-04-23T17:32:45+00:00

In our last post we covered how to use S3 and CloudFront to host your Jekyll site on AWS statically, which keeps it fast and very inexpensive to operate. In this post, we will cover how to set up CodePipeline and CodeBuild to automatically rebuild the site and publish it to the web whenever you commit a change to your CodeCommit repository. When you’ve hooked up your website in this way, you will never want to go back to the old way: you can just concentrate on adding content and designing your site without having to concern yourself with how it actually gets live on the web. It’s a wonderful way to work.

Building the Jekyll Site Automatically

Whenever we make a change to our Jekyll site – adding a blog post, making a change to the theme, whatever – we need to rebuild it using the command jekyll build. To do this on AWS, we use the CodeBuild service. We’ll only pay for the compute power in the cloud we use as we use it, by the second, so it’s a very cost effective solution.

Create a new file in your infra directory called codebuild.yml. If you’ve been following along, you’ll recognize in the CloudFormation template below some variables that the Serverless Framework will fill in for us, for example ${self:custom.account} is our account number.

Here’s the file:

# The build project that runs buildspec.yml in the tree
Resources:

  CodeBuildServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
              - codebuild.amazonaws.com

  CodeBuildServicePolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: CodeBuildServicePolicy
      Roles:
        - !Ref CodeBuildServiceRole
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
            Resource: "arn:aws:logs:${self:provider.region}:${self:custom.account}:*"
          - Effect: Allow
            Action:
              - codecommit:GitPull
            Resource: "arn:aws:codecommit:${self:provider.region}:${self:custom.account}:${self:custom.repo}"
          - Effect: Allow
            Action:
              - s3:GetObject
              - s3:GetObjectVersion
              - s3:PutObject
            Resource:
              - "arn:aws:s3:::${self:custom.s3Bucket}*"
          - Effect: Allow
            Action:
              - cloudfront:CreateInvalidation
            Resource: "*"

  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: ${self:custom.project}-${self:custom.stage}
      Description: Build for ${self:custom.subdomain}.${self:custom.domain}
      ServiceRole: !GetAtt CodeBuildServiceRole.Arn
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        Type: linuxContainer
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
        EnvironmentVariables:
          - Name: CLOUDFRONT_DISTRO_ID
            Value: ${self:custom.cloudfront_id}
          - Name: S3_BUCKET
            Value: ${self:custom.s3Bucket}
      LogsConfig:
        CloudWatchLogs:
          Status: ENABLED
          GroupName: /build/${self:custom.project}-${self:custom.stage}
      Source:
        Type: CODEPIPELINE
      TimeoutInMinutes: 60

It’s a pretty long file, it’s true, but there’s nothing complicated here.

The first two stanzas are policies that give the CodeBuild service the privileges it needs to do its work: the ability to write to CloudWatch logs, to pull source code from our git repo that we set up last time, and the ability to copy the created HTML files (the actual website) to the static S3 bucket that houses them.

Then the CodeBuildProject resource is the CodeBuild project itself, which tells AWS what kind of EC2 instance we need to do the build, and where it should be getting the source code from (CodePipeline; see below). Note that there are two environment variables that we’re setting for the instance: S3 bucket (where the static HTML files that Jekyll builds should go) and the CloudFront Distribution that’s fronting it.

Triggering the Build on a Code Commit

Now we want to set up our CodePipeline, which will automatically trigger that build whenever we push any changes to the master branch of our repo.

Create a file codepipeline.yml in your infra directory with this content:

Resources:

  CodePipelineBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: ${self:custom.s3Bucket}-codepipeline
      PublicAccessBlockConfiguration:
        BlockPublicAcls : true
        BlockPublicPolicy : true
        IgnorePublicAcls : true
        RestrictPublicBuckets : true
        
  CodePipelineServiceRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - codepipeline.amazonaws.com
            Action: "sts:AssumeRole"
      Path: /
      Policies:
        - PolicyName: CodePipelineServicePolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "codecommit:CancelUploadArchive"
                  - "codecommit:GetBranch"
                  - "codecommit:GetCommit"
                  - "codecommit:GetUploadArchiveStatus"
                  - "codecommit:UploadArchive"
                Resource: "arn:aws:codecommit:${self:provider.region}:${self:custom.account}:${self:custom.repo}"
              - Effect: Allow
                Action:
                  - "codebuild:BatchGetBuilds"
                  - "codebuild:StartBuild"
                Resource: "*"
              - Effect: Allow
                Action:
                  - "s3:*"
                Resource: "arn:aws:s3:::${self:custom.s3Bucket}-codepipeline/*"

  AmazonCloudWatchEventRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - events.amazonaws.com
            Action: "sts:AssumeRole"
      Path: /
      Policies:
        - PolicyName: CodePipelineExecutionPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: "codepipeline:StartPipelineExecution"
                Resource:  "arn:aws:codepipeline:${self:provider.region}:${self:custom.account}:${self:custom.project}-${self:custom.stage}"
  
  AmazonCloudWatchEventRule:
    Type: "AWS::Events::Rule"
    Properties:
      EventPattern:
        source: 
          - aws.codecommit
        detail-type: 
          - CodeCommit Repository State Change
        resources: 
          - "arn:aws:codecommit:${self:provider.region}:${self:custom.account}:${self:custom.repo}"
        detail:
          event:
            - referenceCreated
            - referenceUpdated
          referenceType: 
            - branch
          referenceName: 
            - master
      Targets:
        - Arn: "arn:aws:codepipeline:${self:provider.region}:${self:custom.account}:${self:custom.project}-${self:custom.stage}"
          RoleArn: !GetAtt 
            - AmazonCloudWatchEventRole
            - Arn
          Id: codepipeline-AppPipeline


  AppPipeline:
    Type: "AWS::CodePipeline::Pipeline"
    Properties:
      Name: ${self:custom.project}-${self:custom.stage}
      RoleArn: !GetAtt 
        - CodePipelineServiceRole
        - Arn
      Stages:
        - Name: Source
          Actions:
            - Name: SourceAction
              RunOrder: 1
              ActionTypeId:
                Category: Source
                Owner: AWS
                Version: 1
                Provider: CodeCommit
              Configuration:
                BranchName: master
                RepositoryName: ${self:custom.repo}
                PollForSourceChanges: false
              OutputArtifacts:
                - Name: SourceArtifact
        - Name: Build
          Actions:
            - Name: BuildAction
              RunOrder: 2
              ActionTypeId:
                Category: Build
                Owner: AWS
                Version: 1
                Provider: CodeBuild
              InputArtifacts:
                - Name: SourceArtifact
              OutputArtifacts:
                - Name: BuildArtifact
              Configuration:
                ProjectName: ${self:custom.project}-${self:custom.stage}
      ArtifactStore:
        Type: S3
        Location: ${self:custom.s3Bucket}-codepipeline

This is another long CloudFormation template, but again there’s nothing conceptually challenging here.

The first stanza just sets up an S3 bucket for the pipeline to place the source code in, and from which CodeBuild picks it up automatically. We configure that bucket to be private to us.

Then we set up a policy for the pipeline. It needs the ability to work with CodeCommit, our git repo; to start builds in CodeBuild; and of course to put source code into that S3 bucket we just created.

We also need to give CloudWatch the ability to trigger the builds themselves. Whenever a file is created or updated in the repo, that will kick off the pipeline itself.

Finally we have the AppPipeline pipeline itself. This sets up two distinct phases: a Source phase, which does a full pull from git and which copies the output into the code pipeline bucket. This is called the SourceArtifact. That artifact is referenced as input in the second Build phase.

So, to sum up: whenever there’s a push done to the git repo, the Pipeline will kick in. It will do a full get on the source tree, put a copy of it in S3, and will then trigger the Build.

The BuildSpec File

But how does CodeBuild know how to build a Jekyll website? Well, we need a build specification file that tells CodeBuild what steps exactly it should take.

This is pretty simple: essentially, in the build phase we tell CodeBuild to take the source code that was given to it by CodePipeline, and run Jekyll on it with the production configuration settings. That will put the appropriate files in the _site directory on the EC2 instance doing the build.

Then, in the post_build phase, copy the _site directory over to our S3 bucket for serving. Finally, invalidate the CloudFront cache so that our reading public can get the latest and the greatest right away.

If you’ve been following along, your buildspec.yml file should be at the root of your git repo and should look like this:

version: 0.2

phases:
  install:
    commands:
      - gem install jekyll bundler
      - cd website
      - bundle install
  build:
    commands:
      - JEKYLL_ENV=production bundle exec jekyll build --config _config.yml,_config_production.yml

  post_build:
    commands:
      - aws s3 sync _site s3://$S3_BUCKET
      - aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRO_ID --paths "/*"

Note that the build is getting the S3_BUCKET and the CLOUDFRONT_DISTRO_ID as environment variables. Those were placed there by CodeBuild, in the EnvironmentVariables section.

Revisiting the Serverless Framework Configuration

As we discussed last time, the serverless.yml file includes the configuration settings to make all of this work properly. We need to make a few adjusments so that when we deploy our infrastructure stack, everything is hooked up properly.

Here’s how ours looks now:

service: jekyll-web

custom:
  project: jekyll-web
  account: '[YOUR ACCOUNT ID]'
  certificate: [CERTIFICATE ID]
  domain: [YOUR DOMAIN]
  sslCertArn: arn:aws:acm:us-east-1:${self:custom.account}:certificate/${self:custom.certificate}
  stage: ${opt:stage, self:provider.stage}
  subdomain: ${self:provider.environment.${self:custom.stage}_subdomain}
  s3Bucket: jekyll-web-${self:custom.stage}
  cloudfront_id:  ${self:provider.environment.${self:custom.stage}_cloudfront_id}
  repo: [YOUR REPO NAME]

provider:
  name: aws
  stage: prod
  region: [YOUR PREFERRED AWS REGION]
  environment:
    dev_subdomain: dev
    dev_cloudfront_id: 
    prod_subdomain: www
    prod_cloudfront_id: [YOUR CLOUDFRONT ID]
  stackTags:
    Project: Corporate

resources:
  - ${file(s3.yml)}
  - ${file(cloudfront.yml)}
  - ${file(codebuild.yml)}
  - ${file(codepipeline.yml)}

We’ve added our CodeBuild and CodePipeline infrastructure pieces to the bottom, and we’ve added some custom variables that help us parameterize the scripts so we can use them for other purposes.

Note that the Certificate Arn and the CloudFront ID have to be manually added to the serverless.yml file for this to work. That’s kind of lame, I admit – but whatever. It works.

Add the Files to Git

OK, we have a number of new files that we’ve added. If you’re doing it exactly the same way we are, your file tree should look like this:

.
├── buildspec.yml
├── infra
│   ├── cloudfront.yml
│   ├── codebuild.yml
│   ├── codepipeline.yml
│   ├── s3.yml
│   └── serverless.yml
├── README.md
└── website
    ├── 404.html
    ...

Be sure to add all these files to your git repo.

As soon as you push your commit to the git repo, CodePipeline will recognize that there’s been a change to the tree and will spring into action. If you log into your AWS console and navigate to your new CodePipeline, you should see something like this:

Deploying and Triggering the Build

Deploy all that infrastructure using the Serverless Framework:

$ sls deploy

It will take some time for Serverless to build the CodeBuild and CodePipeline infrastructure elements in AWS.

To test it you can make a trivial change to your Jekyll site – change the site description in the _config.yml file, for example – and push it up to git. You should be able to watch CodePipeline start processing.

Conclusion: What Have we Wrought?

We’ve done a lot in this three-step blog series, so it’s worth reviewing.

First, we set up a Jekyll development environment fully in the cloud: we have a Cloud9 environment as our Jekyll IDE, for which we only pay when we are actually using it. The source of our Jekyll site is safely stored in a fully managed git repo in CodeCommit. When we make changes to our Jekyll site and commit those changes to git, a pipeline springs into being and builds static HTML files for the site automatically. These files are then automatically copied over to an S3 bucket, from which our site is securely served behind an SSL certificate. And these HTML files are distributed by CloudFront, a Content Delivery Network, for extremely speedy rendering no matter where in the world your readers are.

Not only does that save a ton of time over the long run, but it does it extremely cheaply and reliably.

We hope this blog series has been useful to you.

Hosting Your Jekyll Site on AWS with S3 and CloudFront

2021-03-19T14:22:08+00:00

In our last post we described how to install Jekyll on a Cloud9 environment, with the website source stored in git in CodeCommit. In this post, we’ll describe how to host that site on AWS statically, via S3 and CloudFront. This is an incredibly cost effective way to host a site – and, as you’ll see, your web pages will load incredibly fast from anywhere in the world.

SSL Certificates with AWS Certificate Manager

Even for static websites like this one, the public expects a secure browsing experience, so you simply must host your site over https. To do this, you need an SSL certificate for your domain, properly configured for the webserver that is hosting up your site. In the past, that was a time consuming and expensive process, but AWS has you covered here: so long as you actually use the certificate with your website hosted on AWS, it’s free. And it’s pretty easy.

Navigate to your AWS console, then go to the AWS Certificate Manager service. Click on Provision Certificate, and then – this is very important – switch to the us-east-1 region, no matter what your standard region is. We will be configuring CloudFront to use this cert, and that only works with certs that are in us-east-1.

We’ll be hosting the website at www.[YOUR DOMAIN].com, but we may as well ask for a wildcard cert as long as we are doing this. Further, you’ll want to protect your bare domain (i.e., with no subdomain) in the same cert:

Next, you will have to prove that you have ownership of the domain. You can do this in one of two ways: either via an email process, or a DNS process. The DNS one is cleaner; all you need to do is set up a CNAME with the key and value that ACM provides you with. The Kickbox domain uses DreamHost as DNS provider, because it’s cheap and good enough, so we created a CNAME on the domain with the details that ACM gave us:

It can take some time for your DNS provider to serve it up, and for AWS to notice that it exists – so you can either get a cup of coffee and come back, or move on to the next stage while you’re waiting.

Serverless Framework on Cloud9

Here at Kickbox Frownyface we are big fans of the Serverless Framework, which standardizes, to some extent, how you interact with cloud providers. Even though we are all in on AWS, we think it really simplifies maintaining infrastructure as code. And it’s free.

So let’s install Serverless on our development box.

Log into your Cloud9 instance, and then issue this from the terminal:

$ npm install -g serverless
...
$ sls --version
Framework Core: 2.18.0
Plugin: 4.4.2
SDK: 2.3.2
Components: 3.4.7

Configure your S3 Bucket

We will be using S3’s static website hosting feature for the site that Jekyll is going to build for us. Instead of going into the AWS console and clicking around, we are going to create a YAML file that describes our S3 bucket fully, and then have Serverless take care of spinning it up and maintaining it for us.

The infrastructure configuration files for our website are going to go into a directory sibling of website. So create a directory parallel to where our Jekyll work goes:

├── kickbox-web
│   ├── infra
│   └── website
│       ├── 404.html
...

Into this infra directory we’ll put s3.yml, which will specify an S3 bucket that will contain the Jekyll site as built (i.e., the static HTML pages, styleshets, images, and so on for our site). It looks like this:

Resources:

  # The bucket holding the static site
  JekyllBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: ${self:custom.s3Bucket}
      AccessControl: PublicRead
      WebsiteConfiguration:
        IndexDocument: index.html
        ErrorDocument: index.html
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders:
              - '*'
            AllowedMethods:
              - GET
              - HEAD
            AllowedOrigins:
              - '*'

  JekyllBucketPolicy:
    Type: "AWS::S3::BucketPolicy"
    DependsOn: JekyllBucket
    Properties:
      Bucket: !Ref JekyllBucket
      PolicyDocument:
        Statement:
          - Sid: PublicReadGetObject
            Effect: Allow
            Principal: "*"
            Action:
            - s3:GetObject
            Resource: arn:aws:s3:::${self:custom.s3Bucket}/*

This sets up a bucket that S3 can host statically, with public read access. Note that the ${self:custom.s3Bucket} and the like are going to be filled in appropriately by Serverless when we deploy our infrastructure.

(A quick note about public read: normally you would want to have CloudFront, our CDN, to be the only entity allowed to read from that S3 bucket. However, the way Jekyll builds out directories and index.html files means this won’t fly. A public S3 bucket is OK for us because our website is fully public anyway – but if your use case is different, you may want to make a different choice.)

Configure CloudFront as CDN

Next, we will set up CloudFront to serve files from that S3 bucket. To do this, we’ll create another YAML configuration file, this time for the distribution:

Resources:

  ## The CF distribution for the front end
  JekyllCloudFrontDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Origins:
          - DomainName: ${self:custom.s3Bucket}.s3-website.${self:provider.region}.amazonaws.com
            Id: WebApp
            CustomOriginConfig:
              HTTPPort: 80
              HTTPSPort: 443
              OriginProtocolPolicy: http-only
        Enabled: true
        Aliases:
          - ${self:custom.subdomain}.${self:custom.domain}
        DefaultRootObject: index.html
        CustomErrorResponses:
          - ErrorCode: 404
            ResponseCode: 200
            ResponsePagePath: /404.html

        # The default cache behavior: standard pages
        DefaultCacheBehavior:
          AllowedMethods:
            - GET
            - HEAD
            - OPTIONS
          ## The origin id defined above
          TargetOriginId: WebApp
          ForwardedValues:
            QueryString: true
            Cookies:
              Forward: none
          ViewerProtocolPolicy: redirect-to-https
          Compress: true
          DefaultTTL: 2628000
          MinTTL: 2628000

        ## The certificate to use when viewers use HTTPS to request objects.
        ViewerCertificate:
          AcmCertificateArn: ${self:custom.sslCertArn}
          MinimumProtocolVersion: "TLSv1.2_2018"
          SslSupportMethod: sni-only

## In order to print out the hosted domain via `serverless info` we need to define the DomainName output for CloudFormation
Outputs:
  JekyllCloudFrontDistributionOutput:
    Value:
      'Fn::GetAtt': [ JekyllCloudFrontDistribution, DomainName ]

A few things to note about this file: we have a very long “time to live” set up for these files. That’s because our site is, after all, static – and performance will be better (and cost lower) the longer those files are at CloudFront’s edge. This means every time we rebuild our Jekyll site, we need to invalidate the CloudFront cache. (We will do this automatically in our next step; stay tuned.)

Serverless Configuration

Finally, we a single configuration file for the Serverless framework that references these two files. The standard name for this file is serverless.yml, so create one in the infra directory that looks like this:

service: jekyll-web

custom:
  account: '[YOUR ACCOUNT ID]'
  certificate: [CERTIFICATE ID]
  domain: [YOUR DOMAIN]
  sslCertArn: arn:aws:acm:us-east-1:${self:custom.account}:certificate/${self:custom.certificate}
  stage: ${opt:stage, self:provider.stage}
  subdomain: ${self:provider.environment.${self:custom.stage}_subdomain}
  s3Bucket: [BUCKET NAME THAT WILL HOLD YOUR SITE]-${self:custom.stage}

provider:
  name: aws
  stage: prod
  region: [YOUR PREFERRED AWS REGION]
  environment:
    dev_subdomain: dev
    prod_subdomain: www

resources:
  - ${file(s3.yml)}
  - ${file(cloudfront.yml)}

What wer’e doing in this file is setting up values for the variables in the s3.yml and cloudfront.yml files. You will obviously need to edit this for your own circumstances: you will need to put in your AWS account ID, what region you run your infrastructure in, and so on.

The only non-obvious part step here is that you need the SSL certificate ARN (that stands for AWS Resource Name), so that CloudFront knows how to secure your site. How do we get this?

By now, Amazon Certificate Manager should have verified that you own the domain in question, by looking for that CNAME setting in DNS that you set up at the start of this process. So pop back over to your console and see if the certificate has been issued. If so, you can click on Details and see this:

)

You want the Identifier, smudged out here.

Deploy Your Infrastructure

At this point our project directory looks like the following, with the Jekyll source in website and the infrastructure support in infra:

├── kickbox-web
│   ├── infra
│   │   ├── cloudfront.yml
│   │   ├── s3.yml
│   │   └── serverless.yml
│   └── website
│       ├── 404.html

Now that the hard work is done, you can simply tell Serverless Framework that you want your site built out:

$ sls deploy

That’s it! Serverless will use these files to create CloudFormation scripts and post them as stacks to your AWS account, using the privileges in your Cloud9 instance. This will generate an S3 bucket, configured for static website hosting, and a CloudFront distribution pointing to it, properly configured to be serving up content, securely via your SSL certificate, at https://www.[YOUR DOMAIN].

(Of course, you will have to add a CNAME for www pointing to your CloudFront domain, so that www.[YOUR DOMAIN] actually points at your website. If you don’t know how to do this, check with your domain registrar.)

But how, you might ask, do we get our Jekyll site up into that S3 bucket? Well, that is the topic of our next post, automatically building and deploying Jekyll on AWS with CodePipeline.

Installing Jekyll on AWS Cloud9 with CodeCommit

2021-02-25T23:40:19+00:00

Hosting a website like this one on serverless AWS is easy, fast, and dirt cheap. Combine that with a static website generator like Jekyll and you have the best of both worlds: a static website that is easy to contribute to, easy to theme, and which loads fast no matter where in the world your users are.

But there’s no doubt that getting everything set up to get everything automated can be a bit of a headache. So we offer this, the first in a series that shows how we at Kickbox Frownyface build our corporate website on AWS, starting with the Cloud9 development environment for editing, and CodeCommit for source code control.

It should only take you ten or fifteen minutes to follow these steps.

We hope you find it useful.

Cloud9 as Jekyll IDE

It’s shouldn’t surprise anyone that we have a pretty small development team at Kickbox Frownyface, and we can’t spend a lot on hardware. Cloud9 is a perfect fit for us when it comes to editing the website: there’s no hardware to install or machines to configure, and we only pay for the IDE by the minute, for the time we spend editing the website. We don’t need to install a virtual machine or have a Linux box of our own, or any of that nonsene. And we don’t have to worry about dirtying up our own game dev machines by installing other software, which is a huge win for us.

Create the Cloud9 Environment

Your first step is to create a new Cloud9 environment. This will be the Linux box that we’ll install Jekyll to.

Sign in to your AWS console and then navigate to the Cloud9 service. Click the Create Environnment button, and fill out the form. Here are our settings:

We choose the cheapest instance type, t2.micro, because Jekyll doesn’t need much horsepower. We name ours kickbox-web because of our internal standards.

Install Ruby and Jekyll

Open the Cloud9 IDE. You’ll see a terminal window in the lower part of your screen. This is a Linux prompt that we can use to install all the tools we’ll need onto the EC2 instance that is backing our Cloud9.

Here are the steps we took to do this. First we installed Ruby, which Jekyll needs:

$ sudo amazon-linux-extras install ruby2.6
$ sudo yum install ruby-devel -y

If that worked Ruby will tell you its version, something like this:

$ ruby -v
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]

Next up is Jekyll:

$ gem update --system
$ gem install jekyll bundler
$ bundle config path ~/.gem

If that worked Jekyll will report its version too:

$ jekyll -v
jekyll 4.2.0

If you run into trouble, things may have changed since we wrote this post. Check out the Jekyll installation instructions, which will have up to date instructions.

Code Commit

Next up, we’ll create a git repository which will store our website source – that is, the Jekyll website files that Jekyll will build into HTML files for our website.

First, you need to make sure that the user you are running Cloud9 as has git privs to code commit so you can just git push to the repo from the IDE. So make sure that your IAM user, and the other editors’ users, belong to a group that includes the correct policies. To do this:

Navigate to the IAM service
Choose Groups
Select an existing group, or click on Create New Group and type in website-editors
Check the box next to AWSCodeCommitFullAccess
Complete the wizard and create the group
Click into the group and click on Add Users to Group
Add yourself and any other IAM users to the group

Next, head back over to your AWS console and navigate to CodeCommit. You will create a new Git repository, which will hold the source code and content for the Jekyll website. We call ours kickbox-web.

Now we just go back into Cloud9 and git clone it into the ~/environment directory that you have on that Linux box. Copy the HTTPS path and then issue:

$ git clone https://git-codecommit.[YOUR REGION].amazonaws.com/v1/repos/[YOUR REPO NAME]
Cloning into '[YOUR REPO NAME]'...
warning: You appear to have cloned an empty repository.

The warning doesn’t bother us, because it is expected to be empty. Let’s finish configuring your git settings and push up a README.md file:

$ cd [YOUR REPO NAME]
$ git config --global user.name "[YOUR NAME]"
$ git config --global user.email [YOUR EMAIL]
$ cat >README.md
A website.
[CTRL-D]
$ git add README.md 
$ git commit -m "Initial commit"
$ git push

Now that our tools are all set up, let’s just create a new empty Jekyll website and verify that everything is ready.

Create the New Website

We will ask Jekyll to build its new site in a directory named website, rather than the root. The reason for this will become clear in future steps – we also want an infrastructure directory, into which we can put our CloudFormation scripts.

$ jekyll new website
...
New jekyll site installed in /home/ec2-user/environment/[YOUR REPO NAME]/website. 

If that worked, you’ll see that your file tree in Cloud9 is now full of the files that are part of a standard Jekyll website.

Let’s cd into that directory and ask Jekyll to host it during our development:

$ cd website
$ jekyll serve -H $IP -P $PORT --baseurl ""

Keep your eyes open – Cloud9 will show you a little window with a hyperlink on it to your website running in dev. If you click on it you should see your new Jekyll website:

Commit to Git

Don’t forget to commit the starter website to CodeCommit.

$ git add .
$ git commit -m "Initial empty Jekyll site"
$ git push

Conclusion

So what have we accomplished? Well, we have a new cloud-based IDE all set up for us to build out our Jekyll website, and we have a git repository in which to store it.

Next step: hosting it on S3 and Cloudfront.