Archive

Posts Tagged ‘Continuous Deployment’

Creating a staging + production environment for a Node.js service on AWS Elastic Beanstalk with Cloudbees as CD service in 51 steps

February 24, 2015 Leave a comment
  1. In AWS:
  2. Login to AWS, add a new AIM user called Jenkins_EBS_Access (or whatever name makes you happy), save the credentials, attach the following policy to the user as an inline, custom policy:

  3. {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Sid": "Stmt1414820271000",
    "Effect": "Allow",
    "Action": [
    "elasticbeanstalk:*",
    "elasticloadbalancing:*",
    "autoscaling:*",
    "ec2:*",
    "s3:*",
    "cloudformation:*",
    "sns:*",
    "cloudwatch:*",
    "iam:*"
    ],
    "Resource": [
    "*"
    ]
    }
    ]
    }
  4. Go to S3, add a new bucket called deployments
  5. Add two plugins in the Jenkins manage plugins area:
    1. Cloudbees Amazon Web Services Credentials plugin
    2. Cloudbees Amazon Web Services deploy engine plugin
  6. In Cloudbees:
  7. Create your Cloudbees repository, commit a working node.js service
  8. Goto builds and add a new build
  9. In build configuration, below deploy now click on Advanced, remove the default cloudbees host service, and add an Elastic Beanstalk deployment, create a new credentials entry and add the credentials that you saved for the AWS user you just created.
  10. Click add application
  11. Pick your region (us-east-1 is the default and cheapest option)
  12. In S3 bucket, enter deployments/${BUILD_NUMBER}
  13. In application name, enter service
  14. In version label, enter ${JOB_NAME}.#${BUILD_NUMBER}
  15. In environment name, enter myservice-stg
  16. Click “promote builds when”
  17. in name enter production
  18. pick the star color you want (I like green)
  19. Click “only when manually approved”
  20. Add action “deploy application”
  21. Repeat process 8 – 13 (minus the add new credentials step, just use the ones you already added)
  22. In environment name, enter myservice-prd
  23. In build->execute shell->command, enter the following:

  24. node_version=0.10.21
    if [ ! -e nodejs-plugin-$node_version.zip ]
    then
    wget https://s3.amazonaws.com/clickstacks/admin/nodejs-plugin-$node_version.zip
    unzip nodejs-plugin-$node_version.zip
    tar xf node.tar.gz
    mv node-v* node_lib
    fi

    rm -rf target
    mkdir target

    export PATH=$PATH:$WORKSPACE/node_lib/bin

    npm install
    npm install time
    npm install grunt grunt-cli

    export PATH=node_modules/grunt-cli/bin/:$PATH
    #export PATH=$PATH:node_modules/grunt-cli/bin/

    grunt

    rm -rf target/app.war
    cd dist
    zip -r ../target/app.war .

  25. Note that you might need to change the node.js version number, and also the last couple of lines, if your directory structure is a bit different than mine.  My code uses grunt and deploys to a directory called dist
  26. Again, repeat steps 8 – 13
  27. In environment name, enter myservice-stg
  28. In post build actions, in Archive the artifacts, change *.zip to *.war
  29. Click save
  30. Click Build NOW
  31. Go to the build console output view and make sure there are no errors in the build
  32. Your node.js main file should be called app.js or server.js, there are the app names which AWS calls by default (configurable), or npm start should trigger your app if all else fails.
  33. Once the build has finished, you should see that the app has been uploaded to s3, but you should expect to see a failure which says: “No Application named ‘service’ found”.  This is ok, we have not created the app yet, but we now have the app in S3 and this is where we’ll go next.
  34. Go to AWS S3, to the deployments bucket, you should see a new file under a folder with your current version number called service.  Click it and hit properties.
  35. Copy the link url
  36. Go to Elastic Beanstalk, and create a new application.  The name should be the same name you entered in cloudbees build, in the Application Name field.
  37. Now create a new web server
  38. select node.js platform, and keep the load balanced configuration
  39. under application version, pick s3, and paste the s3 bucket URL you just copied
  40. In environment name, enter the environment name you entered in cloudbees (myservice-stg)
  41. next, next
  42. In instance type pick t2.micro, this is actually cheaper than t1.micro and has better performance
  43. Go to EC2 -> key pairs (pick from the left side list), and create a new key pair.  You will automatically download a PEM file which you will later use to access your instances if you wish
  44. Back in Elastic Beanstalk, refresh the EC2 key pairs listing and pick your newly created key pair
  45. In environment tags, add the following tags:
    1. key: type, value: service
    2. key: class, value: staging
  46. These tags will let you report on usage later on
  47. Launch your environment
  48. Repeat the process, but this time with myservice-prd as the environment name, and in the class, enter production
  49. In Cloudbees, Click Build now.  The build should now succeed, and auto-deploy to the staging environment.
  50. Once it’s deployed and you see in AWS that the staging service is updated with the new build, go back to Cloudbees and
  51. Click the new build, go to Promotion Status, and on the right hand, side, click Force Promotion
  52. The build should now be promoted to the production environment
  53. Voila, you now have a CD process with staging and production environments on AWS
Advertisements

Online service development in java – a beginner’s guide

November 4, 2012 1 comment

In the coming series of posts, I will share my development experience (code included) with creating an online service from scratch.  If you read through all posts, you should reach a point where you are capable of writing code and deploying it to a production like environment which is available online, using some of the latest (and greatest) open source libraries available in java.  You will definitely not be at the end of the journey, but will get a pretty good head start.

I’ve created a sample project, which basically does nothing (well, almost nothing), but it does so in a (semi) secure way, using several top libraries.  Basic service, is a service which lets you register, login, and… do nothing.  But, it does so using:

Spring, Maven, ESAPI (+AppSensor) for input validation, JSP’s (JSTL+Tiles 2), jQuery (+UI), Mongo DB, Logback, Jackson, Mockito (+PowerMock) for unit testing, and it does it in a RESTful way, using complete Data and Presentation separation (all JSP’s get their data through REST requests, meaning you can add easily add mobile support without almost any change to the  interface), and is fully internationalized (currently supports hebrew and english, meaning we have RTL languages covered).  The complete source code for Basic service is available on google code, it is a fully functional (and GPL open source) example of combining all of these technologies , which means that you can concentrate on the next steps rather than spend time building the foundation from scratch.  It took me about 3 months to go through all the baby steps in each of those technologies (after spending some time picking the right ones of course), I hope this series will save you at least some of that time.

Please keep in mind that I do not claim this to be a complete and bullet proof piece of software, there is still much to do in order to make it “production ready”.

Basic service is deployed on Cloudbees, using Continuous Deployment,  you’ll get some insights on how to do that as well.

One final note: the title says “a beginner’s guide”, but should be read with two different interpretations: one is (possibly) you, who is just getting started with creating an online service, and the other is me, sharing my experience with jumping into these deep water a couple of months ago.  If you like what you read, you’re welcome to comment, if you see me making horrible mistakes, constructive criticism is more than welcome.

Let’s move on.  The next post will cover the basics of setting up a functional development environment, one which will allow you to develop and instantly deploy your code to production so it is available to the rest of the world

Post 2 – The basics – Online hosting

August 19, 2012 Leave a comment

Well, I’m not a real expert at hosting solutions, but here’s what I’ve learned:

There are 4 possibilities when hosting online:

1. Buy cage space at any number of hosting spots.  They give you a cage, with electricity, network ,and air conditioning, and you have to do the rest (fill in hardware, cables, etc.).  You also get a service of a person who will go in and flip a power switch if you need it, so you don’t have to have someone on site).

I’ve been told that the costs start to balance out at around $1m, anything below that you should use the other options.  This is the cheapest solution if you have enough traffic and requirements to fill in the marginal cost, but is not for the average startup with $100k in the bank.

2. Use hosting solutions.  There are many, of all kinds, they vary in reliability and options, and are mid-range as far as costs are concerned.  A few months ago I would have said: use that if you want to get going as quick as possible, but now I know there are better options (see #4).  I guess the only reason to use that solution is if you are too big for option #3 (coming up), and too small for option #1.

3. Use PAAS such as EC2.  This is a very nice option, gives you endless scaling capabilities and easy deployment of (almost) any kind of environment, and best of all: you pay only for what you use, for as long as you use it.  You need 1000 Linux computers for the next 30 minutes?  setup one, duplicate 1000 times, and there you go.  Will cost you a couple of hundred bucks, but that’s it.  Amazing!  By the way, Amazon has free hosting for small environments so if you only want to play around, you don’t pay anything.  Haven’t played around with Azure, but I assume it works the same.

As far as I know this is the most expensive option, if you need an online solution which is constantly available and has high traffic and data requirements, I think you’ll balance out with option #2 in no time, and at some point will obviously get to #1 as well.

4. PAAS Level 2 (Cloudbees, Google apps engine, and the likes).  This is the best solution possible for startups at their early stages in my opinion.  I’ve been using Cloudbees for the past 2 months now, COMPLETELY FREE, and their solution rocks.  You get code hosting (I’m using GIT but you can also use SVN), as soon as you checkin your code and push it upstream to their repository (built into Eclipse if you use it), your code is copied over to Jenkins (I’ll get to CI in another post), and if all tests pass, your application is deployed in the cloud (on their app servers, I think it’s Tomcat).  You get access to a couple of DB’s (I’ve used MySQL and MongoDB) so you can save data, they have different monitoring solutions to complement, so in the end, you get a complete Continuous Deployment solution, with minimal effort and knowledge.

Just to be clear, I don’t work at Cloudbees, nor do I have any stocks, I just think what they did is absolutely phenomenal.  You can look at a youtube video which shows from zero to CD in 30 minutes using Cloudbees right here.  Check it out, it’s well worth your time.