Posts Tagged ‘Git’

git: undo all working dir changes including new files

January 17, 2013 Leave a comment

From : stack overflow

git reset --hard # removes staged and working directory changes
git clean -f -d # remove untracked files

Since this is so useful, I thought I’d post this for myself for reference and anyone else who needs it (you probably do if you use git)

Categories: R&D Tags: ,

Creating an online service – Using cloudbees for Continuous Deployment with Basic Service based projects

November 5, 2012 2 comments

The previous post discussed getting the basic service application to your local development environment and getting it setup.  In this post I’ll describe the process of setting up cloudbees for working with a Basic Service like project, so we can close the loop all the way to production.

We’ll be using cloudbees’ SEND-GRID services to send emails (registration verification process uses it to send a confirmation email), so before going further, go into cloudbees->services, and activate SendGrid.  You can pick the free account for now.

Before creating a new git repository in cloudbees, we need to do some housekeeping in Git.  Git has a file called .gitignore, which helps tell git what you don’t want to monitor and commit.  At the very least, add the following entry:


to the .gitignore file (create the file if it does not exist in your main project directory).  This will prevent you from accidentally checking in your targe directory (which is built automatically by the cloudbees CI environment).  Now that we have this covered, it’s time to get started.

Setting up a git repository in cloudbees is easy, just pick Repositories->create new repository, give it a name and pick Git as your repository type, and click enter.

Once that repository is done, you need to connect your local repository to the cloudbees one so you can push your code.  After creating your repository, you will see some helpful short examples of how to connect to cloudbees (just below the details paragraph).  Pick “Migrating to your cloudbees repository”, and follow the steps.  Don’t pull/push from legacy, just use the remote rm command to remove the old repository.  Alternatively, you can delete the .git directory and start a new git repository from scratch (without the basicservice history).  Just follow the “committing an existing project to git” cloudbees instructions to do that instead.

You repository should now be about 0.6MB, and contain the project files.

Once you’ve setup your repository, the next thing you need to do is setup a Build configuration.

Before going to builds, open your repository details, and copy the uri starting with ssh:// (you will need it shortly)

Hit builds, and create a New Job.

Give it a name, and pick “Build a maven 2/3 project”.  Hit next.

Mark “one click deployment”, select “Oracle 1.7 latest” as your JDK, unclick “restrict where this project can run”, select “git” as your repository.

In the git url paste the ssh:// url you copied earlier.  Put two stars (**) in the branches to build input field.

In build->goals and options put: clean install

In post steps, select “run only if build succeeds”, in the build settings, mark the email notification and add your email so you get notified of checkins and builds.

Pick “abort the build if it’s stuck” and pick whatever makes sense to you (I chose elastic)

Add a post-build-action and deploy-to-cloudbees, add web application and pick “maven artifact selection”.  Add an application id (your project name is fine).  Click “Advanced”, and enter production in the “Environment” field.

Don’t worry about the “not found” warning, click save.

Click build Now, and when an available worker is ready your project will start building. Click the running build and then console Output to see the process in action.  When the build is complete, you should see something like:

[cloudbees-deployer] Deployed to application id your_account/your_project
[cloudbees-deployer] Can be accessed at

Click it and start playing around with your production online service!

One last small thing left to do: setup our domain.  We’ll cover this in the next post.

Categories: R&D Tags: , , , ,

Creating an online service – don’t start from scratch!

November 5, 2012 4 comments

The previous post discussed setting up our development environment, now that we got that covered, it’s time to move on to the main event.

A word about open source before I begin:  in the past couple of months I’ve been exposed to more open source code than I have been for the past 20 years of being a developer.  This unbelievable amount of freely available software and information has saved and will save me years of development and research, and is the real true enabler to what I am about to do.  All code written by me and presented here is GPL, which means you can copy/modify/improve it at will, with no limitations what-so-ever (and no warrantees, read the GPL agreement please).  Some small parts of the code had been copied from internet sources, and there’s even one case where a different license applies (BSD, another open license), I’ve added that distinction to code which has been copied in whole and needs that different license (actually only one place in the javascript code).  If you believe any parts of the code presented here are under a different license or should not be used, please let me know and I will take actions to resolve the problem, but all in all this should all be GPL free, as most of it has been written by your truly.

To get started, go to basic server google page and clone my repository from there.  Using git, this should be as simple as typing in console:

git clone your_project_name

Note that even though a password may be requested, you are using a read-only pull and can just click enter.

From here on, I’ll assume you are using Eclipse as your IDE, if you’re using Idea things may be a little bit different but the general idea (pun intended) is the same.

Once you get the code to your work environment, use Eclipse->File->Import->Existing maven project, browse and choose the skeleton directory (on my system it’s /Volumes/srcvault/your_project_name/), and import the project to your workspace.

You’ll now have to make some changes to make this project your own.

Let’s take care of the settings and file system changes: 

I use case-sensitive search-replace on the file system starting with the main project directory and including sub-directories.  In my opinion it’s the fastest and safest option (TextWrangler does this perfectly on my Mac OSX). Replace all instances of basicservice and basic-service, with capital letters and without.  After doing that the project won’t compile because the package directories are still basicservice, so remember to rename them manually after you do the search-replace.

At this point, your code should compile, but there’s still a couple of things to do.  Before moving on, go to src/test/java, right click it and run as->JUnit test.  If everything is green, you’ve successfully cloned and migrated the code to your project.

Next go to, and change the to yourprojectname_db.  This is the mongo db you will be using locally, so remember it if you want to access it later (you can also see all the db’s available so this is not a big deal).

Open, login to and create a new Mongo DB repository in Cloudbees and update the relevant production information which you’ll find in Cloudbees after you create the new Mongo DB repository (you should know how to do that by now if you’ve followed the recommended youtube movie from the previous post).  Don’t forget to add a new mongo-db user and update that information as well.

After everything compiles and all data has been filled in, we have one last thing we must do:

Next we need to make some changes to the keys used by ESAPI for encryption (in

Note: you MUST set these keys or else your site’s security will be compromised!  Do not use Basic Service’s keys as EVERYONE has them.

Generate  (and assign) the following two keys by using Utils.generateESAPIKeys().  To do that, you can create a simple main method in Utils, which looks like this:

public static void main(String[] args) throws Exception {

Right click Utils, and run as java application.  When you run it, you’ll see an AppSensor exception.

This happens because ESAPI is not configured properly for this project.  Go to Run->Run configurations, click on argument, and in VM arguments, add the following:


* NOTE:  because wordpress changes regular quotes into styled quotes in this post, you can’t simply copy the above line.  After you copy and paste this line in eclipse, you need to manually delete the orange styled quotes and replace them with regular double quotes or this will not work.

Now try running again, and you should see the following output:

Attempting to load via file I/O.
Attempting to load as resource file via file I/O.
Found in ‘org.owasp.esapi.resources’ directory: /<the path to your project>/your_project_name/src/main/webapp/WEB-INF/esapi/
Loaded ‘’ properties file
Generating a new secret master key
Copy the two keys to instead of the existing Basic Service keys.

Last, to run the actual service, you need to create a Maven Build configuration, and add the ESAPI configuration there as well.

Go to Maven Build (click Run->Debug configurations and go to Maven Build) and create a new configuration.  Change the name to your project name, and add the following:

  • In Main:
    • In base directory, enter: ${workspace_loc:/your_project_name}
    • In Goals, enter: install jetty:run
    • In profiles, enter: development
    • In parameters (right above Maven Runtime), click [Add…] and add the following:
      • Name:org.owasp.esapi.resources
      • Value:target/your_project_name/WEB-INF/esapi/
  • In refresh, make sure the “Refresh resources upon completion” checkbox is checked, and the “The entire workspace” radio button is selected.
  • In source, click [Add…]->Project, and add your project (so you can debug your code).  You might also want to add the ESAPI and AppSensor source jars so you can debug those as well if you have to.
  • In environment, add the following:
    • value:na
    • value:na
    • value:na
    • value:development

Click the DEBUG button, and if all went well, you should see a lot of debug prints, and eventually: [INFO] Started Jetty Server

Go to localhost:8080 and watch your new service in action.  If you forgot to run MongoDB locally, you will see an exception when trying to login, since your service can’t connect to a database.

Either run mongo locally, or change the development properties to point to your production db (remember to change it later, working in dev on production db is a big NoNo!)

Now that we have everything setup locally, it’s time to get this service to production.

The next post will cover that part in details, although you should have a general idea of what’s needed if you’ve followed the recommended youtube movie.

Creating an online service – getting started

November 5, 2012 4 comments

In the previous post of this series, I talked about the different technologies we’ll be using and the sample project which is freely available, in this post we’ll cover the basics of setting up a functional development environment, one which will allow you to develop and instantly deploy your code to production so it is available to the rest of the world.

I think that the hardest thing about starting a new service for the first time is, well, getting started.  If you’re a seasoned developer, you’ve done a couple of things in your life, maybe even led some projects, you feel you can do anything.  But when starting out on your own, you are faced with a challenge you haven’t faced before.  In most cases, even as a fifth or sixth developer at a startup, there’s a skeleton on which you work.  Somebody already setup the framework, built the necessary scripts, setup the build/deploy process, created the minimal classes.  Your job is now to add the functionality, possibly adding new infrastructure that was never there, but most times there’s something to work on top of.

For a completely new service there’s nothing.  You need to download the IDE, set it up, configure your application from scratch, deploy it somewhere (locally or all the way to production), and make it work.

You can check out my post on some of the tools we’ll use just to get a general idea of the different stuff you might need to install at different stages of your project.  One thing to notice, all of the tools I’m going to use in this series of posts are free, open source tools, or those that are free for use by small startups which are just getting started.  You might need to pay for it at some point when your project grows, but not right now.

You’ll need to do the basic stuff of getting an OS (virtual or real, preferably Unix based, I use Mac OSX, but Ubunto is a good choice for development as well), installing JAVA, a basic text editor, and some kind of image editing software (you can get adobe photoshop for a free one month trial, more than you’ll ever need for starting out).  I use Chrome for all my browsing, if you’re on Windows you might still be using IE, but as it is no longer the most popular browser, I suggest switching to Chrome for the initial testing stages of your work.

So once you have all those setup, you’ll need an IDE.   You can checkout my short post on choosing an IDE, I personally believe that for a startup Eclipse is more than enough.  I use “Eclipse Java EE IDE for Web Developers.”, you should get some plugins installed, at least EGit (git support) and m2e (maven support).

Next step is to get source control software, I use GIT and the Basic Service project is hosted in a GIT repository on Google code, but Subversion is a decent choice as well.  After that, install Maven, you’ll use it extensively.  Last thing you will need to go through this tutorial is SSH, this will be used for deploying your work to production (cloudbees uses SSH for its repository communication).  Note that I’m using Cloudbees as the git repository, it’s also possible to use GitHub and run from there but we won’t cover that right now.

So now you can checkout stuff, and open it in your IDE, that’s cool.  You might be tempted to install an application server like Tomcat, and that’s ok, but it’s not really needed.  Jetty is a very good application server which you can use directly from your IDE and it will automatically update your project as the source changes, and you can run it using project configuration only (maven FTW).  We’ll get to that later, for now, let’s leave Tomcat out of the equation.

At this point, the best thing for you to do, is watch a video!  Not just any video, you should watch this one.  This is one of the most useful tutorials I’ve ever watched, it will take you from (almost) zero (where you are at right now or at least where I was when I started) to 100 (a fully operational, continuously deployed project) in 30 minutes (actually 26).

Come back when you’re done (meaning you followed the instructions, setup everything, built and deployed it to production, and it works for you).

So now you’re a couple of steps further than you’ve been when you started reading.  You already have your work environment setup, you’re compiling and deploying an app to production, you even have it saving stuff to a production repository.

Next you’ll want to setup your local environment so you can test stuff locally.  You’re almost ready, just need to install MongoDB so you can point your application to your local DB instance (it will also work against the instance on Cloudbees if you prefer, but I’d rather have my local instance which I can play with and explore).  So once that’s installed, try to get it running and point your app to use the local instance.  When that’s done, if you’re on a laptop like me, it means you can work from your favorite spot (on the beach?) and not require an internet connection to develop stuff.

Great, we’re ready to move on.  The next post will talk about getting the Basic Service project over to your development environment, and setting it up so you play around with it and learn the technology involved, or even start developing your own online service.

Copying a git repository from Cloudbees to Github

October 16, 2012 Leave a comment

I recently had to take my cloudbees hosted repository hosted on, well, cloudbees, and move it to github.  This was a bit tricky for someone will little knowledge of ssh and git (me), so hopefully if you’re in my situation this will help.  Note that this was done on a Mac OSX, should be similar for other unix based OS’s but probably slightly different for Windows.

I’ve used green color to represent terminal commands, so if you see something in green, it obviously needs to go into terminal…

First create a new account and repository on github.

Once you do that, you need to setup your ssh connection to git (you can also connect using http, but I’ll stick to SSH for now).  There’s a pretty good explanation by git-hub here, which I followed (well, some of it), but basically here’s what you need to do:

  1. Go to command terminal, and enter the following command which will copy your ssh public key to the clipboard
    pbcopy < ~/.ssh/
  2. If was not found, you need to generate a new ssh key locally, follow the git-hub tutorial mentioned above to do that.
  3. Go to your github account settings->ssh keys, and click “add SSH key”.  Give it a name and paste your key.
  4. In terminal, enter
    ssh-add -l
  5. This should give you a print of your key hash, make sure it is identical to what you see in your git-hub listing.  It should look like this:
  6. In terminal, go to your repository directory (e.g. ~/code/basicservice/)
  7. make sure you have a git repository there:
    git status
  8. Create a new repository on git-hub, don’t initialize it with README
  9. Push your repository to git-hub
    git push<youraccount>/<your-repository>.git master
    you can get the git url from your git repository page, make sure you click on SSH to get the right url, or just replace <youraccount>/<your-repository> with your data.
  10. That’s it.  Your repository is now hosted on git-hub as well, you can now drop the cloudbees repo and start using git-hub.  To make it push both to cloudbees and to github simultaneously (if you want to keep both) you need to add some configuration, and I’ll pass on that right now.  You can check out this post for help on that.
Categories: R&D Tags: , , , ,

Post 4 – The basics – Version control:Git vs. SVN

August 19, 2012 Leave a comment

I don’t want to open a comment war here, so I won’t say which is best, I’ll only say that I’ve used Subversion for the past 9 years, and it is a very good solution, but I’m using Git for my current adventure, and I’ll tell you why.

SVN is a service based solution, you need to install and maintain a central service if you want to store your code, and versions are stored on the hosting server.  So if want to be able to check diffs or check-in your code, you need to have access to the hosting service.

Git works differently, Git stores all your history locally in every repository, and every repository is a full clone of all data available (current and past code).  This means you can do checkins and diffs locally without access to a centralized service, meaning you can work offline, and if you want to do diffs it’s as fast as it can be since it’s all local disk access.

I’m no expert on neither solutions, but in my humble opinion Git suits smaller teams who need to work in separate locations, and want to be able to do many quick checkins per day, whereas SVN suits a centralized team where the server is hosted in a LAN and history searches and compare can benefit from LAN speed.  I know there’s also a difference in source control strategy between the two but I haven’t researched it enough to comment on it.

As far as online repositories are concerned, there’s Github if you want a centralized Git repository which everyone can sync to, and I’m sure you can find a good repository for Subversion or do it yourself in EC2.

Installing Git is as simple as a download, and SVN (service) is a little harder but manageable.  SVN client (tortoisesvn) is VERY good, there are good git clients out there from what I’ve heard, but I get all I need within Eclipse so I haven’t even tried to find one.

Bottom line: I chose Git because it fits my current needs, moving to SVN later is a feasible option if I ever feel the need.