Archive

Posts Tagged ‘Mongo DB’

Creating an online service – don’t start from scratch!

November 5, 2012 4 comments

The previous post discussed setting up our development environment, now that we got that covered, it’s time to move on to the main event.

A word about open source before I begin:  in the past couple of months I’ve been exposed to more open source code than I have been for the past 20 years of being a developer.  This unbelievable amount of freely available software and information has saved and will save me years of development and research, and is the real true enabler to what I am about to do.  All code written by me and presented here is GPL, which means you can copy/modify/improve it at will, with no limitations what-so-ever (and no warrantees, read the GPL agreement please).  Some small parts of the code had been copied from internet sources, and there’s even one case where a different license applies (BSD, another open license), I’ve added that distinction to code which has been copied in whole and needs that different license (actually only one place in the javascript code).  If you believe any parts of the code presented here are under a different license or should not be used, please let me know and I will take actions to resolve the problem, but all in all this should all be GPL free, as most of it has been written by your truly.

To get started, go to basic server google page and clone my repository from there.  Using git, this should be as simple as typing in console:

git clone https://thezuck@code.google.com/p/basic-service/ your_project_name

Note that even though a password may be requested, you are using a read-only pull and can just click enter.

From here on, I’ll assume you are using Eclipse as your IDE, if you’re using Idea things may be a little bit different but the general idea (pun intended) is the same.

Once you get the code to your work environment, use Eclipse->File->Import->Existing maven project, browse and choose the skeleton directory (on my system it’s /Volumes/srcvault/your_project_name/), and import the project to your workspace.

You’ll now have to make some changes to make this project your own.

Let’s take care of the settings and file system changes: 

I use case-sensitive search-replace on the file system starting with the main project directory and including sub-directories.  In my opinion it’s the fastest and safest option (TextWrangler does this perfectly on my Mac OSX). Replace all instances of basicservice and basic-service, with capital letters and without.  After doing that the project won’t compile because the package directories are still basicservice, so remember to rename them manually after you do the search-replace.

At this point, your code should compile, but there’s still a couple of things to do.  Before moving on, go to src/test/java, right click it and run as->JUnit test.  If everything is green, you’ve successfully cloned and migrated the code to your project.

Next go to development.spring.properties, and change the mongo.db.name to yourprojectname_db.  This is the mongo db you will be using locally, so remember it if you want to access it later (you can also see all the db’s available so this is not a big deal).

Open production.spring.properties, login to cloudbees.com and create a new Mongo DB repository in Cloudbees and update the relevant production information which you’ll find in Cloudbees after you create the new Mongo DB repository (you should know how to do that by now if you’ve followed the recommended youtube movie from the previous post).  Don’t forget to add a new mongo-db user and update that information as well.

After everything compiles and all data has been filled in, we have one last thing we must do:

Next we need to make some changes to the keys used by ESAPI for encryption (in ESAPI.properties).

Note: you MUST set these keys or else your site’s security will be compromised!  Do not use Basic Service’s keys as EVERYONE has them.

Generate  (and assign) the following two keys by using Utils.generateESAPIKeys().  To do that, you can create a simple main method in Utils, which looks like this:

public static void main(String[] args) throws Exception {
generateESAPIKeys();
}

Right click Utils, and run as java application.  When you run it, you’ll see an AppSensor exception.

This happens because ESAPI is not configured properly for this project.  Go to Run->Run configurations, click on argument, and in VM arguments, add the following:

-Dorg.owasp.esapi.resources=src/main/webapp/WEB-INF/esapi

* NOTE:  because wordpress changes regular quotes into styled quotes in this post, you can’t simply copy the above line.  After you copy and paste this line in eclipse, you need to manually delete the orange styled quotes and replace them with regular double quotes or this will not work.

Now try running again, and you should see the following output:

Attempting to load ESAPI.properties via file I/O.
Attempting to load ESAPI.properties as resource file via file I/O.
Found in ‘org.owasp.esapi.resources’ directory: /<the path to your project>/your_project_name/src/main/webapp/WEB-INF/esapi/ESAPI.properties
Loaded ‘ESAPI.properties’ properties file
Generating a new secret master key
#==============================================================
Encryptor.MasterKey=kdP+LK4OIIAdWDc+heyN3E==
Encryptor.MasterSalt=hgR9kW/bQWlGGD4JHESwnUY35a2=
#==============================================================
 
Copy the two keys to ESAPI.properties instead of the existing Basic Service keys.

Last, to run the actual service, you need to create a Maven Build configuration, and add the ESAPI configuration there as well.

Go to Maven Build (click Run->Debug configurations and go to Maven Build) and create a new configuration.  Change the name to your project name, and add the following:

  • In Main:
    • In base directory, enter: ${workspace_loc:/your_project_name}
    • In Goals, enter: install jetty:run
    • In profiles, enter: development
    • In parameters (right above Maven Runtime), click [Add…] and add the following:
      • Name:org.owasp.esapi.resources
      • Value:target/your_project_name/WEB-INF/esapi/
  • In refresh, make sure the “Refresh resources upon completion” checkbox is checked, and the “The entire workspace” radio button is selected.
  • In source, click [Add…]->Project, and add your project (so you can debug your code).  You might also want to add the ESAPI and AppSensor source jars so you can debug those as well if you have to.
  • In environment, add the following:
    • name:SENDGRID_PASSWORD
    • value:na
    • name:SENDGRID_SMTP_HOST
    • value:na
    • name:SENDGRID_USERNAME
    • value:na
    • name:spring.profiles.active
    • value:development

Click the DEBUG button, and if all went well, you should see a lot of debug prints, and eventually: [INFO] Started Jetty Server

Go to localhost:8080 and watch your new service in action.  If you forgot to run MongoDB locally, you will see an exception when trying to login, since your service can’t connect to a database.

Either run mongo locally, or change the development properties to point to your production db (remember to change it later, working in dev on production db is a big NoNo!)

Now that we have everything setup locally, it’s time to get this service to production.

The next post will cover that part in details, although you should have a general idea of what’s needed if you’ve followed the recommended youtube movie.

Creating an online service – getting started

November 5, 2012 4 comments

In the previous post of this series, I talked about the different technologies we’ll be using and the sample project which is freely available, in this post we’ll cover the basics of setting up a functional development environment, one which will allow you to develop and instantly deploy your code to production so it is available to the rest of the world.

I think that the hardest thing about starting a new service for the first time is, well, getting started.  If you’re a seasoned developer, you’ve done a couple of things in your life, maybe even led some projects, you feel you can do anything.  But when starting out on your own, you are faced with a challenge you haven’t faced before.  In most cases, even as a fifth or sixth developer at a startup, there’s a skeleton on which you work.  Somebody already setup the framework, built the necessary scripts, setup the build/deploy process, created the minimal classes.  Your job is now to add the functionality, possibly adding new infrastructure that was never there, but most times there’s something to work on top of.

For a completely new service there’s nothing.  You need to download the IDE, set it up, configure your application from scratch, deploy it somewhere (locally or all the way to production), and make it work.

You can check out my post on some of the tools we’ll use just to get a general idea of the different stuff you might need to install at different stages of your project.  One thing to notice, all of the tools I’m going to use in this series of posts are free, open source tools, or those that are free for use by small startups which are just getting started.  You might need to pay for it at some point when your project grows, but not right now.

You’ll need to do the basic stuff of getting an OS (virtual or real, preferably Unix based, I use Mac OSX, but Ubunto is a good choice for development as well), installing JAVA, a basic text editor, and some kind of image editing software (you can get adobe photoshop for a free one month trial, more than you’ll ever need for starting out).  I use Chrome for all my browsing, if you’re on Windows you might still be using IE, but as it is no longer the most popular browser, I suggest switching to Chrome for the initial testing stages of your work.

So once you have all those setup, you’ll need an IDE.   You can checkout my short post on choosing an IDE, I personally believe that for a startup Eclipse is more than enough.  I use “Eclipse Java EE IDE for Web Developers.”, you should get some plugins installed, at least EGit (git support) and m2e (maven support).

Next step is to get source control software, I use GIT and the Basic Service project is hosted in a GIT repository on Google code, but Subversion is a decent choice as well.  After that, install Maven, you’ll use it extensively.  Last thing you will need to go through this tutorial is SSH, this will be used for deploying your work to production (cloudbees uses SSH for its repository communication).  Note that I’m using Cloudbees as the git repository, it’s also possible to use GitHub and run from there but we won’t cover that right now.

So now you can checkout stuff, and open it in your IDE, that’s cool.  You might be tempted to install an application server like Tomcat, and that’s ok, but it’s not really needed.  Jetty is a very good application server which you can use directly from your IDE and it will automatically update your project as the source changes, and you can run it using project configuration only (maven FTW).  We’ll get to that later, for now, let’s leave Tomcat out of the equation.

At this point, the best thing for you to do, is watch a video!  Not just any video, you should watch this one.  This is one of the most useful tutorials I’ve ever watched, it will take you from (almost) zero (where you are at right now or at least where I was when I started) to 100 (a fully operational, continuously deployed project) in 30 minutes (actually 26).

Come back when you’re done (meaning you followed the instructions, setup everything, built and deployed it to production, and it works for you).

So now you’re a couple of steps further than you’ve been when you started reading.  You already have your work environment setup, you’re compiling and deploying an app to production, you even have it saving stuff to a production repository.

Next you’ll want to setup your local environment so you can test stuff locally.  You’re almost ready, just need to install MongoDB so you can point your application to your local DB instance (it will also work against the instance on Cloudbees if you prefer, but I’d rather have my local instance which I can play with and explore).  So once that’s installed, try to get it running and point your app to use the local instance.  When that’s done, if you’re on a laptop like me, it means you can work from your favorite spot (on the beach?) and not require an internet connection to develop stuff.

Great, we’re ready to move on.  The next post will talk about getting the Basic Service project over to your development environment, and setting it up so you play around with it and learn the technology involved, or even start developing your own online service.

Online service development in java – a beginner’s guide

November 4, 2012 1 comment

In the coming series of posts, I will share my development experience (code included) with creating an online service from scratch.  If you read through all posts, you should reach a point where you are capable of writing code and deploying it to a production like environment which is available online, using some of the latest (and greatest) open source libraries available in java.  You will definitely not be at the end of the journey, but will get a pretty good head start.

I’ve created a sample project, which basically does nothing (well, almost nothing), but it does so in a (semi) secure way, using several top libraries.  Basic service, is a service which lets you register, login, and… do nothing.  But, it does so using:

Spring, Maven, ESAPI (+AppSensor) for input validation, JSP’s (JSTL+Tiles 2), jQuery (+UI), Mongo DB, Logback, Jackson, Mockito (+PowerMock) for unit testing, and it does it in a RESTful way, using complete Data and Presentation separation (all JSP’s get their data through REST requests, meaning you can add easily add mobile support without almost any change to the  interface), and is fully internationalized (currently supports hebrew and english, meaning we have RTL languages covered).  The complete source code for Basic service is available on google code, it is a fully functional (and GPL open source) example of combining all of these technologies , which means that you can concentrate on the next steps rather than spend time building the foundation from scratch.  It took me about 3 months to go through all the baby steps in each of those technologies (after spending some time picking the right ones of course), I hope this series will save you at least some of that time.

Please keep in mind that I do not claim this to be a complete and bullet proof piece of software, there is still much to do in order to make it “production ready”.

Basic service is deployed on Cloudbees, using Continuous Deployment,  you’ll get some insights on how to do that as well.

One final note: the title says “a beginner’s guide”, but should be read with two different interpretations: one is (possibly) you, who is just getting started with creating an online service, and the other is me, sharing my experience with jumping into these deep water a couple of months ago.  If you like what you read, you’re welcome to comment, if you see me making horrible mistakes, constructive criticism is more than welcome.

Let’s move on.  The next post will cover the basics of setting up a functional development environment, one which will allow you to develop and instantly deploy your code to production so it is available to the rest of the world