Archive

Posts Tagged ‘Eclipse’

Connecting to an existing Cloudbees repository from Windows

December 29, 2012 Leave a comment

This post is the result of trying to figure out how to setup a repository clone of one of my projects on my windows machine.  I work on my Mac most of the time, but at times it’s more convenient to work on my Win machine (usually when I have to debug IE issues, and need to try js fixes to see what impact they make)

So, lets start by setting up our tools and IDE on Windows.  We obviously need Java, Eclipse, and Maven, and actually that’s it.  In Eclipse, install m2e and EGit plugins.  Make sure you follow all the installation instructions for Maven with all the variables in the right place.

Now we need to setup SSH.  This needs to happen in two places, in Eclipse settings and in Cloudbees settings.  Here are the steps:

  1. In Eclipse, go to preferences->General->Network Connections->SSH2, select key management, and generate a DSA key.  Enter the pass phrase and remember it, you’ll need it in a minute.  Copy the public key to the clipboard.
  2. Click “Save private key” and save it, go to general and add the saved private key.   Apply and save.
  3. Go into Cloudbees settings->SSH keys, choose a name and paste the public key from the clipboard.  Click “Add”.
  4. Go to Cloudbees->Repositories, go into your project and copy to clipboard the SSH url unique to your repository.
  5. Go back to Eclipse, File->import, choose Git->Projects from Git.  Choose URI and paste the SSH URL you copied from your repository.  Choose ssh as the protocol.  The user should be git, and password empty.  Port can remain empty (it will be filled with the default which is 22)
  6. You will now need to enter your pass phrase.
  7. Next next next and your project will be cloned locally.
  8. To help maven work you need to go into Eclipse->Preferences->Maven, and go to installation->Add and pick the Maven installation you installed locally (instead of the Embedded one).
  9. If you run into problems with your project when you open the IDE, see this post.  If you still have problems, and you had Maven installed before, try deleting the .m2 resources directory.  You might also need to go into Preferences->Maven->User settings and reindex or update settings.
  10. Your project should automatically compile.

Problems you may encounter:

1. M2E installation fails and maven does not seem integrated into eclipse.  I couldn’t fix that and had to reinstall eclipse

2. You get complaints about javac and JRE: you’re using JRE instead of JDK, switch to JDK in eclipse->java->installed jdk’s

Categories: R&D Tags: , , , , , , ,

Creating an online service – don’t start from scratch!

November 5, 2012 4 comments

The previous post discussed setting up our development environment, now that we got that covered, it’s time to move on to the main event.

A word about open source before I begin:  in the past couple of months I’ve been exposed to more open source code than I have been for the past 20 years of being a developer.  This unbelievable amount of freely available software and information has saved and will save me years of development and research, and is the real true enabler to what I am about to do.  All code written by me and presented here is GPL, which means you can copy/modify/improve it at will, with no limitations what-so-ever (and no warrantees, read the GPL agreement please).  Some small parts of the code had been copied from internet sources, and there’s even one case where a different license applies (BSD, another open license), I’ve added that distinction to code which has been copied in whole and needs that different license (actually only one place in the javascript code).  If you believe any parts of the code presented here are under a different license or should not be used, please let me know and I will take actions to resolve the problem, but all in all this should all be GPL free, as most of it has been written by your truly.

To get started, go to basic server google page and clone my repository from there.  Using git, this should be as simple as typing in console:

git clone https://thezuck@code.google.com/p/basic-service/ your_project_name

Note that even though a password may be requested, you are using a read-only pull and can just click enter.

From here on, I’ll assume you are using Eclipse as your IDE, if you’re using Idea things may be a little bit different but the general idea (pun intended) is the same.

Once you get the code to your work environment, use Eclipse->File->Import->Existing maven project, browse and choose the skeleton directory (on my system it’s /Volumes/srcvault/your_project_name/), and import the project to your workspace.

You’ll now have to make some changes to make this project your own.

Let’s take care of the settings and file system changes: 

I use case-sensitive search-replace on the file system starting with the main project directory and including sub-directories.  In my opinion it’s the fastest and safest option (TextWrangler does this perfectly on my Mac OSX). Replace all instances of basicservice and basic-service, with capital letters and without.  After doing that the project won’t compile because the package directories are still basicservice, so remember to rename them manually after you do the search-replace.

At this point, your code should compile, but there’s still a couple of things to do.  Before moving on, go to src/test/java, right click it and run as->JUnit test.  If everything is green, you’ve successfully cloned and migrated the code to your project.

Next go to development.spring.properties, and change the mongo.db.name to yourprojectname_db.  This is the mongo db you will be using locally, so remember it if you want to access it later (you can also see all the db’s available so this is not a big deal).

Open production.spring.properties, login to cloudbees.com and create a new Mongo DB repository in Cloudbees and update the relevant production information which you’ll find in Cloudbees after you create the new Mongo DB repository (you should know how to do that by now if you’ve followed the recommended youtube movie from the previous post).  Don’t forget to add a new mongo-db user and update that information as well.

After everything compiles and all data has been filled in, we have one last thing we must do:

Next we need to make some changes to the keys used by ESAPI for encryption (in ESAPI.properties).

Note: you MUST set these keys or else your site’s security will be compromised!  Do not use Basic Service’s keys as EVERYONE has them.

Generate  (and assign) the following two keys by using Utils.generateESAPIKeys().  To do that, you can create a simple main method in Utils, which looks like this:

public static void main(String[] args) throws Exception {
generateESAPIKeys();
}

Right click Utils, and run as java application.  When you run it, you’ll see an AppSensor exception.

This happens because ESAPI is not configured properly for this project.  Go to Run->Run configurations, click on argument, and in VM arguments, add the following:

-Dorg.owasp.esapi.resources=src/main/webapp/WEB-INF/esapi

* NOTE:  because wordpress changes regular quotes into styled quotes in this post, you can’t simply copy the above line.  After you copy and paste this line in eclipse, you need to manually delete the orange styled quotes and replace them with regular double quotes or this will not work.

Now try running again, and you should see the following output:

Attempting to load ESAPI.properties via file I/O.
Attempting to load ESAPI.properties as resource file via file I/O.
Found in ‘org.owasp.esapi.resources’ directory: /<the path to your project>/your_project_name/src/main/webapp/WEB-INF/esapi/ESAPI.properties
Loaded ‘ESAPI.properties’ properties file
Generating a new secret master key
#==============================================================
Encryptor.MasterKey=kdP+LK4OIIAdWDc+heyN3E==
Encryptor.MasterSalt=hgR9kW/bQWlGGD4JHESwnUY35a2=
#==============================================================
 
Copy the two keys to ESAPI.properties instead of the existing Basic Service keys.

Last, to run the actual service, you need to create a Maven Build configuration, and add the ESAPI configuration there as well.

Go to Maven Build (click Run->Debug configurations and go to Maven Build) and create a new configuration.  Change the name to your project name, and add the following:

  • In Main:
    • In base directory, enter: ${workspace_loc:/your_project_name}
    • In Goals, enter: install jetty:run
    • In profiles, enter: development
    • In parameters (right above Maven Runtime), click [Add…] and add the following:
      • Name:org.owasp.esapi.resources
      • Value:target/your_project_name/WEB-INF/esapi/
  • In refresh, make sure the “Refresh resources upon completion” checkbox is checked, and the “The entire workspace” radio button is selected.
  • In source, click [Add…]->Project, and add your project (so you can debug your code).  You might also want to add the ESAPI and AppSensor source jars so you can debug those as well if you have to.
  • In environment, add the following:
    • name:SENDGRID_PASSWORD
    • value:na
    • name:SENDGRID_SMTP_HOST
    • value:na
    • name:SENDGRID_USERNAME
    • value:na
    • name:spring.profiles.active
    • value:development

Click the DEBUG button, and if all went well, you should see a lot of debug prints, and eventually: [INFO] Started Jetty Server

Go to localhost:8080 and watch your new service in action.  If you forgot to run MongoDB locally, you will see an exception when trying to login, since your service can’t connect to a database.

Either run mongo locally, or change the development properties to point to your production db (remember to change it later, working in dev on production db is a big NoNo!)

Now that we have everything setup locally, it’s time to get this service to production.

The next post will cover that part in details, although you should have a general idea of what’s needed if you’ve followed the recommended youtube movie.

Creating an online service – getting started

November 5, 2012 4 comments

In the previous post of this series, I talked about the different technologies we’ll be using and the sample project which is freely available, in this post we’ll cover the basics of setting up a functional development environment, one which will allow you to develop and instantly deploy your code to production so it is available to the rest of the world.

I think that the hardest thing about starting a new service for the first time is, well, getting started.  If you’re a seasoned developer, you’ve done a couple of things in your life, maybe even led some projects, you feel you can do anything.  But when starting out on your own, you are faced with a challenge you haven’t faced before.  In most cases, even as a fifth or sixth developer at a startup, there’s a skeleton on which you work.  Somebody already setup the framework, built the necessary scripts, setup the build/deploy process, created the minimal classes.  Your job is now to add the functionality, possibly adding new infrastructure that was never there, but most times there’s something to work on top of.

For a completely new service there’s nothing.  You need to download the IDE, set it up, configure your application from scratch, deploy it somewhere (locally or all the way to production), and make it work.

You can check out my post on some of the tools we’ll use just to get a general idea of the different stuff you might need to install at different stages of your project.  One thing to notice, all of the tools I’m going to use in this series of posts are free, open source tools, or those that are free for use by small startups which are just getting started.  You might need to pay for it at some point when your project grows, but not right now.

You’ll need to do the basic stuff of getting an OS (virtual or real, preferably Unix based, I use Mac OSX, but Ubunto is a good choice for development as well), installing JAVA, a basic text editor, and some kind of image editing software (you can get adobe photoshop for a free one month trial, more than you’ll ever need for starting out).  I use Chrome for all my browsing, if you’re on Windows you might still be using IE, but as it is no longer the most popular browser, I suggest switching to Chrome for the initial testing stages of your work.

So once you have all those setup, you’ll need an IDE.   You can checkout my short post on choosing an IDE, I personally believe that for a startup Eclipse is more than enough.  I use “Eclipse Java EE IDE for Web Developers.”, you should get some plugins installed, at least EGit (git support) and m2e (maven support).

Next step is to get source control software, I use GIT and the Basic Service project is hosted in a GIT repository on Google code, but Subversion is a decent choice as well.  After that, install Maven, you’ll use it extensively.  Last thing you will need to go through this tutorial is SSH, this will be used for deploying your work to production (cloudbees uses SSH for its repository communication).  Note that I’m using Cloudbees as the git repository, it’s also possible to use GitHub and run from there but we won’t cover that right now.

So now you can checkout stuff, and open it in your IDE, that’s cool.  You might be tempted to install an application server like Tomcat, and that’s ok, but it’s not really needed.  Jetty is a very good application server which you can use directly from your IDE and it will automatically update your project as the source changes, and you can run it using project configuration only (maven FTW).  We’ll get to that later, for now, let’s leave Tomcat out of the equation.

At this point, the best thing for you to do, is watch a video!  Not just any video, you should watch this one.  This is one of the most useful tutorials I’ve ever watched, it will take you from (almost) zero (where you are at right now or at least where I was when I started) to 100 (a fully operational, continuously deployed project) in 30 minutes (actually 26).

Come back when you’re done (meaning you followed the instructions, setup everything, built and deployed it to production, and it works for you).

So now you’re a couple of steps further than you’ve been when you started reading.  You already have your work environment setup, you’re compiling and deploying an app to production, you even have it saving stuff to a production repository.

Next you’ll want to setup your local environment so you can test stuff locally.  You’re almost ready, just need to install MongoDB so you can point your application to your local DB instance (it will also work against the instance on Cloudbees if you prefer, but I’d rather have my local instance which I can play with and explore).  So once that’s installed, try to get it running and point your app to use the local instance.  When that’s done, if you’re on a laptop like me, it means you can work from your favorite spot (on the beach?) and not require an internet connection to develop stuff.

Great, we’re ready to move on.  The next post will talk about getting the Basic Service project over to your development environment, and setting it up so you play around with it and learn the technology involved, or even start developing your own online service.

Another one of those irritating “you can’t do this but I won’t tell you why” in eclipse

November 4, 2012 Leave a comment

I decided to clone my base project (basic service) to start a new project, so I used git to fetch it to my local drive, then tried to import to my workspace using file->import->maven->existing maven projects.  Picked my new cloned project, and indeed the POM.xml file appeared, but the checkbox next to it was unclickable!  Of course eclipse won’t tell you why, so after searching the web and trial and error, I finally figured it out:

If you already have a project by the name specified in the POM, you can’t create a new one with that name from the POM.xml

In order to get through this issue, you need to add a suffix in the Advanced->Name template field (just adding “clone” was enough for me) or change it in the POM.

HTH

Eclipse quirks #1 – sometimes the project just stops doing what it should

October 17, 2012 1 comment

This happened to me today:  I added google Guava to my pom (for using their ImmutableSet collection), maven imported the jar correctly, but for some reason Eclipse refused to show it in the maven dependencies list, and the ImmutableSet import did not work (would not show me the option to add a java import statement, and adding it manually did not find the google library).

The reason for this eludes me, so can’t solve the bug, but I do know a quick work around:

  1. close Eclipse
  2. go to the project directory
  3. in terminal, enter:
    mvn eclipse:clean
  4. do NOT run mvn eclipse:eclipse as this will create an eclipse project which has a weird structure in eclipse, instead, do the following:
  5. re-run eclipse, go to file->import, select maven->existing maven project, choose your project directory, eclipse should recognize your pom, click finish, and voilla, your project will start working again.

This will probably work for any unclear state the project goes into in eclipse, not just import issues.

Post 5 – The basics – Software for SAAS development

August 20, 2012 1 comment

The following is a list of programs I’ve played with.  It is by no means a complete list of the software you’ll need, simply describes some of the stuff I’ve looked at.

 

SVN repository: Subversion source control, used to manage your code versions.  Your life can not be complete without having one of those, be it SVN or Git (see a separate post on the topic)
Tortoise SVN client: A very good and free subversion client, don’t look anywhere else.  I know there’s one for windows, don’t remember seeing one for Unix.
Maven 2: THE right way to manage your java dependencies, run your tests, and even manage the whole product packaging life cycle.  Downloading JAR’s yourself takes time, you need to look for them, and upgrading a version is a pain, not to mention managing the different dependencies.  With maven, you simply say what your project needs, and maven takes care of it.  It’s as simple as that.  If you’re building a product from scratch, don’t even think of working any other way.
Maven Artifactory: used to host maven artifacts.  When maven builds your war, you can host it in your artifactory for easy access by developers, production, or other projects using maven which depend on your project.
Tomcat: one of the best application servers out there.  Very light weight, works on all OS’s, old and mature, and constantly under development.
Jetty: Another very good app server, I use it mainly as a maven plugin because of a very nice feature it has of real time resource reloading, which means I can change a JSP or even a class and it will immediately reload it and reflect my changes. 
Apache web server (httpd): One of the best Web servers out there, I’d say it’s the Tomcat equivalent of a web server (some will say it is the other way around, but w/e).  The question of whether or not to use a web server as an app server front end is too deep for this blog post, but if you do decide to use one, Apache is one of the two top options.
ngineX: The other top web server on the market, said to be x50 faster than Apache.  It is still rather new, but quickly making its way up, with usage levels raising.  I haven’t looked at the statistics but if the trend continues it will eventually take over and become #1.
Teamcity: A Continuous Integration server by Jetbrains (if the name sounds familiar it should, it’s the same guys who made Idea).  Considered to be one of the best of its kind, it is free to use for small startups with limited deployment requirements, and will cost about $8k for standard use.
Jenkins: Built from the ruins of Hudson, the open source Continuous Integration project taken over by Oracle after their Sun acquisition, this is considered the best open source CI project, used widely by many.  A while back I had to choose between Jenkins and Teamcity, we did a wide research  on the subject (also looked at Bamboo, from Atlassian, the guys who also make Jira).  We finally went with Teamcity after realizing that support costs for Jenkins will be around the same price that Teamcity costs, and Teamcity is a little bit better in some ways, with a better interface and smoother integration with Jira.  But honestly, if you don’t need the full support package, Jenkins has all you need to get going, and it’s free.
MySQL DB: well, not much to say here, free, small, easy to use and manage using PhpMyAdmin.  Unless you want to spend millions on Oracle, this is a very good choice for SQL based databases.
PhpMyAdmin for MySQL administration: Used for administrating MySQL.
Eclipse: The best open source IDE out there, with an incredible number of plugins and fully featured to the level that it is a fair competitor to Idea, the #1 IDE on the market made by Jetbrain.  See a separate post on this subject.
Idea, Community Edition: For those of us wanting to save some $$$, this version of Idea has almost all you need, the limited functionality which makes it a little bit of a problem to use for development of certain types of applications is server support (can’t officially run it from Idea, you need to do it on your own, although you can run Tomcat as a standard program (not as a “server”) and so if you use maven or ant to deploy your war and configure it correctly you can have it run within Idea, and even debug.  Haven’t found a way to overcome the JSP limitation, so if you need that one you’ve got a problem.   If you want to buy the full version, they have a “personal” license for 200$, which is more affordable than the 500$ they ask for  in their so called “company” license.
I’m a big fan of Idea, can’t say I’ll spend 200$ on it for now though.  Perhaps when I get some funding 🙂
Puppet Enterprise 2.5: For those of us fortunate enough to need deployment to a large number of servers, PuppetLabs has a fine product for automated deployment.  I know there are open source products out there which are pretty good, haven’t done the research on that one yet.
SpringI’m not going to talk too much about Spring here, there are very good resources out there and springsource have a fine web site you can explore.  
Besides, I have a feeling I’ll be talking about it extensively in future posts.  For now I’ll just say that if you’re building something from scratch, taking Spring as your container will make life simpler and force you to build your code in a way that is organized and supportable in the future.
Tiles 2: Tiles (2) is a framework for building web pages from fragments, so for example you can have a header, a body, and a footer each in their own file, and within each you can inject different content based on what you need.  I have a feeling I’ll post something on this subject later, since I’ve done my research trying to decide between Tiles, Spring MVC, and Struts.  I eventually picked Tiles 2, more on that later.
jQuery: A very popular Javascript library built to make life easier when dealing with the different browsers on the market.  Writing browser compatible javascript code is difficult, and requires extensive testing, if you can limit yourself to a 3rd party library which promises full browser compatibility, you’re set.  It also has a very wide range of plugins to do really cool stuff.
MongoDB: The trend nowadays is moving away from SQL databases into the so called “No-SQL” DB’s.  Mongo is one of those light-weight No-SQL databases which lets you store your data as a “document”, actually in a JSON format.  Compared to its size, this database has really strong capabilities.
Logback: Once upon a time there was log4j, a very popular logging framework for java.  Then the creator of this framework came up with slf4j, an interface for using logging engine in a standard way.  And finally there was Logback, the latest from this very capable developer, and what you should now use to do your logging in java.

Post 1 – The basics – Eclipse vs. Idea

August 19, 2012 1 comment

Having worked with Idea (by Intellij) for a long long time I have to admit that I’m a big fan.  They’ve managed to conjure up every single useful feature you could ask for, and excluding v. 7 & 8 their IDE is very well behaved, works fast and responsive even in big  projects (the one I worked on had over 1m lines of code).

However, their personal license costs 200$, and community version has some limitations (I’ve looked at v.11) especially with running external servers and with debugging JSP’s, which made Eclipse a better choice for me at this stage (early startup stages, no extra money to spend).

I was a bit fearful of using Eclipse and the IDE did not feel intuitive at first, but having played around with it for the past couple of months now I know what to expect and can say that, while not as comfortable as Idea, it is still a valid choice.

Bottom line: choose Eclipse if you can’t afford the 200$, you’ll get used to it eventually and won’t miss Idea’s extra features that much (aside for ALT-F7, the ever so useful find-usages, which has only a slower alternative in Eclipse called Show-Hierarchy)

Categories: R&D Tags: , ,