Clean the slaves

In my current project we’re generating jobs for each branch of each project with the help of the Jenkins Job DSL. Branches are created for each feature and bugfix – hotfixes are branched from release branches but these are treated a little differently. Anyway, as you can imagine we tend to have a lot of branches even though we delete them once they’re merged back into the master. Since we also have a lot of different jobs for building/testing/deploying, the amount of jobs explodes quite quickly ;)
Also, to have acceptable job execution times and fast feedback for the developers, we’re using multiple slaves.

Problem

When deleting a job, e.g. because the branch is removed, its workspace on the slave is not deleted. This is a little annoying since it’s wasting a lot of space on the slaves’ disks and in the past required some manual effort to clean them up. Luckily, Jenkins provides some nice APIs :)

Solution

We created a job which is executing a “system groovy script” (important!) and must be run on the Jenkins master. It simply runs through all slaves, checks if their workspace contains a folder which is orphaned and deletes it. And yes, it is as simple as it sounds. Just a few lines of code which are run after the job generation and all orphaned workspaces are removed :mrgreen:

import hudson.model.*
import hudson.node_monitors.*
import jenkins.model.*

for (node in Jenkins.instance.nodes) {
    computer = node.toComputer()
    if (computer.getChannel() == null) {
        continue
    }

    def rootPath = node.getWorkspaceRoot()
    def space = DiskSpaceMonitor.DESCRIPTOR.get(computer)

    println "Checking node ${node.name}..."
    println "Got path ${rootPath} and remaining space ${space}"

    rootPath.list().sort().each { dir ->
        if (!Jenkins.instance.getItem(dir.name)) {
            println "${dir} orphaned. Deleting..."
            dir.deleteRecursive()
        } else {
            println "${dir} still in use. Skipping."
        }
    }
}

Using your MacBook as Jenkins slave and thereby avoid any sleeping

As a requirement in my current project, we have to support IE11 and Safari 9 & 10. Since running Mac OS X virtualized comes with some legal issues, we decided to simply use a MacBookPro (El Capitan) as our Jenkins slave of choice. This works out of the box until the MacBook decides to hibernate, suspend or simply start any energy saving feature Mac OS offers. For normal usage, these are some great features but as a Jenkins slave, they are pretty annoying. So here are some hints you can follow to disable the annoying sleeping features:

At first, simply go to the energy preferences and set them up as shown here:
Energy System Settings

This is not enough though. To disable hibernation, you have to execute the following command:

sudo pmset -a autopoweroff 0

And again, this is not enough, as closing the lid will put the MacBook asleep… Therefore you should install a tool like NoSleep and configure it like this:
nosleep

…but, Apple was pretty busy trying to avoid any kind of energy consuming background task. Since we want to test within Safari, we have to disable Safari’s background sleeping as well. So you have to run the following command:

sudo defaults write com.apple.Safari NSAppSleepDisabled -bool YES

Watch out with running this on your own machine though. Safari will drain your battery from now on ;)

From now on, your device will silently run and wait for work :mrgreen:

Update:

You can also set this globally:

defaults write NSGlobalDomain NSAppSleepDisabled -bool YES

Also, when using Safari for automated tests, you might want to set the following, to prevent Safari from reopening the latest windows:

defaults write com.apple.Safari ApplePersistenceIgnoreState YES

By the way, if you want to be notified for new posts, just sign up with your email address on the right :)

Beertastic

Aus Spaß haben wir mal wieder eine neue App gebastelt :mrgreen:

beertastic   beertastic2

Verfügbar für Android und iPhones kannst Du jetzt leicht tracken wieviel Du getrunken hast und weißt es besser als der Kellner! Außerdem bekommst Du je nach Pegel lustige Namenszusätze und siehst, was in Deinem Umkreis los ist! Sehr praktisch ;)
Es sind noch einige Features geplant und werden im Laufe der nächsten Wochen nach und nach kommen. Wünsche & Feedback sehr willkommen!

   

Übrigens, wenn Du über meine neuen Posts benachrichtigt werden möchtest, einfach rechts Deine Emailadresse eintragen :)

The 7±2 reasons why gibbon’s five failed

Yes, gibbon’s five failed. Even though I am still convinced that our vision of providing very specific, high-value recommendations for users of what to do in their spare time was a great one, we did a lot of things wrong and did not manage to create enough growth and traction.

We’ve been working on our first product for quite some time and always got great feedback from the existing users. The servers ran fine and we made a lot of connections with local businesses in Munich. Nevertheless, we took our first product offline some weeks ago. It was a really tough step for us and even though we never really talked about it, I know it was pretty emotional for everyone of us. Now, some time later, one of the co-founders published an article about it on his blog. Of course he doubled checked with us ;)

Last but not least. Like almost all startups, we barely ever had technical problems in our way and tech was definitely not part of the problem.

http://www.leanelephant.net/why-gibbons-five-failed/

On top I’d like to mention that I learned a lot of things you won’t find in books. You have to experience it yourself. And yes, I am willing to do it again and so are the others. Probably without doing the same mistakes again, but there will be mistakes and we know about that. But we improved our way of dealing with it ;)

By the way, to get notified about new posts, just enter your email address on the right :)

Some remarks on SubGit

I’ve been evaluating SubGit with a colleague for some time now. It seems to be a decent tool if you have to use Subversion and can’t simply switch to Git completely. Nevertheless, we stumbled on a few (pretty annoying) things:

First, SubGit writes changes to the Git tree right into Subversion. As Subversion doesn’t know things like fast-forwards, this results in funny commits with messages like “trunk:$rev replaced by branch:$rev“. That kinda freaked out some colleagues of mine before we understood what’s really going on[1]. A solution for this is the –no-ff flag when merging. You might just want to set it for the whole repository or even globally:

git config --add merge.ff false # add --global if you want to set it globally

Also, if you have a lot of Maven modules and want to ignore things like .classpath, .projects, target etc, this will result in SubGit trying to burn down your CPU to create a lovely .gitignore[2], for us with ~5500 lines. The solution is quite easy. Edit your $subgit_project_dir/subgit/config and add the following:

[translate]
    ignores = false

This simply tells SubGit to stop translating the Subversion ignores into Git ignores. Now you have to manage the ignores yourself but at least for us it’s worth it ;)

Last but not least: Use the beta of version 3. It’s much more mature than the name suggests and for us it’s also more stable.

[1] http://lists.tmatesoft.com/pipermail/subgit-users/2013-June/000129.html

[2] http://issues.tmatesoft.com/issue/SGT-828

By the way, to get notified about new posts, just enter your email address on the right :)

subversion, git, subgit & Windows

On client-side we’re currently working with subversion, which is a (serious) pain. On top we have Windows7 machines. To deal with the first problem, I used git-svn for quite some time and in combination with Sourcetree it worked quite well. When I came back from my xmas break last week, the subversion structure was changed drastically and git-svn didn’t work anymore. It couldn’t clone the new repository and died after hours of trying with weird error messages. As I wanted to try out subgit (“git mirror for svn”) for quite some time, I didn’t really dive into the git-svn problems but gave subgit a go for evaluation :)

First thing to do is quite obvious: Download and install subgit.
Afterwards you have to configure the repository you want to clone:

subgit configure --svn-url $YOUR_SVN_URL $DESTINATION

I also made some tiny changes to the $DESTINATION/subgit/config file. I enabled httpSpooling (httpSpooling = true) to avoid timeouts and added the following lines at the end of the file:

[translate]
    eols=false

(More about line endings? -> Mind the end of your line)

Next, set your credentials within $DESTINATION/subgit/passwd. Careful: The (unlimited) evaluation version of subgit stores the passwords in plaintext.
Finally, tell subgit to “initialize” the project and start the demon – this might take some time, e.g. for me it took 9h:

subgit install $DESTINATION

Now you’re ready to use whatever git client you like to clone the repository from $DESTINATION and treat it like every other git repos you’re working with. Subgit is dealing with all the annoying svn stuff.
What I still need to test is good the branching and merging “translation” works.

If subgit is running on your dev system, you have to restart the demon by executing the install command again.

A small hint: If you want to use git 2.x on your Windows machine, you won’t find an official download for it yet. The current latest Windows version is 1.9.5. But you can install cygwin with git 2.x and simply add the $CYGWIN_DIR/bin to the Windows $PATH, as Cygwin provides you with a git.exe :P

Grails Unit Testing and a little fun with @Before

So, we’ve been using Grails at gibbon’s five for a while now and stumbled into different problems with our unit tests. One of the weirdest ones were occasionally failing tests on our Jenkins that never failed locally. So we digged into this a little and came up with a simple explanation, that I’d like to share with you.

Let’s assume we have a controller and want to test it. First thing you’d probably do is go to the Grails Guide, Chapter 10: Unit Testing Controllers and start with something like this:

import grails.test.mixin.TestFor

@TestFor(SimpleController)
class SimpleControllerTests {
    void testSomething() {
    }
}

Now, if you have multiple tests which need some variable set inside the (controller.)request and you have a little knowledge of JUnit, you’d probably come up with something like this:

import grails.test.mixin.TestFor
import org.junit.Before

@TestFor(SimpleController)
class SimpleControllerTests {
    @Before
    void initialize() {
        request.variable = "foobar"
    }
    void testSomething() {
        // some code here needs the request.variable
    }
    void testSomethingElse() {
        // some code here needs the request.variable
    }
}

This is bad.

Continue reading “Grails Unit Testing and a little fun with @Before” »

FindBugs & Reject.if* False Positives

In my current project we’re using Reject.if* (ifNull, ifNotNull, ifTrue…) methods for several validations, error management etc. As we’re also using FindBugs, we stumbled upon a problem with code like the following, in which FindBugs comes up with a NP_NULL_ON_SOME_PATH warning, in this case a false positive:

void foo(String bar) {
    Reject.ifNull(bar);
    bar.getBytes();
}

After searching for a solution and asking some people, we finally found a solution that works pretty well in this case: Passing a system property called findbugs.assertionmethods [1] through Maven to FindBugs. The code for it in the pom looks somewhat like this:

...
<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>findbugs-maven-plugin</artifactId>
    <version>2.5</version>
    <configuration>
        <effort>Default</effort>
        <threshold>Medium</threshold>
        <xmlOutput>true</xmlOutput>
        <findbugsXmlOutputWithMessages>true</findbugsXmlOutputWithMessages>
        <omitVisitors>FindReturnRef</omitVisitors>
        <jvmArgs>-Dfindbugs.assertionmethods=de.mopri.utils.Reject.ifNull</jvmArgs>
    </configuration>
</plugin>
...

By using the jvmArgs element the system property is properly passed down to FindBugs and telling it that the method Reject.ifNull is used as an assertion. Now FindBugs acknowledges that bar.getBytes() won’t throw a NullPointerException :)

Control flow with exceptions :D

Reminds me about some discussions I had about controlling your program flow with exceptions :mrgreen:

Using exceptions to control program flow is a bad idea. Creating exceptions requires taking a snapshot of the call stack, and that’s costly. – John Flinchbaugh Oct 16 ’08 at 12:17

Not if you override the method that generates the call stack in the exception you throw to do nothing. That’s the advantage of a custom exception. – shemnon Oct 17 ’08 at 14:03

The whole thread can be found here on stackoverflow.

Grails with time-based rolling logs

Logging in grails is somehow confusing me a little. I don’t know why but even though I’m familiar with log4j and never had trouble working with it, grails logging sometimes manages to make me feel like a total newbie. After looking for a nice and simple solution for a timebased rolling and archiving, I thought someone else might be interested in it as well and I don’t have to look for it again ;) So here is the daily rolling log configuration, which is also archiving (and zipping) the old log:
Continue reading “Grails with time-based rolling logs” »

Jahre später kommen wir zu Ruhm :)

PSDetrisHeute Morgen trudelte eine Email mit einem Link auf diesen “großen Beleg” der TU Dresden ein. Nichts ahnend habe ich ihn erstmal aufgemacht. Als ich dann den Titel Remote-Interaktionen mit Handhelds auf Großprojektionen las, war ich etwas verwirrt, bis mir ein Projekt einfiel, an dem ich mal mit Kommilitonen gebastelt habe: PSDetris!
Ab Seite 37 wird unser kleines Projekt relativ gut beschrieben, auch wenn es ein wenig Namenschaos gibt – kann der Autor ja nicht wissen. Auf einem der Bilder sieht man mich sogar bei der PSD Bank “spielen” :P

Wir haben es damit also doch noch zu ungeahntem Ruhm gebracht :mrgreen:

Deshalb hier noch mal das legendäre Promotion-Video :D

Ajax Upload tweaked for grails

While coding a little on gibbon’s five we stumbled upon a nice JavaScript library called file uploader. The first tests we did looked good so we took it into the project. While checking the browser compability I ran into some problems with the Internet Explorer that need to be fixed.

First of all, the return type for the IE has to be a little different – the IE needs text/plain. Also, the data send from the IE to the server looks a little different from the one WebKit and gecko: It’s a CommonsMultipartFile. Therefore I had to do some tweaking on the server side which ended in some code pretty similar to this:

if (params.qqfile instanceof org.springframework.web.multipart.commons.CommonsMultipartFile){
    returnType = 'text/plain'
    content = params.qqfile.getInputStream()
    originalFilename = params.qqfile.originalFilename
    mimeType = params.qqfile.getContentType()
} else {
    returnType = 'application/json'
    content = request.inputStream
    originalFilename = params.qqfile
    mimeType = URLConnection.guessContentTypeFromName(params.qqfile)
}

Furthermore, the plugin has to deal with older browsers and the IE by using iframes (mentioned on the website and explained in the code). This works pretty well until you come across grails, which, for whatever reason, sends pre-tags around the json code. Of course our friend the Internet Explorer is confused and is simply not evaluating the answer. A simple solution to this is to change a few lines of code in the function (fileuploader.js):

_getIframeContentJSON:function (iframe) {
    // iframe.contentWindow.document - for IE<7
    var doc = iframe.contentDocument ? iframe.contentDocument : iframe.contentWindow.document, response;

    this.log("converting iframe's innerHTML to JSON");
    this.log("innerHTML = " + doc.body.innerHTML);

    // Grails is answering with pre-tags. That might confuse the browser script therefore we better remove them.
    var cleanedAnswer = doc.body.innerHTML.replace(/<(\/)*pre>/g, '');
    this.log("grails cleaned: " + cleanedAnswer);

    try {
        response = eval("(" + cleanedAnswer + ")");
    } catch (err) {
        response = {};
    }

    return response;
}

Overall it’s a simple tweak and now it works perfectly within all the browsers we tried :)

1 2