Book Review: Programming Concurrency on the JVM

Overview
Programming concurrency is a tough task to get right, just ask Venkat, he'll tell you so!  I attended a presentation by Dr. Venkat Subramaniam on this very subject, Concurrency on the JVM, at my local Java user group a couple months ago and he really caught my attention.  The presentation was a condensed version of the book, and needless to say, I bought the book to help fill in all the details.

Not only is Venkat a 'subject matter expert',  but his presentation kept everyone engaged and laughing!

Contents:
The book is broken down in 5 parts, with 10 chapters total, at just over 250 pages.  The main sections of the book are:

• Strategies for Concurrency - this discusses the impact of multi-core machines and the three design approaches that are discussed throughout the rest of the book, Shared Mutability, Isolated Mutability and Pure Immutability.


• Modern Java/JDK Concurrency - covers the Java 1.5+ concurrency features and talks about how to use the newer concurrency classes, like ExecutorService, CountDownLatch, Locks and Fork/Join to solve concurrency problems.


• Software Transactional Memory - covers Software Transactional Memory (STM), popularized by Clojure, which places access to memory within transactions.  STM is one option for getting away from the 'Synchronize and Suffer' model as Venkat call it.  The intention is to make threaded code more deterministic.   Using the JDK concurrency tools, there is no way to be sure you code is correct because it doesn't always fail when/where you need it to.   By putting the memory access inside transactions, the transaction manager helps resolve the conflicts without explicit locking.


• Actor-based Concurrency - covers actor design approaches as another option for concurrency design without using synchronization.  This section exercises actors from the Akka library from Scala, Actors from the Groovy library GPars, and mixing Actors and STM as a possible solution.


• Epilogue - is a short recap of the topics presented previously and points out the scenarios when each of the solutions would be appropriate.

The book lays out a couple example problems and then develops solutions for these problems across a range of JVM languages (Java, Groovy, Scala, Clojure and JRuby).   The examples are short and concise,   enough to help you to get started prototyping against your own concurrency issues.

Summary
This is a good book to get you started on some of these newer concurrency concepts and libraries.  It was an easy read, and I completed the reading fairly quickly, as opposed to the "Java Concurrency in Practice" book, which I started and stopped twice and still have not completed yet!

I would recommend this book to anyone looking to improve their programming skills in the concurrency area.  The book provides you with some alternatives to the old standby of ' Synchronize and Suffer' model.
My last recommendation, if you get the opportunity, attend one of Venkat's presentations, you won't be sorry!

Passing parameters into Groovy script using Binding class

I recently saw a question posted on the Groovy/Grails group on LinkedIn asking about ways to pass in parameters to a Groovy script.  There were several responses pointing to the CliBuilder class which is certainly one way to handle the problem.   I had just finished reading an article by Ken Kousen in the November issue of GroovyMag  where Ken mentioned another option: using the Binding class.

The example below shows a couple ways of setting variables in the binding, getting the variable values from the binding and how to capture standard output from the script.  The one tricky part is the question "When is something in the Binding and when not?"  The answer is: when it's not defined, it is in the binding!  In the example below, variable c is placed in the binding, but since it is def'd in the script, the value for that variable comes from the local variable rather than the binding.

// setup binding
def binding = new Binding()
binding.a = 1
binding.setVariable('b', 2)
binding.c = 3
println binding.variables

// setup to capture standard out
def content = new StringWriter()
binding.out = new PrintWriter(content)

// evaluate the script
def ret = new GroovyShell(binding).evaluate('''
def c = 9
println 'a='+a
println 'b='+b
println 'c='+c 
retVal = a+b+c
a=3
b=2
c=1
''')

// validate the values
assert binding.a == 3
assert binding.getVariable('b') == 2
assert binding.c == 3 // binding does NOT apply to def'd variable
assert binding.retVal == 12 // local def of c applied NOT the binding!

println 'retVal='+binding.retVal
println binding.variables
println content.toString()

Output

[a:1, b:2, c:3]
retVal=12
[a:3, b:2, c:3, out:java.io.PrintWriter@1e0799a, retVal:12]
a=1
b=2
c=9

Hope this helps!

Book Review: SQL Antipatterns Avoiding the Pitfalls of Database Programming

Overview
This is book for software developers that are either new to database programming (SQL) or those that may not have had any formal database education and may have learned 'on-the-job'.

The book is bit longer, approximately 300 pages, than some of the other titles I've read from The Pragmatic Programmers shelf, but it is a very easy read.  The chapters all follow a similar format: Object, Antipattern, How to Recognize the Antipattern, Legitimate Uses of the Antipattern and Solution.  Each chapter the author leads you through a problem that needs correcting within the scope of the database.   Next the antipattern is laid out for you and the author shows the disadvantages of using the antipattern.   Tips are provided to help identify the antipattern and he also suggests some possible legitimate uses of the antipattern.  Finally, the author describes better solutions to the problem presented; ones without the disadvantages discussed previously and solutions that are considered 'best practices' and that will provide better database performance.


Contents
The book is broken down into 4 main sections:

• Logical Database Design Antipatterns - planning database tables, columns and relationships
• Physical Database Design Antipatterns - defining tables, indexes and choosing  datatypes
• Query Antipatterns - SQL command usage, for example, SELECT, UPDATE and DELETE
• Application Development Antipatterns - correct usage within the scope of a language

Some of that antipatterns described may make you laugh and wonder, "Who the heck would do something like that?" while others you may have run into previously and some might even hit very close to home from your work or personal projects.

There is also an appendix that covers the Rules of Normalization.  I had heard of terms like Normalization, First Normal Form, and Second Normal Form before, but never actually read any explanations. The appendix includes discussions on normalization and first normal form through fifth normal form with examples to help clarify the topics. 

Summary
This is a very good book for software developers just starting to 'cut their teeth' with databases and SQL.  It also serves as a good refresher for those with a bit more database experience.  In my case, I fall into the later scenario.  You can view it as a book of short stories, showing what not to do in these cases.

Jars signed with mutliple code signing certificates?

For anyone that has built and maintained Web Start applications, you have probably been through this issue before.   Your QA group or worse yet, a customer, calls to tell you that they cannot start your application because the download failed because "jar resources in jnlp are not signed by the same certificate".

If want your Web Start application to have full access permissions, then you need to sign all the jars that get downloaded.  This generally presents the opportunity to encounter one of the following errors:

1. None of the jars are signed.  Most likely a build issue and generally only happens once, (I hope!)
2. Single jar is not signed.  Again, most likely either a build or process error.
3. Not all jars signed with the same certificate.

The first two are easy to resolve.  The last is a bit of a pain because Web Start doesn't bother to tell you exactly which jars where signed with different code signing certificates.  Now, it's time to make some educated guesses as to what changed recently and look at the most likely/usual suspects.   Another option is to get the list of download jars from the JNLP document and figure out which certificates(s) each jar has been signed with.

We have been through this at work once or twice and so I finally decided to make it easier for the next time this happens.   The approach I took was to read all the jars in a deployment folder, open them and look for the signature files inside the jar file.   For the results, I created a map keyed by the jar name and the value being a list of the signature file names (*.RSA, where the * represents our code signing certificate alias).  Keeping a list in the map allows for cases where a jar may have multiple signature files, which was encountered when we switched over code signing certificates.  Writing this in Java would be possible, but why write so much code? Why not write a script in Groovy, it would be some much shorter and concise.

Groovy Script

def dir = new File("C:\\MyDeployment\\jboss\\myserver\\deploy\\myserver.ear")
def jars = dir.list( { d, f-> f ==~ /.*.jar/ } as FilenameFilter)
def map = [:]
jars.each() {
   def zipFile = new java.util.zip.ZipFile(new File(dir, it)) 
   zipFile.entries().each { zipEntry ->
      if (zipEntry.name.endsWith('.RSA')) {
          if (map.containsKey(it)) {
             def list = map[it]
             list << zipEntry.name
          } else {
             map[it] = [zipEntry.name]
          }
      }
   }  
}
map.each{ println it}

Hope this helps!

Lessons learned from recent PA/DSS audit

The following are some tips/hints and lessons learned during our recent PA/DSS audit.   Everyone's mileage will vary. 

Our application runs on both Windows and Linux.   We support JBoss and Weblogic for the application server.  We support Oracle, SQL Server and  MySQL for the database.  We have three client applications which are all WebStart clients, so this is NOT a web application.  Our EJBs are EJB 2.1 Session Beans.

Hints

• Determine exact software stacks needed including all software levels, build levels and whatever else you need.  Collect all of this in one safe place.
• Get the hardware ready and any scripts that may be needed to use those devices.  Our application is a POS, so we had a collection of pin pads that we were required to test with during the audit.
• Get any utility software needed
• Ghost for Win* images (or Win 7 provides a Backup/Restore feature)
• We ran into some problems with Windows Server 2008 and had to use one of the  Acronis tools to create the backup image.
• PartImage for Linux images
• Darik's Boot and Nuke
• anything else you might need
• Wipe the hard drives before loading any software, including the OS, using Darik's Boot and Nuke to be absolutely sure that the drive does contain anything that might match a credit/debit card number.
• Install the OS and any 'base' software, which isn't the software to be verified by the audit
• Keep the partition sizes as small as possible without causing disk output problems.  The larger the disk, the longer the forensic scans will take to run.
• On Windows - you can have a page file, just don't allow it to grow and shrink.   Set the min and max file size to the same value.   
• Problem here is that if card holder data is written to the page file then later the page file size is shrunk, some of that card holder data MAY be inside freespace, which is NOT cleaned up by the OS.   This can cause some instances of card holder data to be found during a forensic scan and NO ONE wants card holder data found during the scans.
• On Window - you should also set the Windows Update setting to be Notification because you don't want Windows Updates being applied to a machine during the audit.
• Our auditor also recommended we turn off the Windows Restore points.  Again, you don't want Windows doing any extra work that could impact the audit results.
• Take images of all boxes.  These may be needed to reset a machine back to a 'clean' state for ad additional set of tests.
• Talk to each of your authorization providers
• let them know when your audit is scheduled and ensure you will have connectivity to their test servers
• Verify that each provider can supply 'magic authorization values' that will help your trigger the following responses:
• approvals
• declines
• referrals
• timeouts
• void
• split tenders
• partial auths
• If any of the authorization providers encrypt their communications, then you may need to ensure you have whatever is needed for that.   We needed to download and install the JCE .
• If you support multiple authorization providers, you will have to run tests against each provider so be ready to do whatever it takes to switch your application from one provider to the next.
• Practice taking images and restoring images on all boxes
• The Linux tools are not as user friendly as the Windows tools, so you need to be ready and able to make and restore your images in a timely manner.
• Now you can install your software and you should be ready for the audit.
• If you do any testing before the auditor arrives, be ready to restore the machines back to a 'clean' state because you don't want to take a chance that your early testing wrote any card holder data somewhere on the hard drive.
• Our auditor previously used WinHex to do the forensic scans.  We used the evaluation version to try and double-check ourselves.  The problem with that version is that it is slow and can only scan for one card number at a time.  This time the auditor had a new set of software for the scans and unfortunately I didn't get the name of the package, although I know it was a purchased product.

Good luck - hope this helps!

Book Review: Building and Testing with Gradle

Overview

Gradle is one of the new "cool kids on the block".  It  is quickly gaining momentum as projects like Hibernate and Spring migrate their build processes over to Gradle.   Gradle can be considered the 3rd generation of Java build tools, with Ant and Maven representing the earlier generations.

Gradle is built on top of Groovy, which provides the ability to define a Domain Specific Language (DSL).  In this case, the Gradle DSL provides a language tailored to the task of building code.  If the DSL does not provide exactly what you need, then you can extend the DSL using plug-ins.  It is possible to learn Gradle without knowing Groovy, but some initial knowledge of Groovy is beneficial.

This book is short (89 pages) .  It brings back memories of the O'Reilly "A Developer's Notebook" series.

Contents

Chapter 1, Hello Gradle  This chapter leads the reader thru the installation and configuration of Gradle and demonstrates the ubiquitous Hello World (build file) example.

Chapter 2, Gradle Tasks  This is the  best chapter in the book.  It drives home the concepts associated with a 'task' and how the tasks can be configured and customized.    The most helpful  suggestion in the book is found in this chapter.  The  recommendation is that you read the Gradle DSL documentation.  I had previously read the Gradle Users Guide but was still a bit fuzzy on what  the properties and methods were and how they were used. .   Reading the DSL documentation provides the full picture.  It also  helps drive home  that you are working with a programming language/DSL and not a set of XML tags like Ant and Maven.

Chapter 3, Ant and Gradle  This chapter compares and contrasts Ant and Gradle.  It also demonstrates that Gradle can run Ant tasks by importing an Ant build.xml or by using the Ant Builder that is part of the Groovy distribution.

Chapter 4, Maven and Gradle  This  is the longest chapter in the book comprising  about 25% of the book.  Like all of the Gradle documentation, this book re-assures the reader that any previous investment in Maven or Ivy will not be lost in a move to Gradle.   Gradle provides support for Ivy, Maven's Central repository and a local repository.

Chapter 5, Testing with Gradle  This chapter is a  good overview of using  JUnit , TestNG, Spock, Geb and EasyB.  Coverage of testing is a bit light given the title of this book. 

Chapter 6,  Multiproject Builds  This chapter  does a nice job of taking a sample project and showing three different approaches to creating the project build structure.  The approaches shown are one master build file, project-specific build files and a hybrid approach.  While the sample projects are very small, they provide the reader a starting point to converting their own multiproject builds.

Summary

This is an excellent book for getting started with Gradle.   There are plenty of working examples provided in the source code download package.   The authors do a very good job of explaining the concepts and presenting alternative ways of expressing the same options within the DSL.

One minor issue is that matching up the downloaded source examples to the examples in the printed chapters is a bit of an exercise because they are not organized in a folder structure by chapter like many other books.

  

Initially the slim size of the book worried me as to it's ability to provide a depth of coverage of Gradle, but the authors stated in the Preface that "Future volumes will cover the Gradle plug-in ecosystem, how to extend Gradle with your own business logic and even more advanced topics."    Sounds good to me, bring it on Tim and Matthew!

Gradle dependencies with jars that have no version number

I have seen presentations and read blogs and articles on Gradle. It sounds pretty good.  Gradle is gathering momentum as some of the major players in the open source community (Hibernate and Spring Security to name a few) have switched their builds over to using Gradle.  I figured it is time to give it a try.

The project I work on is fairly old, pre-Maven I suspect.  The project is a multi-layer, multi-component Java project built using Ant.  The project does not use Maven or Ivy.  The project includes a folder that contains all of our dependent jars. These jars are committed to SVN and included as part of the project when it is checked out. This way we have all the dependencies we need to build the project.  We have quite a few dependent jars that do not have version numbers in the jar name.  Examples include a copy of junit.jar and log4j.jar.  This all existed before I started working on the project, so I have no idea how it got that way - it just is.  And since it's not really broken, we haven't attempted to fix it!

Not having version numbers in the jars is a minor inconvienence.  If you read the Gradle User Guide chapter on Dependency Management, they seems to push dependency management for jars with version numbers.  Most of the examples shown in the documentation and elsewhere declare the dependencies as seen in Example 34.1 in Dependency Management chapter.  Great, but what do I do if I don't have version numbered jars in my dependencies?  I did not believe the Gradle guys would leave me hanging...  And they didn't!

I really wanted to figure out how to setup my classpath to include these jars without version numbers.  I figured out how to do this and started writing this blog.   Then as I re-read the Dependency Management chapter, I saw a clue that I must have blown right by the first time I read it.  The clue is in Section 34.3.5 File dependencies.  You can use the FileTree  to help define the classpath to include these non-versioned jars.

I created a small project called GradleTest that did not use the standard Java plugin conventions for the source and resource files and has dependencies in the "libs" folder, none of which have version numbers in the jar names.

I created two different options, which both seem to work.  The first is shown below.  This option handles the dependencies more specifically, assigning just the jars that are needed for either compile or runtime.

apply plugin: 'java'

sourceCompatibility = 1.6
version = '1.0'

sourceSets {
    main {
        java {
            srcDir 'src'
        }
        resources {
            srcDir 'resources'
        }
        compileClasspath += fileTree(dir: './libs', includes: ['**/log4j.jar'])
        runtimeClasspath = classes + sourceSets.main.classes + fileTree(dir: './libs', includes: ['**/log4j.jar'])
        
    }
    test {
        java {
            srcDir 'test'
        }
        resources {
            srcDir 'testresources'
        }
        compileClasspath += fileTree(dir: './libs', includes: ['**/junit.jar'])
        runtimeClasspath += classes + fileTree(dir: './libs', includes: ['**/junit.jar', '**/log4j.jar']) 
        
    }
}

sourceSets.main.classesDir = new File("./build/classes")
sourceSets.test.classesDir = new File("./build/classes")

The second option looks very similiar to the first option,  but uses the dependencies definition and it cheats by assigning all the jars in the "libs" folder to both the compile and runtime classpath, rather than just exactly what is needed.

apply plugin: 'java'

sourceCompatibility = 1.6
version = '1.0'

sourceSets {
    main {
        java {
            srcDir 'src'
        }
        resources {
            srcDir 'resources'
        }
        
    }
    test {
        java {
            srcDir 'test'
        }
        resources {
            srcDir 'testresources'
        }
        
    }
}

sourceSets.main.classesDir = new File("./build/classes")
sourceSets.test.classesDir = new File("./build/classes")

configurations {
    compile {
        description = 'compile classpath'
        transitive = true
    }
    runtime {
        extendsFrom compile
    }
}

dependencies {
    compile fileTree(dir: 'libs', include: '*.jar')
    runtime fileTree(dir: 'libs', include: '*.jar')

}

I am sure there are other,  possibly better or cleaner alternatives.  Please feel free to share improvements or other alternatives.   Hope this helps....