Install Java 6

First, we need to get Java and Eclipse installed. I’ve had some troubles with OpenJDK, so I’ll be using Sun’s (now Oracle’s) JDK. You can install the sun-java6-jdk package with synaptic or at the command line:

  sudo apt-get install sun-java6-jdk

Double-check that you’re using Sun’s JDK. Run java -version at the command-line. You should get something like this:

 java version "1.6.0_13"
  Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
  Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)

If OpenJDK, IcedTea, or GIJ is still being used by default, you can change it with the following command (more info here):

  sudo update-java-alternatives -s java-6-sun

All set?

Install Eclipse Ganymede

Unfortunately, even Jaunty still comes packaged with Eclipse 3.2 (Callisto) and not 3.4 (Ganymede). Repast Simphony calls for Ganymede, so here goes a manual install:

With wget or your favourite web browser, download Eclipse Classic appropriate to your Linux install (32-bit or 64-bit). Extract it to a directory (replacing the filename with the file you’ve downloaded):

  tar xzvf eclipse-SDK-3.4.2-linux-gtk-x86_64.tar.gz

For me, this extracted it to a subdirectory called eclipse. Feel free to move it and delete the .tar.gz archive. I moved my install to /opt/eclipse and changed the owner to root (by running sudo mv eclipse /opt && sudo chown root:root -R /opt/eclipse).

GUI and Command-Line Shortcuts to Eclipse

If you want a shortcut from the GNOME Applications menu, you can right-click on the Applications menu and click Edit Menus. Select the Programming menu and click Add New Item.

Adding an item to the Programming menu

From here, you can add a shortcut to your Eclipse executable.

Edit the item to point to the Eclipse executable

Want a command-line shortcut to Eclipse? You can add a soft-link to Eclipse in /usr/bin by running sudo ln -s /opt/eclipse/eclipse /usr/bin/eclipse. Now you can run Eclipse by running eclipse at the command line or with the Run Application window (Alt-F2).

Other components (JOGL, JAI, Java3D)

JAI

Download the current binary build from the JAI site. You can read the installation instructions here. This installs JAI right into the JDK.

Note that the JDK is (likely) installed in /usr/lib/jvm/java-6-sun.

Java3D

Download the zipped binary build from the Java3D site. Installation instructions are here. The installation’s a bit tougher, since it’s not recommended to install directly into the JDK, but here’s what I did:

Unzip the libraries and move them to /opt:

  unzip j3d-1_5_2-linux-amd64.zip
  unzip j3d-1_5_2-linux-amd64/j3d-jre.zip
  rm j3d-1_5_2-linux-amd64/j3d-jre.zip
  sudo mv j3d-1_5_2-linux-amd64 /opt

From here, you should add it to the classpath, but we'll do that later, once Repast Simphony is installed.

JOGL

Download the current release build from the JOGL development site. Installation is similar to the above libraries: Extract to some directory and make sure it's on your classpath (later).

Install Repast Simphony

Almost.

This is important: If you ran the chown command earlier to change the owner to root, start Eclipse as root:

  gksudo eclipse &

If you're still the owner of Eclipse's files, start Eclipse normally.

You'll need to install Groovy. Also, if it's not already there, you'll need to add the Ganymede update site since some dependencies will be needed. From the Software Updates window (Help > Software Updates), click the Manage Sites button.

Eclipse Update: Manage Sites button

Add http://download.eclipse.org/releases/ganymede and http://dist.codehaus.org/groovy/distributions/update/. Eclipse should be able to find the dependencies from there.

Repast Simphony can be installed as an Eclipse plugin from the update site, http://mirror.anl.gov/pub/repastsimphony. Check off Repast and Groovy and install!

Select Repast Simphony and Groovy

Adding libraries to the Classpath

Since JAI was installed right into the JDK, it's already included in the classpath. Java3D and JOGL still need to be added to the classpath. If you installed Java3D and JOGL right into the JDK (tsk, tsk), you can skip this section: You're all done! (But other things might be broken... who knows)

From Window > Preferences, then under Java > Build Path > User Libraries, add JOGL. Click New and type the library name. With the new library selected, click Add Jars and select both JAR files (hold down Ctrl). Click OK.

Add both JAR files

Not done with JOGL yet... select Native library location and click Edit. Set it to the same directory as the JAR files (/opt/jogl-1.1.1-linux-amd64/lib in my case).

Do the same thing for Java3D, except the JAR files come from /opt/j3d-1_5_2-linux-amd64/lib/ext and the native libraries come from /opt/j3d-1_5_2-linux-amd64/lib/amd64. (Locations might vary based on your Linux build and library versions.)

When you create a new project, you'll have to add JOGL and Java3D to the build path (Libraries tab).

Add JOGL and Java3D to the Build Path

All done!

Since JDBC’s been introduced in previous posts, a similar approach can be used by using the RJDBC package and MySQL Connector/J. There’s also a package called RMySQL to connect R to MySQL without JDBC, but I’ll discuss the JDBC approach.

If you haven’t done so already, download and install R. You can install the RJDBC package within R using the menu (Package > Install Package(s) > RJDBC) or with the following command:

  install.packages("RJDBC")

Assuming you’ve downloaded the MySQL Connector/J earlier, you’re ready to collect some data. Taking the same conventions for jdbcUrl, jdbcUser and jdbcPass as last time, initialize a connection to the database:

  library(RJDBC)
  drv <- JDBC("com.mysql.jdbc.Driver",
    "/etc/jdbc/mysql-connector-java-5.1.7-bin.jar",
    identifier.quote="`")
    conn <- dbConnect(drv, jdbcUrl, jdbcUser, jdbcPass)

The “com.mysql.jdbc.Driver” argument refers to the same one used in the Class.forName method used in the earlier post. Now that you have a connection, grab a data frame using a SELECT query. Say I wanted to read the input parameters for a particular run:

  runID = 7
  d <- dbGetQuery(conn,
    "SELECT param1, param2 FROM runs WHERE runID=?",
    runID)
  str(d) // show structure of data frame
  d    // print data frame

Wasn’t too useful? Here’s a basic function to plot a single variable over time given an agent’s run and household ID.

  plotSingleHouseholdCapital = function (runID, householdID) {
    d <- dbGetQuery(conn, "SELECT tick, capital
      FROM householdState WHERE runID = ? and householdID = ?",
      runID, householdID)
    plot(capital ~ tick, data = d,
      xlab="tick (Year)", ylab="Household Capital")
  }

For better performance in most cases, using stored procedures helps. A stored procedure is a block of statements which is executed on the database server. To create a stored procedure, see the MySQL documentation. Here’s an example:

  DELIMITER $$

  CREATE PROCEDURE avghouseholdstate (IN runID INT)
  BEGIN
    select runID, tick, avg(capital) AS capital
    from householdstate
    where runID = runID
    group by runID,tick;
  END $$

  DELIMITER ;

If you’ve created a stored procedure called avghouseholdstate with a single parameter runID, like the one above, it can be called from R like this:

  avgCapital = function (runID) {
    d <- dbGetQuery(conn, "call avghouseholdstate(?)", runID)
    plot(capital ~ tick, data = d,
      xlab="tick (Year)", ylab="Avg Household Capital")
  }

This is a very simplified version of a stored procedure I use myself, but hopefully it’s enough to illustrate at least the basics of retrieving data from MySQL (or any other JDBC driver) to R.

Defining a Database class

If you haven’t done so already, download the latest version of MySQL Connector/J and add it to your classpath. Create a new class called Database, which will act as the interface to the actual database, containing all of the methods you’ll need to log agent and model data. This will simplify the rest of the model by allowing agents to call “friendly” methods like logAgent instead of raw SQL code.

Establishing a Connection

First, you’ll need to make a connection from your Database object to the actual MySQL database server, where conn is a Connection field. The following code can be put into the Database’s constructor or intialization method:

  Class.forName("com.mysql.jdbc.Driver");
  conn = DriverManager.getConnection(jdbcUrl, jdbcUser, jdbcPassword);
  conn.setAutoCommit(false);

If you’re using a JDBC 4.0 driver (like MySQL Connector/J 5.1) and Java 6, the first line can be omitted. I usually leave it in, just in case I need to deploy to a pre-Java 6 environment. In the above, jdbcUrl can be replaced or assigned “jdbc:mysql://localhost:3306/[schema_name]“, where [schema_name] should be replaced with the name of the schema (without square brackets) created in Part 1. The default jdbcUser is “root”.

Transactions

What does that last line do? It turns off auto-commit and implicitly turns on transactions. In an ideal ACID-compliant world, transactions are atomic (as well as concurrent, isolated and durable). That means that if any part of the transaction fails, the whole transaction fails. So if you abort a run midway, none of the run’s results are recorded, which is the way I would prefer it. If that’s your cup of tea, you’ll need to end each run with a call to conn.commit() and abort a failed run (if possible) with conn.rollback(). If the connection closes without a commit, any uncommitted changes are rolled back anyway.

Closing the Database Connection

Since it’s inefficient to create and destroy connections frequently, the connection created above can be used for the entirety of the batch run. It’s a best practice to close the connection when you’re done with it. This can be done with a shutdown hook, which is a thread that runs when the Java Virtual Machine terminates. The shutdown hook below closes the connection. If it fails to close the connection, it silently fails, as shown by the empty catch block. This code can also be added to the Database’s constructor or intialization method.

  Thread shutdownThread = new Thread() {
    @Override
    public void run() {
      try {
        conn.close();
      } catch (SQLException e) {}
    }
  };
  Runtime.getRuntime().addShutdownHook(shutdownThread);

Creating Data Recording Methods

Continuing on from the tables created from Part 1, some useful methods to create are: initRun(), logAgent(Agent a) and logAgentState(Agent a).

  public int initRun() {
    int runID;
    Parameters p = RunEnvironment.getInstance().getParameters();

    PreparedStatement s = conn.prepareStatement(
      "INSERT INTO tblRun(rundate, param1, param2) " +
      "VALUES (?, ?, ?)", PreparedStatement.RETURN_GENERATED_KEYS);
    s.setTimestamp(1, new Timestamp(System.currentTimeMillis()));
    s.setDouble(2, (Double) p.getValue("param1"));
    s.setDouble(3, (Double) p.getValue("param2"));

    s.executeUpdate();

    // get the auto-generated runID
    ResultSet rs = s.getGeneratedKeys();
    if (rs.next()) {
      runID = rs.getInt(1);
    } else {
      runID = -1; // something went wrong: abort or use some default
    }
    s.close();
    return runID;
  }

I’ll leave the other methods unimplemented. They are quite similar to the initRun() method, except instead of pulling parameters from the RunEnvironment, agent attributes are retrieved using get methods. A useful method for getting the current tick is:

  RunState.getInstance().getScheduleRegistry()
    .getModelSchedule().getTickCount();

Maybe there’s a shorter way of doing that.

Using the Database class

Every class (agent, context, etc.) that wishes to write to the database will need a reference to the Database instance. You have a few options to handle that.

1. Pass a reference of the Database instance to every agent.
2. Use the singleton design pattern to allow any Java object to get the reference to the instance.
3. Use Simphony’s general registry to store the Database instance.

With the first approach, every agent writing to the database must have a reference to the Database instance. That could introduce a lot of overhead if there are many agents. (Alternatively, you could set up database logging from the parent context only, in which case each agent would not require a reference to the Database instance.

I prefer to use the singleton pattern in this case, since there is less overhead than in the retrieval of the instance. With the singleton pattern, the instance of the database is stored in a static field in the Database class. Outside classes cannot create new instances of the database, but may only retrieve the instance referenced by the Database class. This ensures that one and only one Database instance exists.

To use the singleton pattern, make the Database’s constructor(s) a private method and put this line inside the Database class definition:

  private static final Database instance = new Database();

Finally, make a new method inside Database to retrieve the singleton instance:

  public static Database getInstance() {
    return instance;
  }

Now that the database is set up as a singleton, any class may make reference to the singleton instance by calling Database.getInstance(). If I have an agent wishing to log its own data, I could call:

  Database.getInstance().logAgentState(this);

If the singleton pattern isn’t appropriate, you can use the general registry. On model startup, say in the master context’s build method, load a new instance of the Database and put it into the registry.

  Database db = new Database();
  RunState.getInstance().addToRegistry("database", db);

When an agent needs to read from the database, retrieve its instance from the registry.

  Database db = RunState.getInstance().getFromRegistry("database");

On model termination, close the Database connection and remove it from the registry.

  Database db = (Database) RunState.getInstance()
    .removeFromRegistry("database");
  db.close();

Installing and Configuring MySQL

Download and install the latest version of MySQL Community Server (currently at version 5.1) appropriate to your platform. After installation, the Server Instance Config Wizard should load. Selecting most of the defaults throughout the wizard should suffice for a typical development workstation. At the very least, you should set a root password and, if desired, start the MySQL server upon startup as a service. This will save you from restarting the MySQL server whenever the computer reboots. Also download and install the MySQL GUI Tools.

(Bonus points: Start MySQL Administrator and create a limited user account with SELECT and INSERT access to your database only. This account will be used within your model.)

Creating the database schema

A relational database is arranged into tables, where each row is a records. Generally, you can have a table listing the runs and their parameters, a table of agents and their initial and constant attributes, and a table of agent states which contains a record at each tick or at each change in state.

I personally prefer to create the database schema prior to running the model. It’s also possible to create tables if they don’t exist on model start-up.

Start MySQL Query Browser (part of MySQL Gui Tools which was installed earlier). Log in as root with the password specified in the config wizard. Create a new database schema by right-clicking in the Schemata area and selecting Create New Schema. Name it after your model. In my case, my schemata is named maria. You’ll later connect to your database with the URL jdbc:mysql://localhost:3306/[schema_name].

Once your schema is created, you’re ready to create some tables. Right-click on your new schema and select Make Default Schema, to save yourself from re-typing your schema name from now on. Starting with the run table, some useful columns would be a run ID and the input parameters.

CREATE TABLE runs (
  runID INT NOT NULL AUTO_INCREMENT,
  rundate DATETIME DEFAULT NULL,
  param1 DOUBLE NOT NULL,
  param2 DOUBLE NOT NULL,
  PRIMARY KEY (`runID`)
)

What does all of the above mean? The above statement creates a table named runs with columns containing an automatically generated run ID, a run date, and 2 double-precision floating-point parameters. The NOT NULL constraint in each column specifies that these blanks cannot be null or blank. (Optionally, a default value may be specified after the NOT NULL constraint.) The PRIMARY KEY clause at the end of the statement indicates that runID should be used to uniquely identify each row in the table. Check the MySQL documentation on the CREATE TABLE if you want details.

A second table lists all of the agents, in which a row is created when an agent is created. You may want a separate table for each agent type (say, a mosquito table and a person table), but I’ll just list a single agent table below. You can also add the agent’s initial conditions here.

CREATE TABLE agents (
  agentID INT NOT NULL,
  runID INT NOT NULL,
  PRIMARY KEY (agentID ,runID),
  FOREIGN KEY (runID) REFERENCES runs (runID) ON DELETE CASCADE
)

The above table is similar to the first, except for the addition of a FOREIGN KEY. The foreign key references the runs table. Specifically, it links the runID column of the agents table to the runID column of the runs table. When inserting a row into the agent table, the runID must also exist in the runs table. The ON DELETE CASCADE

I like to use a third type of table to store agent states. It is like the above table, with columns for the agent ID, run ID and any attributes related to the agent state, plus the addition of a tick or step column.

Next post, I’ll discuss how to integrate the database into the model.

I don’t have a shapefile of agents, but I do have an ASCII grid to load into a GIS projection. (Much of my model is based on cellular automata, where households are placed into the model by assigning them to a grid cell.) Fortunately, the grid is georeferenced with a .prj projection file. The grid uses the SAD69 projection and World Wind uses WGS84. Need to convert from SAD69 (or any other CRS) to WGS84? Not a problem.

Using OpenGIS and Geotools (both included in Simphony’s referenced libraries), you can read an ArcMap projection file (which is just an ASCII file containing well-known text) to get a CoordinateReferenceSystem object. From that, you can find a transformation to the CoordinateReferenceSystem of your target GIS projection. That’s just what this code snippet does, for a target GIS projection, myGeography:

CoordinateReferenceSystem sourceCRS =
    CRS.parseWKT(new BufferedReader(
        new FileReader(projectionFile)).readLine());
GeometryFactory fac = new GeometryFactory();
MathTransform mf =
    CRS.findMathTransform(sourceCRS, myGeography.getCRS());

Using the GeometryFactory and MathTransform, SAD69-referenced agents can be added to the WGS84-referenced GIS projection. You will need to specify (x,y) coordinates as well as the lower-left coordinates and cellsize of the source ASCII grid.

com.vividsolutions.jts.geom.Point point = fac.createPoint(
    getCoordinates(x, y, originx, originy, cellsize));
Geometry geom = JTS.transform(point, mf);
// or: Geometry geom = WWUtils.projectGeometryToWGS84(point, sourceCRS);
personGeography.move(h, geom);

The getCoordinates method referenced above (and shown below) converts grid coordinates to easting/northing. This code assumes that originx and originy refer to the coordinates of the lower-left corner of the grid (indicated by xllcorner and yllcorner in the ASCII grid header), hence the addition of 0.5 to x and y to offset the coordinate to the center of the cell. Alternatively, if your ASCII grid header uses xllcenter and yllcenter, you shouldn’t add 0.5.

public Coordinate getCoordinates(int x, int y,
        double originx, double originy, double cellSize) {
    double easting = (x + 0.5) * cellSize + originx;
    double northing = (y + 0.5) * cellSize + originy;
    return new Coordinate(easting, northing);
}

If you get a warning about missing Bursa Wolf parameters, there’s some extra work to be done. You’ll need to add a TOWGS84 parameter to your projection file. A guide on how to do that is summarized well on this blog post (external link).

	public Context build(Context context) {
		/* ... master context initialization ... */

		context.addSubContext(new SubContextA("SubContextA"));
		context.addSubContext(new SubContextB("SubContextB"));

		return context;
	}

SubContextA and SubContextB should be in your model.score file as sub-contexts of your root context. After that, make sure sure you add Data Loaders for these sub-contexts from the Repast Simphony Scenario Tree. (I won’t explain that in this post, but if there’s interest, I’ll make a later post about creating a Repast Simphony model from scratch.)

There’s a hint in the Simphony reference documentation, which probably wasn’t there when I started the model.