Thoughts by Techxplorer

By: t0msk

The past week or so has seen me continuing my work on exporting a subset of the AusStage dataset into an RDF based datastore using Jena and TDB for the Navigating Networks Service. It’s been a frustrating time as I’ve been working my way through issues related to memory management and garbage collection in Java.

My essential problem was that I kept running out of memory that my application could use. More specifically I was running out of heap space. The purpose of this post is to reflect on some of the lessons that I’ve learned so far.

My first way of reducing the amount of memory required by my application was to switch from using Java Collections to the manage a set of Jena Resource objects. I maintain this list to ensure that any subsequent data that I add to the datastore relates to a contributor that I’ve already processed.

This is necessary because updates to the AusStage data can, and do, happen at any time. To reduce the amount of memory consumed by my application I’ve switched to maintaining a set of integers, representing the unique identifiers or contributors, using the GNU Trove library. Rather than store Jena Resource objects I use the Model.createResource() method to create new resources and return instances of resources that have already been added to the RDF model.

I also optimised my SQL queries to return the exact datasets that I require, and with the assistance of one of my colleagues use SQL to de-duplicate the list of relationships between contributors. Trying to do this in my Java code was causing too much memory to be consumed.

Lastly, after sending a few messages to the Jena-dev mailing list I tweaked some other aspects of the code and gained a great understanding of what was happening. Many thanks to those who responded.

All of this means that my code now successfully runs again, but this is not to say that it doesn’t require a large amount of memory. As can be seen from the instructions for the RdfExport application stored in our Wiki on the Aus-e-Stage project pages the application does require 2 gigabytes of heap space. This necessitated moving to the 64-bit version of Ubuntu and away from the 32-bit version so that I could use the 64bit version of the Java Virtual Machine.

The majority of this space is used when collaborations, the resources representing a collaboration between two contributors, are added to the datastore. At the moment there are in excess of 1.6 million unique collaborations represented in the AusStage database. An attribute of a collaboration resource is how many times the two contributors have collaborated together has occurred. This means that the actual number of times that two contributors have worked together is significantly more than this.

Using the VisualVM application as suggested by a message posted to the mailing list I can see when my code starts to add collaborations and the impact this has on heap space.

Heap Allocation Graph (Click Image for Larger Version)

On the right hand side you can see how the amount of allocated heap space and the amount of heap used increases as the collaboration resources are added. Once all of the resources are added the heap space usage drops off to just under half of the maximum. Without the 2GB of heap space my application would fail with an out of memory error related to the garbage collector spending too much time undertaking garbage collection for little or no heap recovery. This error was usually prefaced with my application slowing to a crawl with what I assume is more and more time spent by the JVM trying to do garbage collection.

The -XX:-UseGCOverheadLimit option was not helpful in this instance as it just prolonged the inevitable.

[Update Monday 19 July 2010]

I forgot to mention that the final change I made to my code was to develop and run it on 64bit operating systems. This is because of the 1.5 – 1.6GB heap space limit that the 32bit version of the JVM has.

[/Update]

By undertaking these changes my application now compiles the dataset using the fields that are currently required. Further changes will need to be made to add some additional fields and manipulate other aspects of the data for use by the Navigating Networks Service.

Some interesting statistics of the current dataset are:

• 81,680 contributors resources
• 101,984 contributor function properties
• 1,640,250 collaboration resources
• 43,706 event resources
• 283,617 contributor functions at events

In total there are 15,924,406 triples in the datastore.

The photo “244/365 Six Rocks” photos was uploaded to Flickr by t0msk and used under the terms of a Creative Commons License.

By: Proggie

My last few posts have focused on my use of Jena as a means of constructing a RDF based dataset using data sourced from the AusStage relational database for use with the Navigating Networks component of the Aus-e-Stage project. I’m taking this approach for two main reasons:

1. It is faster to query than the relational database
2. It gives us experience in using RDF and related technologies for use in other aspects of the AusStage project

One of the primary key components of the dataset is the relationships between contributors. A contributor is someone who has in some way contributed to an event. A relationship is deemed to exist where two contributors have contributed to the same event. For the purposes of this discussion I’m calling this a collaboration.

We use the FOAF ontology to store basic biographical information about contributors and we were using the foaf:knows term to model the relationships between contributors. Continued development of the Networking Service Goals and the Networking Service Specification have shown that the foaf:knows term isn’t sufficient to model the complex data related to a collaboration. One of the properties of interest is the number of times that two contributors have collaborated on the same event and the FOAF ontology doesn’t provide a mechanism for modelling this type of information.

In conjunction with the development of the dataset I’ve been working on the basis of an Aus-e-Stage ontology. This ontology allows me to model the data required for the Navigating Networks service where existing ontologies are not available.

Please Note that this ontology is only meant as a stopgap measure and should in no way be seen as pre-empting the work that will be conducted as part of another AusStage related project in flattening the entire AusStage dataset for periodic export into a data repository.

The ontology provides four terms related to collaborations which are:

1. collaboration – used to store details of a collaboration relationship
2. collaborator -  a reference to a contributor involved in this collaboration (appears twice, one for each contributor)
3. collaborationCount – the number of times the two contributors have collaborated
4. hasCollaboration – stores a reference to a collaboration with a contributor

An example of two contributors and their collaboration is shown in the following snippet of Notation3 code below.

@prefix time:    <http://www.w3.org/2006/time#> .
@prefix tl:      <http://purl.org/NET/c4dm/timeline.owl#> .
@prefix foaf:    <http://xmlns.com/foaf/0.1/> .
@prefix event:   <http://purl.org/NET/c4dm/event.owl#> .
@prefix xsd:     <http://www.w3.org/2001/XMLSchema> .
@prefix ausestage:  <http://code.google.com/p/aus-e-stage/wiki/AuseStageOntology#> .
@prefix dcterms:  <http://purl.org/dc/terms/> .

<ausstage:c:261085>
      a       foaf:Person ;
      ausestage:collaboratorCount
              "17" ;
      ausestage:function "Circus Style Performer" ;
      ausestage:hasCollaboration
              <ausstage:rel:261085-261086> ;
      ausestage:nationality
              "Australian" ;
      foaf:familyName "Ristic" ;
      foaf:givenName "Denjan" ;
      foaf:name "Denjan Ristic" ;
      foaf:page "http://www.ausstage.edu.au/indexdrilldown.jsp?xcid=59&f_contrib_id=261085" .

<ausstage:c:261086>
      a       foaf:Person ;
      ausestage:collaboratorCount
              "17" ;
      ausestage:function "Circus Style Performer" ;
      ausestage:hasCollaboration
              <ausstage:rel:261085-261086> ;
      ausestage:nationality
              "Australian" ;
      foaf:familyName "Sanders" ;
      foaf:givenName "D'Nash" ;
      foaf:name "D'Nash Sanders" ;
      foaf:page "http://www.ausstage.edu.au/indexdrilldown.jsp?xcid=59&f_contrib_id=261086" .

<ausstage:rel:261085-261086>
      a       ausestage:collaboration ;
      ausestage:collaborationCount
              "1" ;
      ausestage:collaborator
              <ausstage:c:261085> , <ausstage:c:261086> .

In the above code there is:

• A foaf:person with the unique identifier ausstage:c:261085
• A second foaf:person with the unique identifier ausstage:c:261086
• A ausestage:collaboration linked to the contributors

The unique identifiers are encoded using this URI scheme which provides a way to easily distinguish between identifiers that are used for contributors, events, and other internal objects such as these collaboration relationships.

Using this type of model to record collaborations is extensible and additional information can be included. One data element that is of particular interest is the time span of the collaboration relationship as determined by the dates associated by the events that form the collaboration. For example if the first event that was part of the collaboration started on 01/01/2010 and the last event finish on 01/06/2010 it could be said that the collaboration relationship lasted 6 months.

The dataset, and by extension the data model, continue to evolve as more is understood about RDF and the supporting technologies as well as the continued development of the Navigating Networks Service goals and specification.

The photo “Contortion” was uploaded to Flickr by Proggie and used under the terms of a Creative Commons license.