<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	xmlns:georss="http://www.georss.org/georss" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:media="http://search.yahoo.com/mrss/"
	>

<channel>
	<title>Thoughts by Techxplorer</title>
	<atom:link href="http://techxplorer.com/feed/" rel="self" type="application/rss+xml" />
	<link>http://techxplorer.com</link>
	<description>Thoughts on my experiences with technology</description>
	<lastBuildDate>Fri, 10 Sep 2010 00:32:10 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.com/</generator>
<cloud domain='techxplorer.com' port='80' path='/?rsscloud=notify' registerProcedure='' protocol='http-post' />
<image>
		<url>http://1.gravatar.com/blavatar/5dd756fac29d9d9db0ec158a09916957?s=96&#038;d=http://s2.wp.com/i/buttonw-com.png</url>
		<title>Thoughts by Techxplorer</title>
		<link>http://techxplorer.com</link>
	</image>
	<atom:link rel="search" type="application/opensearchdescription+xml" href="http://techxplorer.com/osd.xml" title="Thoughts by Techxplorer" />
	<atom:link rel='hub' href='http://techxplorer.com/?pushpress=hub'/>
		<item>
		<title>Integrating AusStage with SMS Messages</title>
		<link>http://techxplorer.com/2010/09/10/integrating-ausstage-with-sms-messages/</link>
		<comments>http://techxplorer.com/2010/09/10/integrating-ausstage-with-sms-messages/#comments</comments>
		<pubDate>Fri, 10 Sep 2010 00:32:10 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[ausstage]]></category>
		<category><![CDATA[mobile]]></category>
		<category><![CDATA[sms]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1696</guid>
		<description><![CDATA[Yesterday I wrote about how we&#8217;ve integrated AusStage with Twitter so that we can use the Mobile Service to collect live audience participation feedback using the Mobile Service. These services are being developed as part of the Aus-e-Stage project, which is part of the larger AusStage project, and are components of the Audience Participation Feedback [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1696&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1703" class="wp-caption alignright" style="width: 190px"><a href="http://www.flickr.com/photos/eliacimlopes/392564107/"><img class="size-full wp-image-1703 " title="sms" src="http://techxplorer.files.wordpress.com/2010/09/sms.jpg?w=180&#038;h=240" alt="" width="180" height="240" /></a><p class="wp-caption-text">By: Eliacim Lopes</p></div>
<p>Yesterday I wrote about how we&#8217;ve integrated <a title="Direct link to the previous post" href="/2010/09/09/integrating-ausstage-with-twitter/">AusStage with Twitter</a> so that we can use the Mobile Service to collect live audience participation feedback using the <a title="More information about the mobile service" href="http://code.google.com/p/aus-e-stage/wiki/MobileService">Mobile Service</a>. These services are being developed as part of the <a title="More information about the Aus-e-Stage project" href="http://code.google.com/p/aus-e-stage/wiki/StartPage">Aus-e-Stage project</a>, which is part of the larger <a title="Official Joseki website" href="http://www.ausstage.edu.au/">AusStage project</a>, and are components of the Audience Participation Feedback service.</p>
<p>In this post I&#8217;m going to focus on how we&#8217;ve integrated AusStage with <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/SMS">Short Message Service</a> (SMS) messages. SMS messages are one of the three ways that we plan on gathering feedback which are:</p>
<ol>
<li><a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/SMS">Short Message Service</a> – every mobile phone has at least this capability</li>
<li><a title="Official Twitter Homepage" href="http://twitter.com/">Twitter</a> – users of twitter can provide feedback using Twitter from their smartphone or other mobile device</li>
<li><a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Mobile_Web">Mobile Web</a> – users of smartphones or other mobile devices can use a website optimised for mobile devices to submit feedback</li>
</ol>
<p>The components that integrated AusStage with SMS messages are as follows:</p>
<ol>
<li>A <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Modem#Mobile_modems_and_routers">Mobile Modem</a> &#8211; used to connect to the <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Mobile_phone_network">mobile phone network</a></li>
<li>A script on the AusStage server that interfaces with the modem to send and receive SMS messages</li>
<li>The <a title="More information about the Mobile Service API" href="http://code.google.com/p/aus-e-stage/wiki/MobileServiceAPIGatherer">Input Gatherer API</a> of the Mobile Service &#8211; processes incoming SMS messages and stores them in the AusStage database</li>
</ol>
<p>The workflow for processing SMS messages is outlined in the diagram below:</p>
<div id="attachment_1700" class="wp-caption aligncenter" style="width: 310px"><a href="http://techxplorer.files.wordpress.com/2010/09/incoming-sms-workflow.png"><img class="size-medium wp-image-1700" title="incoming-sms-workflow" src="http://techxplorer.files.wordpress.com/2010/09/incoming-sms-workflow.png?w=300&#038;h=268" alt="" width="300" height="268" /></a><p class="wp-caption-text">Incoming SMS Message workflow <br />(Click for larger version)</p></div>
<p>Importantly the mobile phone number of the audience member is obfuscated using  a <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/SHA-2">cryptographic hash function</a>. In this way no personally identifying information is kept in the AusStage database or in any related log files. By using a hash it is possible to group messages together in order to identify the same audience member has sent in multiple pieces of feedback, however it is not possible to personally identify them.</p>
<p>In the above workflow an exception report is generated when no matching  performance can be found in the AusStage database. An exception report  can also be generated as a result of an error such as the database not  being available. In this case an email is sent which contains a brief message about  the error that occurred and a copy of the message and associated metadata. This will provide  sufficient information for a member of the AusStage team to take  remedial action as required. For example adding the message text and  associated metadata in manually once the database becomes available.</p>
<p>Once data is stored in the AusStage database it can be used for things  such as data analysis or displayed on a web page that would provide live  feedback at the venue using part of the <a title="More information about the Mobile Service API" href="http://code.google.com/p/aus-e-stage/wiki/MobileServiceAPI">Mobile Service API</a> that is currently under development.</p>
<p>The photo &#8220;<a title="Direct link to the photo" href="http://www.flickr.com/photos/eliacimlopes/392564107/">SMS</a>&#8221; was uploaded to <a title="Flickr homepage" href="http://www.flickr.com/">Flickr</a> by <a title="Flickr profile page" href="http://www.flickr.com/people/eliacimlopes/">Eliacim Lopes</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by-nc/2.0/deed.en">Creative Commons License</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1696/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1696/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1696/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1696/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1696/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1696/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1696/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1696/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1696/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1696/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1696/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1696/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1696/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1696/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1696&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/09/10/integrating-ausstage-with-sms-messages/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/09/sms.jpg" medium="image">
			<media:title type="html">sms</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/09/incoming-sms-workflow.png?w=300" medium="image">
			<media:title type="html">incoming-sms-workflow</media:title>
		</media:content>
	</item>
		<item>
		<title>Integrating AusStage with Twitter</title>
		<link>http://techxplorer.com/2010/09/09/integrating-ausstage-with-twitter/</link>
		<comments>http://techxplorer.com/2010/09/09/integrating-ausstage-with-twitter/#comments</comments>
		<pubDate>Thu, 09 Sep 2010 04:45:34 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[ausstage]]></category>
		<category><![CDATA[mobile]]></category>
		<category><![CDATA[twitter]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1685</guid>
		<description><![CDATA[One of the three services that I&#8217;m working here at the Aus-e-Stage project, part of the larger AusStage project, is the Audience Participation Feedback service. This service will use mobile technologies to gather feedback from audience participants. For this reason we call it the Mobile Service for short. A key component of this service is [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1685&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1694" class="wp-caption alignright" style="width: 250px"><a href="http://www.flickr.com/photos/batmoo/3734837951/"><img class="size-full wp-image-1694" title="humanities-theatre" src="http://techxplorer.files.wordpress.com/2010/09/humanities-theatre.jpg?w=240&#038;h=158" alt="" width="240" height="158" /></a><p class="wp-caption-text">By: Mohammad Jangda</p></div>
<p>One of the three services that I&#8217;m working here at the <a title="More information about the Aus-e-Stage project" href="http://code.google.com/p/aus-e-stage/wiki/StartPage">Aus-e-Stage project</a>, part of the larger <a title="Official Joseki website" href="http://www.ausstage.edu.au/">AusStage project</a>, is the Audience Participation Feedback service. This service will use mobile technologies to gather feedback from audience participants. For this reason we call it the <a title="More information about the mobile service" href="http://code.google.com/p/aus-e-stage/wiki/MobileService">Mobile Service</a> for short. A key component of this service is using mobile devices to gather feedback from audience participants either during, or  within two hours of the performance finishing.</p>
<p>Feedback will be gathered using three different mechanisms and they are:</p>
<ol>
<li><a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/SMS">Short Message Service</a> &#8211; every mobile phone has at least this capability</li>
<li><a title="Official Twitter Homepage" href="http://twitter.com/">Twitter</a> &#8211; users of twitter can provide feedback using Twitter from their smartphone or other mobile device</li>
<li><a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Mobile_Web">Mobile Web</a> &#8211; users of smartphones or other mobile devices can use a website optimised for mobile devices to submit feedback</li>
</ol>
<p>This post will focus on the second feedback mechanism and outline how AusStage is integrated with Twitter.</p>
<p>The key to the integration is the <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Tag_%28metadata%29#Hash_tags">hash tag</a>. In essence hash tags, or hashtags depending on your preference, are defined at the <a title="Hashtags page on the Twitter Fan Wiki" href="http://twitter.pbworks.com/Hashtags">Twitter Fan Wiki</a> as a</p>
<blockquote><p>community-driven convention for adding additional context and metadata  to your tweets. They&#8217;re like tags on Flickr, only added inline to your  post. You create a hashtag simply by prefixing a word with a hash  symbol: <em>#hashtag</em>.</p></blockquote>
<p>Our system uses hash tags to identify twitter messages, or tweets as they&#8217;re known, that are pieces of feedback using hash tags that are associated with our partner organisations.</p>
<p>The integration of AusStage with the Twitter service is achieved using our <a title="More information about the application" href="http://code.google.com/p/aus-e-stage/wiki/TwitterGatherer">TwitterGatherer application</a> that I&#8217;ve developed. This application is written in Java and uses the <a title="More information about this library" href="http://github.com/seejohnrun/tweetStream4J">tweetStream4J</a> library to access the <a title="More information about this API" href="http://dev.twitter.com/pages/streaming_api">Twitter Streaming API</a> and the <a title="More information about this library" href="http://github.com/mzsanford/twitter-text-java">twitter-text-java</a> library to parse messages and retrieve a list of hash tags.</p>
<p>The workflow for the TwitterGatherer application is outlined in the diagram below:</p>
<div id="attachment_1689" class="wp-caption aligncenter" style="width: 310px"><a href="http://techxplorer.files.wordpress.com/2010/09/twitter-gatherer-workflow.png"><img class="size-medium wp-image-1689" title="twitter-gatherer-workflow" src="http://techxplorer.files.wordpress.com/2010/09/twitter-gatherer-workflow.png?w=300&#038;h=257" alt="" width="300" height="257" /></a><p class="wp-caption-text">TwitterGatherer Workflow <br /> (Click for larger version)</p></div>
<p>Importantly no identifying information is stored in either the log files or the AusStage database. The list of changes made to a tweet as a means of de-identifying users is <a title="More information on the de-identification steps" href="http://code.google.com/p/aus-e-stage/wiki/TwitterGatherer#De-Identifying_Information">outlined here</a>.</p>
<p>In the above workflow an exception report is generated when no matching performance can be found in the AusStage database. An exception report can also be generated as a result of an error such as the database not being available. In this case the email contains a brief message about the error that occurred and a copy of the log file. This will provide sufficient information for a member of the AusStage team to take remedial action as required. For example adding the message text and associated metadata in manually once the database becomes available.</p>
<p>Importantly at no time in the process is personally identifying information either stored in the log files, the database or in the exception reports.</p>
<p>Once data is stored in the AusStage database it can be used for things such as data analysis or displayed on a web page that would provide live feedback at the venue using part of the <a title="More information about the Mobile Service API" href="http://code.google.com/p/aus-e-stage/wiki/MobileServiceAPI">Mobile Service API</a> that is currently under development.</p>
<p>The &#8220;<a title="Direct link to the photo" href="http://www.flickr.com/photos/batmoo/3734837951/">Audience at Humanities Theatre</a>&#8221; photo was uploaded to <a title="Flickr homepage" href="http://www.flickr.com/">Flickr</a> by <a title="Flickr profile page" href="http://www.flickr.com/people/batmoo/">Mohammad Jangda</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by-sa/2.0/deed.en">Creative Commons License</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1685/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1685/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1685/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1685/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1685/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1685/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1685/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1685/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1685/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1685/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1685/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1685/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1685/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1685/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1685&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/09/09/integrating-ausstage-with-twitter/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/09/humanities-theatre.jpg" medium="image">
			<media:title type="html">humanities-theatre</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/09/twitter-gatherer-workflow.png?w=300" medium="image">
			<media:title type="html">twitter-gatherer-workflow</media:title>
		</media:content>
	</item>
		<item>
		<title>Reflections on THATCamp Canberra</title>
		<link>http://techxplorer.com/2010/09/06/reflections-on-thatcamp-canberra/</link>
		<comments>http://techxplorer.com/2010/09/06/reflections-on-thatcamp-canberra/#comments</comments>
		<pubDate>Mon, 06 Sep 2010 04:27:40 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[ausstage]]></category>
		<category><![CDATA[thatcamp]]></category>
		<category><![CDATA[canberra]]></category>
		<category><![CDATA[unconference]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1672</guid>
		<description><![CDATA[On the weekend of the 28th and 29th of August I was fortunate to be able to attend the first THATCamp in Australia, THATCamp Canberra. It was the first unconference that I&#8217;ve ever attended. I wasn&#8217;t sure what to expect but was determined to attend with an open mind and a willingness to participate. I [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1672&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1679" class="wp-caption alignright" style="width: 250px"><a href="http://www.flickr.com/photos/ms101/524827297/"><img class="size-full wp-image-1679" title="parliament-house-canberra" src="http://techxplorer.files.wordpress.com/2010/09/parliament-house-canberra.jpg?w=240&#038;h=160" alt="" width="240" height="160" /></a><p class="wp-caption-text">By: Matt Stewart</p></div>
<p>On the weekend of the 28th and 29th of August I was fortunate to be able to attend the first <a title="Official THATCamp website" href="http://thatcamp.org/">THATCamp</a> in Australia, <a title="Official THATCamp Canberra website" href="http://thatcampcanberra.org/">THATCamp Canberra</a>. It was the first <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Unconference">unconference</a> that I&#8217;ve ever attended.</p>
<p>I wasn&#8217;t sure what to expect but was determined to attend with an open mind and a willingness to participate. I was struck by the similar feelings of the other participants. From the first group meeting where scheduling for the two days occurred through to the individual sessions and the bootcamp sessions as well. There was always a willingness to share, collaborate and discuss, even if discussions got a little passionate during some sessions.</p>
<p>I didn&#8217;t realise it at the time but my suggestion of a session on <a title="Direct link to the session proposal" href="http://thatcampcanberra.org/2010/08/exploring-semantic-technologies-in-the-humanties/">Exploring the Semantic Technologies in the Humanities</a> was to prove a popular session and one that I would need to facilitate. This caught me by surprise and I hope the participants of the session were pleased with the direction that it took.</p>
<p>One of the main themes from the session was a sense that those of us using these types of technologies are all working through similar issues with regards to data formats, selection of ontologies, building our own ontologies and exploring the potentials of the technologies.</p>
<p>During the discussions an <a title="Direct link to the THATCamp Canberra post" href="http://thatcampcanberra.org/2010/08/back-to-the-future-unpacking-the-semantic-web/">interesting link to a paper</a> from 2000 highlighted that many of the issues that we are working through are similar to those of 10 years ago. Showing that while technologies improve and change the underlying questions, assumptions, and goals remain the same. Another useful outcome from this session was the start of a document that listed all of the <a title="Direct link to the THATCamp Canberra post" href="http://thatcampcanberra.org/2010/08/who-is-working-on-semnatic-web-projects/">semantic web projects</a> that we are working on.</p>
<p>During the two days it became clear to me that <a title="Official Twitter Website" href="http://twitter.com">Twitter</a> was an invaluable tool for <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Backchannel">backchannel</a> communication. We all used the same <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Tag_%28metadata%29#Hash_tags">hashtag</a> #thatcamp which means that it is possible to <a title="Direct link to an archive of Tweets sent during the unconference" href="http://twapperkeeper.com/hashtag/thatcamp?sm=8&amp;sd=28&amp;sy=2010&amp;em=8&amp;ed=29&amp;ey=2010&amp;o=a&amp;l=10&amp;from_user=&amp;text=&amp;lang=">group all of our messages together</a> using a service such as <a title="Official Twapper Keeper homepage" href="http://twapperkeeper.com/">Twapper Keeper</a>. My <a title="Direct link to the feed from my Twitter account" href="http://twitter.com/techxplorer">own tweets</a> are an eclectic mix of notes on discussions, thoughts for future consideration, and topics to explore.</p>
<p>The other sessions I attended included:</p>
<p>A session on <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Linked_Data">linked data</a> where one participant discussed a current project to make there data available in a variety of different formats to promote collaboration and sharing in the form of <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Mashup_%28web_application_hybrid%29">mashups</a>. What was surprising about this project is that they used screen scraping techniques to convert their HTML based website into data available for sharing because it was easier than trying to interface directly with the backend systems.</p>
<p>Some interesting thoughts from this session are:</p>
<ul>
<li><a title="Official OpenSearch website" href="http://www.opensearch.org/Home">OpenSearch</a> could provide a useful means of sharing data and search results with others that may be useful in the <a title="Official AusStage website" href="http://www.ausstage.edu.au">AusStage</a> website redevelopment</li>
<li>It is easy to focus on the data and lose sight of the documentation and other materials that users may require</li>
<li><a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Persistent_Uniform_Resource_Locator">Persistent URLs</a> are key to any linked data project as it provides a reliable way for users to build on your dataset by including reliable links to it from their own systems</li>
</ul>
<p>A session on crowd sourcing data and participation highlighted a number of very interesting projects including</p>
<ul>
<li><a title="More information about this project" href="http://www.invisibleaustralians.org/">Invisible Australians</a> &#8211; Living under the White Australia Policy</li>
<li><a title="More information about this project" href="http://www.londonlives.org/">London Lives</a> &#8211; 1690 to 1800 &#8211; Crime, Poverty and Social Policy in the Metropolis</li>
<li><a title="More information about this project" href="http://www.foundersandsurvivors.org/">Founders &amp; Survivors</a> &#8211; Australian Life Courses in historical context 1803 to 1920</li>
</ul>
<p>During this session there was a discussion about project scope and managing the desire to have a large project with the reality of resource constraints such as time and money. In this middle of this discussion one of the participants had one of the best quotes of the day namely:</p>
<blockquote><p>one way to move forward is to narrow, another is to make the idea bigger which opens up bigger possibilities</p></blockquote>
<p>I finished the day with a session exploring the topic of what a digital humanist needs to know in relation to technology. This session proved to have some of the most lively and passionate discussion of the day and highlighted a number of issues facing digital humanists. Including the issue of managing the needs to use and understand technologies whilst at the same time maintaining focus on the research and those elements that make a researcher a humanist.</p>
<p>The discussion predominantly focused on the issue from a researchers point of view. That is, they have the research skills but not necessarily the technology skills. My own individual focus is the reverse. I have the technology skills but not the research skills. I&#8217;m still not sure how to bridge the gap.</p>
<p>The first session I attended on Sunday was on mapping and this session highlighted many of the issues that we&#8217;re working through as part of the development for the <a title="Direct link to the mapping service" href="http://beta.ausstage.edu.au/mapping/">Mapping Service</a> as part of the <a title="Aus-e-Stage project website" href="http://beta.ausstage.edu.au/">Aus-e-Stage project</a>. One thing to keep in mind is that even small datasets, when approached from the perspective of a map, can provide insight and answers to research questions. Not every mapping activity requires large and complex systems.</p>
<p>It was also useful to see that others who have worked on these types of systems have grappled with similar issues to those that we&#8217;re working through with our own development.</p>
<p>The final session was nominally on digital space and was a wide ranging discussion. The main themes that I noted in my Twitter Stream included:</p>
<ul>
<li>Consideration for the social justice issues in providing services that require the latest smartphones / tablet devices</li>
<li>The impact that these services have on the way we view the environment around us</li>
<li>Now that smartphones have GPS capabilities, is it necessary to create an explicit link between the physical and digital worlds using technologies such as <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/QR_Code">QR Codes</a>. Or is location simply enough?</li>
<li>how do we visualise the virtual space that we create as we move through our lives in both the physical and virtual world?</li>
</ul>
<p>It was an exhausting and rewarding weekend and one that gave me plenty to think about.</p>
<p>My thanks and congratulations to the organisers of the THATCamp and my thanks also to my fellow participants without whom the experience wouldn&#8217;t have been so positive.</p>
<p>The &#8220;<a title="Direct link to the photo" href="http://www.flickr.com/photos/ms101/524827297/">Parliament House, Canberra, Australia</a>&#8221; photo was uploaded to <a title="Flickr homepage" href="http://www.flickr.com/">Flickr</a> by <a title="Flickr Profile Page" href="http://www.flickr.com/people/ms101/">Matt Stewart</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by-nc-nd/2.0/deed.en">Creative Commons License</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1672/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1672/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1672/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1672/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1672/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1672/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1672/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1672/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1672/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1672/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1672/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1672/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1672/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1672/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1672&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/09/06/reflections-on-thatcamp-canberra/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/09/parliament-house-canberra.jpg" medium="image">
			<media:title type="html">parliament-house-canberra</media:title>
		</media:content>
	</item>
		<item>
		<title>Exploring twitter-text-java</title>
		<link>http://techxplorer.com/2010/08/16/exploring-twitter-text-java/</link>
		<comments>http://techxplorer.com/2010/08/16/exploring-twitter-text-java/#comments</comments>
		<pubDate>Mon, 16 Aug 2010 02:30:48 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[ausstage]]></category>
		<category><![CDATA[java]]></category>
		<category><![CDATA[twitter]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1666</guid>
		<description><![CDATA[As I mentioned in my last post I&#8217;m currently working on a Java based application to retrieve messages from Twitter that represent feedback sourced from audience participants. This is an important part of infrastructure for the Mobile Service is under development as part of the Aus-e-Stage project which is in turn part of the larger [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1666&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1669" class="wp-caption alignright" style="width: 181px"><a href="http://www.flickr.com/photos/davemott/4368864493/"><img class="size-full wp-image-1669" title="twittdroid" src="http://techxplorer.files.wordpress.com/2010/08/twittdroid.jpg?w=171&#038;h=240" alt="" width="171" height="240" /></a><p class="wp-caption-text">By: Dave Mott</p></div>
<p>As I mentioned in <a title="Direct link to the previous post" href="/2010/08/02/exploring-tweetstream4j/">my last post</a> I&#8217;m currently working on a Java based application to retrieve messages from <a title="Official Twitter Homepage" href="http://twitter.com">Twitter</a> that represent feedback sourced from audience participants. This is an important part of infrastructure for the <a title="More information about the service" href="http://code.google.com/p/aus-e-stage/wiki/MobileService">Mobile Service</a> is under development as part of the <a title="Aus-e-Stage Project Homepage" href="http://beta.ausstage.edu.au/">Aus-e-Stage</a> project which is in turn part of the larger <a title="AusStage Homepage" href="http://www.ausstage.edu.au/">AusStage</a> project.\</p>
<p>The <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Hashtag#Hash_tags">hash tag</a> in the message is one of the pieces of information that is used to match a message to a performance. To make it easier to extract the hash tags from the messages I&#8217;m exploring the use to the <a title="More information about the library" href="http://github.com/mzsanford/twitter-text-java">twitter-text-java</a> library. More information about the library is <a title="Blog post introducing the library" href="http://engineering.twitter.com/2010/02/introducing-open-source-twitter-text.html">available here</a>.</p>
<p>To keep things separate and easy to manage I’ve decided to set aside a  directory specifically for compiling and packaging the library so that I  can use my already established processes with my own preferred tools  for my development.</p>
<p>To do this I undertook the following tasks:</p>
<p><em>Note: This is not intended as a full procedure. It is rather a  list of notes to act as a reminder to me and a prompt to others who may  be experiencing similar issues to those that I had. </em></p>
<p><strong>1. Configuring my System</strong></p>
<ol>
<li>Ensure the <a title="Ubuntu package information" href="http://packages.ubuntu.com/lucid/git-core">git-core</a> package, including dependencies, is installed</li>
<li>Ensure the <a title="More information about the package" href="http://packages.ubuntu.com/lucid/ant1.8">ant1.8</a> package, including dependencies, is installed</li>
</ol>
<p><strong>2. Checkout / Clone the twitter-text-java Repository</strong></p>
<ol>
<li>Create a directory to store files related to development<br />
<code>mkdir mobile-tools</code></li>
<li>Navigate to the newly created directory<br />
<code>cd mobile-tools</code></li>
<li>Clone the twitter-text-java repository<br />
<code>git clone http://github.com/mzsanford/twitter-text-java.git</code></li>
<li>A newly created <code>twitter-text-java</code> directory will be created in the <code>mobile-tools</code> directory that contains the repository</li>
</ol>
<p><strong>3. Checkout / clone the twitter-text-conformance Repository</strong></p>
<ol>
<li>Navigate to the <code>twitter-text-java</code> directory<br />
<code>cd twitter-text-java</code></li>
<li>Navigate to the test-data directory<br />
<code>cd test-data</code></li>
<li>Clone the twitter-text-conformance repository<br />
<code>git clone http://github.com/mzsanford/twitter-text-conformance.git</code></li>
</ol>
<p><strong>4. Compile the twitter-text-java code using ant</strong></p>
<ol>
<li>Navigate to the parent <code>twitter-text-java</code> directory<br />
<code>cd ../../</code></li>
<li>Compile the source code using ant<br />
<code>ant compile</code></li>
</ol>
<p><strong>5. Optionally compile the code and run the tests</strong></p>
<p>At the time of writing the twitter-text-conformance repository contained tests for the ruby implementation that were not supported by the java version of the library. For this reason the tests were skipped in this procedure. To optionally compile the library and execute the tests execute the ant command without any parameters in step 4.2 above.</p>
<p><strong>6. Package the library</strong></p>
<ol>
<li>Ensure you are in the parent <code>twitter-text-java</code> directory</li>
<li>Package the library using ant<br />
<code>ant dist</code></li>
</ol>
<p>The library, packaged as a jar file in the <code>dist</code> directory, can now be included in your own development workflow.</p>
<p>The &#8220;<a title="Direct link to the photo" href="http://www.flickr.com/photos/davemott/4368864493/">TwittDroid</a>&#8221; image was uploaded to <a title="Flickr homepage" href="http://www.flickr.com/">Flickr</a> by <a title="Flickr profile page" href="http://www.flickr.com/people/davemott/">Dave Mott</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by-nd/2.0/deed.en">Creative Commons license</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1666/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1666/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1666/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1666/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1666/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1666/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1666/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1666/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1666/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1666/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1666/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1666/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1666/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1666/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1666&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/08/16/exploring-twitter-text-java/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/08/twittdroid.jpg" medium="image">
			<media:title type="html">twittdroid</media:title>
		</media:content>
	</item>
		<item>
		<title>Exploring tweetStream4J</title>
		<link>http://techxplorer.com/2010/08/02/exploring-tweetstream4j/</link>
		<comments>http://techxplorer.com/2010/08/02/exploring-tweetstream4j/#comments</comments>
		<pubDate>Mon, 02 Aug 2010 06:19:42 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[ausstage]]></category>
		<category><![CDATA[java]]></category>
		<category><![CDATA[twitter]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1651</guid>
		<description><![CDATA[My attention today has been focussed on the third service that is under development as part of the Aus-e-Stage project which is in turn part of the larger AusStage project. The focus of this third service is to use mobile based technologies to solicit feedback in real time from audience members who are attending, or [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1651&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1662" class="wp-caption alignright" style="width: 250px"><a href="http://www.flickr.com/photos/davemott/4257582655/"><img class="size-full wp-image-1662" title="twitter-bling" src="http://techxplorer.files.wordpress.com/2010/08/twitter-bling.jpg?w=240&#038;h=240" alt="" width="240" height="240" /></a><p class="wp-caption-text">By: Dave Mott</p></div>
<p>My attention today has been focussed on the third service that is under development as part of the <a title="Aus-e-Stage Project Homepage" href="http://beta.ausstage.edu.au/">Aus-e-Stage</a> project which is in turn part of the larger <a title="AusStage Homepage" href="http://www.ausstage.edu.au/">AusStage</a> project. The focus of this third service is to use mobile based technologies to solicit feedback in real time from audience members who are attending, or have just attended, a performance.</p>
<p>The two main mechanisms for soliciting feedback I&#8217;m currently focused on are <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/SMS">SMS</a> and <a title="Official Twitter Homepage" href="http://twitter.com/">Twitter</a> messages.</p>
<p>My current task is exploring the way in which we can interact with Twitter in order to retrieve messages, via the <a title="Official Information about the Twitter API" href="http://dev.twitter.com/">Twitter API</a>, that are pieces of feedback for performances that we&#8217;re interested in. All of our development to-date has been in the <a title="Official Java website" href="http://java.com">Java</a> programming language and as such I&#8217;ve been looking into libraries that I can used to make it easier for my development work. The library I&#8217;m starting with the <a title="More information about tweetStream4J" href="http://github.com/seejohnrun/tweetStream4J">tweetStream4J</a> library by <a title="Homepage for Jogn Crepezzi" href="http://johncrepezzi.com/">John Crepezzi</a>.</p>
<p>He uses <a title="Official Git homepage" href="http://git-scm.com/">Git</a> for his source code repository and <a title="Official Apache Maven homepage" href="http://maven.apache.org/index.html">Apache Maven</a> as his build tool. Whereas we&#8217;re using <a title="Official Subversion homepage" href="http://subversion.apache.org/">Apache Subversion</a> and <a title="Official Apache Ant homepage" href="http://ant.apache.org/">Apache Ant</a> respectively. All of our source code is hosted as the <a title="Aus-e-Stage project on Google Code" href="http://code.google.com/p/aus-e-stage/">Aus-e-Stage project on Google Code</a>. To keep things separate and easy to manage I&#8217;ve decided to set aside a directory specifically for compiling and packaging the library so that I can use my already established processes with my own preferred tools for my development.</p>
<p>To do this I undertook the following tasks:</p>
<p><em>Note: This is not intended as a full procedure. It is rather a list of notes to act as a reminder to me and a prompt to others who may be experiencing similar issues to those that I had. </em></p>
<p><strong>1. Configuring my System</strong></p>
<ol>
<li>Install the <a title="Ubuntu package information" href="http://packages.ubuntu.com/lucid/git-core">git-core</a> package including dependencies</li>
<li>Install the <a title="Ubuntu package information" href="http://packages.ubuntu.com/lucid/maven2">maven2</a> package including dependencies</li>
</ol>
<p><strong>2. Checkout / Clone the tweetStream4J Repository</strong></p>
<ol>
<li>Create a directory to store files related to development<br />
<code>mkdir mobile-tools</code></li>
<li>Navigate to the newly created directory<br />
<code>cd mobile-tools</code></li>
<li>Clone the tweetStream4J repository<br />
<code>git clone http://github.com/seejohnrun/tweetStream4J.git</code></li>
<li>A newly created <code>tweetStream4J</code> directory will be created in the <code>mobile-tools</code> directory that contains the repository</li>
</ol>
<p><strong>3. Compile the tweetStream4J code using Maven</strong></p>
<ol>
<li>Navigate to the newly created tweetStream4J directory<br />
<code>cd tweetStream4J</code></li>
<li>Compile the source code using Maven<br />
<code>mvn compile</code></li>
<li>All of the required dependencies will be automatically downloaded into the <code>~/.m2/repository</code> directory</li>
<li>The compiled classes will be in the ./target directory</li>
</ol>
<p><strong>4. Edit the Tests in preparation for packaging</strong></p>
<ol>
<li>Stay in the tweetStream4J directory</li>
<li>Navigate to the following directory<br />
<code>./src/test/java/com/crepezzi/tweetStream4j</code></li>
<li>Open the following file for editing<br />
<code>./APITest.java</code></li>
<li>Locate the following line<br />
﻿﻿﻿﻿
<pre class="brush: java; light: true;">
private static final TwitterStreamConfiguration tws = new TwitterStreamConfiguration(&quot;seejohnrun&quot;, &quot;&quot;);
</pre>
</li>
<li>Change the line so that it includes your Twitter username and password. For example:
<pre class="brush: java; light: true;">
private static final TwitterStreamConfiguration tws = new TwitterStreamConfiguration(&quot;myuser&quot;, &quot;mypass&quot;);
</pre>
</li>
<li>Save the changed file</li>
</ol>
<p><strong>5. Package the tweetStream4J code using Maven</strong></p>
<ol>
<li>Navigate to the main tweetStream4J directory<br />
<code>cd ../../../../../../</code></li>
<li>Package the compiled classes using Maven<br />
<code>mvn package</code></li>
<li>All of the required dependencies will be automatically downloaded into the <code>~/.m2/repository</code> directory</li>
<li>The packaged jar file will be in the <code>./target</code> directory</li>
</ol>
<p><strong>6. Use the Packaged tweetStream4J Library</strong></p>
<ol>
<li>Copy the packaged library to the required location</li>
<li>Ensure the other required dependencies are in your class path or in the same location as the library</li>
<li>At the time of writing the list included:
<ul>
<li><a title="More information about the package" href="http://json-lib.sourceforge.net/">JSON-LIB 2.3</a></li>
<li><a title="More information about the package" href="http://ezmorph.sourceforge.net/">EZMorph 1.0.6</a></li>
<li><a title="More information about the package" href="http://commons.apache.org/beanutils/">Commons &#8211; BeanUtils</a></li>
<li><a title="More information about the package" href="http://commons.apache.org/lang/">Commons &#8211; Lang</a></li>
<li><a title="More information about the package" href="http://commons.apache.org/collections/">Commons &#8211; Collections</a></li>
<li><a title="More information about the package" href="http://commons.apache.org/logging/">Commons &#8211; Logging</a></li>
</ul>
</li>
</ol>
<p><strong>7. Keeping the Source Code Updated</strong></p>
<ol>
<li>Navigate to the mobile-tools directory</li>
<li>Navigate to the tweetStream4J directory<br />
<code>cd tweetStream4J</code></li>
<li>Update the source code with the following command<br />
<code>git pull</code></li>
<li>Repeat steps 4 and 5 above to recompile and package the library</li>
</ol>
<p>The &#8220;<a title="Direct link to the image" href="http://www.flickr.com/photos/davemott/4257582655/">Twitter Bling</a>&#8221; image was uploaded to <a title="Flickr homepage" href="http://www.flickr.com/">Flickr</a> by <a title="Flickr profile page" href="http://www.flickr.com/people/davemott/">Dave Mott</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by-nd/2.0/deed.en">Creative Commons License</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1651/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1651/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1651/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1651/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1651/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1651/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1651/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1651/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1651/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1651/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1651/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1651/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1651/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1651/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1651&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/08/02/exploring-tweetstream4j/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/08/twitter-bling.jpg" medium="image">
			<media:title type="html">twitter-bling</media:title>
		</media:content>
	</item>
		<item>
		<title>Exploring Joseki with AusStage Data</title>
		<link>http://techxplorer.com/2010/07/16/exploring-joseki-with-ausstage-data/</link>
		<comments>http://techxplorer.com/2010/07/16/exploring-joseki-with-ausstage-data/#comments</comments>
		<pubDate>Fri, 16 Jul 2010 05:30:08 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[jena]]></category>
		<category><![CDATA[rdf]]></category>
		<category><![CDATA[sparql]]></category>
		<category><![CDATA[joseki]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1642</guid>
		<description><![CDATA[My focus in the past few weeks, as evidenced by the posts on this blog, has been the construction of an RDF based datastore for use in the development of the Navigating Networks service as part of the Aus-e-Stage project which is part of the larger AusStage project. In particular I&#8217;m using Java with the [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1642&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1644" class="wp-caption alignright" style="width: 250px"><a href="http://www.flickr.com/photos/cybersaphyra/3508644240/"><img class="size-full wp-image-1644" title="SONY DSC" src="http://techxplorer.files.wordpress.com/2010/07/sparkle.jpg?w=240&#038;h=240" alt="" width="240" height="240" /></a><p class="wp-caption-text">By: Deborah  Herbaut</p></div>
<p>My focus in the past few weeks, as evidenced by the posts on this blog, has been the construction of an RDF based datastore for use in the development of the <a title="More information about the service" href="http://code.google.com/p/aus-e-stage/wiki/NetworkService">Navigating  Networks</a> service as part of the <a title="Aus-e-Stage Project  Homepage" href="http://beta.ausstage.edu.au/">Aus-e-Stage</a> project  which is part of the larger <a title="AusStage Homepage" href="http://www.ausstage.edu.au/">AusStage</a> project. In particular I&#8217;m using Java with the <a title="Official Jena website" href="http://openjena.org/">Jena</a>, <a title="Official TDB website" href="http://openjena.org/wiki/TDB">TDB</a> and related libraries.</p>
<p>The RDF based datastore contains a subset of the data available in the AusStage database specifically targeting those fields and relationships needed to meet the required of the Navigating Networks service as outlined in the <a title="Navigating Networks Wiki page" href="http://code.google.com/p/aus-e-stage/wiki/NavigatingNetworksSpecification">Service Specification</a>. Its best to think of it as an alternative index and supplementary datastore. The AusStage relational database remains the primary source of data.</p>
<p>Access to data in the datastore will be made possible via two mechanisms.</p>
<p>The first is under development and will be an <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Application_programming_interface">API</a> that simplifies access to well defined datasets. One of the first datasets under investigation is known as <a title="More information about this dataset" href="http://code.google.com/p/aus-e-stage/wiki/NavigatingNetworksSpecification#2.1_Key_collaborators">Key collaborators</a> with a related dataset known as <a title="More information about this dataset" href="http://code.google.com/p/aus-e-stage/wiki/NavigatingNetworksSpecification#2.2_Key_organisations">Key organisations</a>. The main aim is to provide a mechanism whereby a URL can be constructed that will query that datastore and return the required dataset. Similar to the way the <a title="Exchange Service homepage" href="http://beta.ausstage.edu.au/exchange/">Data Exchange Service</a> currently works.</p>
<p>The second is currently available for testing, is best thought of as experimental at this stage, and is a <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/SPARQL">SPARQL</a> endpoint powered by the <a title="Official Joseki website" href="http://www.joseki.org/">Joseki SPARQL server</a>. The intention behind providing a SPARQL endpoint is to allow ad-hoc querying of the dataset in ways that is not supported by the API. If a certain query, or style of query, is found to be useful it will be included in the API. More information about the <a title="More information about the SPARQL service" href="http://code.google.com/p/aus-e-stage/wiki/NavigatingNetworksSparqlEndpoint">service is available</a> on our wiki as is a <a title="A list of sample queries on our wiki" href="http://code.google.com/p/aus-e-stage/wiki/NavigatingNetworksSparql">few sample queries</a>. If you plan on using the service it is best to become familiar with the <a title="More information about the dataset in our Wiki" href="http://code.google.com/p/aus-e-stage/wiki/NavigatingNetworksDataset">fields and prefixes</a> used in the construction of the datastore.</p>
<p>The photo &#8220;<a title="Direct link to the photo" href="http://www.flickr.com/photos/cybersaphyra/3508644240/">Sparkle in Spring</a>&#8221; was uploaded to <a title="Flickr homepage" href="http://www.flickr.com/">Flickr</a> by <a title="Flickr profile page" href="http://www.flickr.com/people/cybersaphyra/">Deborah Herbaut</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by-nc-nd/2.0/deed.en">Creative Commons license</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1642/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1642/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1642/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1642/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1642/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1642/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1642/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1642/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1642/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1642/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1642/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1642/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1642/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1642/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1642&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/07/16/exploring-joseki-with-ausstage-data/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/07/sparkle.jpg" medium="image">
			<media:title type="html">SONY DSC</media:title>
		</media:content>
	</item>
		<item>
		<title>VisualVM, Java and Ubuntu</title>
		<link>http://techxplorer.com/2010/07/16/visualvm-java-and-ubuntu/</link>
		<comments>http://techxplorer.com/2010/07/16/visualvm-java-and-ubuntu/#comments</comments>
		<pubDate>Fri, 16 Jul 2010 01:18:05 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[java]]></category>
		<category><![CDATA[ubuntu]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1634</guid>
		<description><![CDATA[My most recent posts have been about my explorations into ways to resolve a memory resource issue I was having with some code I was developing for the Navigating Networks service as part of the Aus-e-Stage project which is part of the larger AusStage project. In particular I was Exploring the Java Heap as a [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1634&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1637" class="wp-caption alignright" style="width: 250px"><a href="http://www.flickr.com/photos/a_mason/4280360776/"><img class="size-full wp-image-1637" title="magnify-circuit-board" src="http://techxplorer.files.wordpress.com/2010/07/magnify-circuit-board.jpg?w=240&#038;h=180" alt="" width="240" height="180" /></a><p class="wp-caption-text">By: Andrew  Mason</p></div>
<p>My most recent posts have been about my explorations into ways to resolve a memory resource issue I was having with some code I was developing for the <a title="More information about the service" href="http://code.google.com/p/aus-e-stage/wiki/NetworkService">Navigating Networks</a> service as part of the <a title="Aus-e-Stage Project Homepage" href="http://beta.ausstage.edu.au/">Aus-e-Stage</a> project which is part of the larger <a title="AusStage Homepage" href="http://www.ausstage.edu.au/">AusStage</a> project. In particular I was <a title="Direct link to the previous post" href="/2010/07/13/exploring-the-java-heap/">Exploring the Java Heap</a> as a way to determine what was happening in my app which caused it to run out of memory.</p>
<p>Using some advice I got from the <a title="Direct link to the group homepage" href="http://tech.groups.yahoo.com/group/jena-dev/">jena-dev group</a> I started investigating the <a title="VisualVM homepage" href="https://visualvm.dev.java.net/">VisualVM</a> tool. If you&#8217;ve read any of my posts before you&#8217;ll know that I use <a title="Official Ubuntu Website" href="http://www.ubuntu.com/">Ubuntu</a> as my primary operating system on all of my machines including my workstation here at work.</p>
<p>When I first started investigating VisualVM I tried to run it from the command line and got a command not found error with a suggestion that it would be available as part of the <a title="Details of this package" href="http://packages.ubuntu.com/lucid/visualvm">visualvm package</a>. Unfortunately on the 64bit version of Ubuntu the <a title="Direct link to the bug report" href="https://bugs.launchpad.net/ubuntu/+source/visualvm/+bug/577958">package has a bug</a> and it is missing a file.</p>
<p>My Java development requires me to use the proprietary version of Java available in the sun-java6-* packages in the <a title="Ubuntu 10.04 Release Notes about this change" href="https://wiki.ubuntu.com/LucidLynx/ReleaseNotes#Sun%20Java%20moved%20to%20the%20Partner%20repository">Ubuntu Partner repository</a>. Using these packages the VisualVM application is installed it just isn&#8217;t available in the default path which means it can&#8217;t be found when you try to run it from outside the directory that it is in.</p>
<p>On my system VisualVM is located here:</p>
<p><code>/usr/lib/jvm/java-6-sun/bin/jvisualvm</code></p>
<p>To be able to run it from anywhere I created a link to it using the ln tool in the /usr/bin directory like this:</p>
<p><code>cd /usr/bin<br />sudo ln -s /usr/lib/jvm/java-6-sun/bin/jvisualvm ./jvisualvm</code></p>
<p>With the link in place the app can be found in the default path and I can execute it from the command line from anywhere.</p>
<p>The &#8220;<a title="Direct link to the photo" href="http://www.flickr.com/photos/a_mason/4280360776/">Magnifying the Circuit Board</a>&#8221; photo was uploaded to <a title="Flickr homepage" href="http://www.flickr.com/">Flickr</a> by <a title="Flickr profile page" href="http://www.flickr.com/people/a_mason/">Andrew Mason</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by/2.0/deed.en">Creative Commons license</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1634/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1634/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1634/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1634/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1634/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1634/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1634/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1634/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1634/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1634/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1634/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1634/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1634/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1634/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1634&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/07/16/visualvm-java-and-ubuntu/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/07/magnify-circuit-board.jpg" medium="image">
			<media:title type="html">magnify-circuit-board</media:title>
		</media:content>
	</item>
		<item>
		<title>Exploring the Java Heap</title>
		<link>http://techxplorer.com/2010/07/13/exploring-the-java-heap/</link>
		<comments>http://techxplorer.com/2010/07/13/exploring-the-java-heap/#comments</comments>
		<pubDate>Tue, 13 Jul 2010 00:31:10 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[ausstage]]></category>
		<category><![CDATA[java]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1627</guid>
		<description><![CDATA[Yesterday I wrote about the explorations I&#8217;ve been undertaking with Jena, TDB, Memory Issues and AusStage. In that post I noted that there was a sharp drop in the amount of heap that was being used during the execution of the code that adds the collaboration relationships to the dataset. I was thinking about this [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1627&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1630" class="wp-caption alignright" style="width: 250px"><a href="http://www.flickr.com/photos/mrzeon/2915859538/"><img class="size-full wp-image-1630" title="minifig-ram-installers" src="http://techxplorer.files.wordpress.com/2010/07/minifig-ram-installers.jpg?w=240&#038;h=180" alt="" width="240" height="180" /></a><p class="wp-caption-text">By: Daniel  Dionne</p></div>
<p>Yesterday I wrote about the explorations I&#8217;ve been undertaking with <a title="Direct link to the earlier post" href="/2010/07/12/jena-tdb-memory-issues-and-ausstage/">Jena, TDB, Memory Issues and AusStage</a>. In that post I noted that there was a sharp drop in the amount of heap that was being used during the execution of the code that adds the collaboration relationships to the dataset. I was thinking about this sharp drop last night and wondering what caused it.</p>
<p>The way that data is added to the <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Resource_Description_Framework">RDF</a> based dataset is as follows:</p>
<ol>
<li>An <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/SQL">SQL</a> statement executed against the <a title="AusStage Hompage" href="http://www.ausstage.edu.au">AusStage</a> database</li>
<li>The set of records returned from the database is iterated over</li>
<li>Each record is used to construct the appropriate RDF resources and properties</li>
<li>Once all records have been processed the SQL related objects are closed</li>
</ol>
<p>Importantly these steps occur for each stage of the export process and are enclosed on their own try / catch block. I started wondering if there was something happening inside the try / catch block that was consuming resources on the heap that would only release the memory once the program had moved to the next try / catch block.</p>
<p>Inside the while loop that processes the records I had code that repeatedly defined variables that were used for a short time and then not used again.</p>
<pre class="brush: java;">

//start of a contrived example

while (resultSet.next()) {
  String firstVariable  = resultSet.getString(1);
  String secondVariable = resultSet.getString(2);
  // do something with the variables
}
</pre>
<p>I made a change to the code this morning whereby all variables using inside the while loop were defined outside the while loop and then had values assigned as necessary inside the while loop. For example the code now looks like this:</p>
<pre class="brush: java;">

//start of a contrived example
String firstVariable  = null;
String secondVariable = null;

while  (resultSet.next()) {
  firstVariable  = resultSet.getString(1);
  secondVariable = resultSet.getString(2);
  // do something with the variables
}
</pre>
<p>Running the updated program and watching the heap usage using the <a title="More information about this application" href="https://visualvm.dev.java.net/">VisualVM</a> application I could see that a lot of the heap was still being used. However, significantly it was less than the amount used before the change by about 250MB.</p>
<p>What this suggests to me is that the second way of implementing the code uses less of the heap and that this can only be a good thing for my application. Especially as it still uses quite a lot of memory due to the caching that the <a title="Official Jena website" href="http://openjena.org/">Jena</a> and <a title="Official TDB website" href="http://openjena.org/TDB/">TDB</a> libraries do.</p>
<p>Later this morning I&#8217;ll be going over the rest of the code and ensuring that the other loops are implemented using the same technique.</p>
<p>The photo &#8220;<a title="Direct link to the photo" href="http://www.flickr.com/photos/mrzeon/2915859538/">Getting some RAM installed</a>&#8221; was uploaded to <a title="Flickr homepage" href="http://www.flickr.com">Flickr</a> by <a title="Flickr profile page" href="http://www.flickr.com/people/mrzeon/">Daniel Dionne</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by-sa/2.0/deed.en">Creative Commons license</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1627/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1627/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1627/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1627/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1627/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1627/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1627/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1627/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1627/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1627/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1627/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1627/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1627/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1627/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1627&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/07/13/exploring-the-java-heap/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/07/minifig-ram-installers.jpg" medium="image">
			<media:title type="html">minifig-ram-installers</media:title>
		</media:content>
	</item>
		<item>
		<title>Jena, TDB, Memory Issues and AusStage</title>
		<link>http://techxplorer.com/2010/07/12/jena-tdb-memory-issues-and-ausstage/</link>
		<comments>http://techxplorer.com/2010/07/12/jena-tdb-memory-issues-and-ausstage/#comments</comments>
		<pubDate>Mon, 12 Jul 2010 08:17:28 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[java]]></category>
		<category><![CDATA[ausstage]]></category>
		<category><![CDATA[jena]]></category>
		<category><![CDATA[rdf]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1617</guid>
		<description><![CDATA[The past week or so has seen me continuing my work on exporting a subset of the AusStage dataset into an RDF based datastore using Jena and TDB for the Navigating Networks Service. It&#8217;s been a frustrating time as I&#8217;ve been working my way through issues related to memory management and garbage collection in Java. [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1617&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1620" class="wp-caption alignright" style="width: 250px"><a href="http://www.flickr.com/photos/t0msk/3804126455/"><img class="size-full wp-image-1620" title="pebble-stack" src="http://techxplorer.files.wordpress.com/2010/07/pebble-stack.jpg?w=240&#038;h=160" alt="" width="240" height="160" /></a><p class="wp-caption-text">By: t0msk</p></div>
<p>The past week or so has seen me continuing my work on exporting a subset of the <a title="AusStage Homepage" href="http://www.ausstage.edu.au/">AusStage</a> dataset into an <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Resource_Description_Framework">RDF</a> based datastore using <a title="Official Jena website" href="http://openjena.org/">Jena</a> and <a title="Official TDB website" href="http://openjena.org/TDB/">TDB</a> for the <a title="More information about the Navigating Networks Service" href="http://code.google.com/p/aus-e-stage/wiki/NetworkService">Navigating Networks Service</a>. It&#8217;s been a frustrating time as I&#8217;ve been working my way through issues related to memory management and <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Garbage_collection_%28computer_science%29">garbage collection</a> in <a title="Official Java website" href="http://java.com">Java</a>.</p>
<p>My essential problem was that I kept running out of memory that my application could use. More specifically I was running out of <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Dynamic_memory_allocation">heap space</a>. The purpose of this post is to reflect on some of the lessons that I&#8217;ve learned so far.</p>
<p>My first way of reducing the amount of memory required by my application was to switch from using <a title="Wikibooks page on this topic" href="http://en.wikibooks.org/wiki/Java_Programming/Collections">Java Collections</a> to the manage a set of <a title="Jena API Documentation " href="http://openjena.org/javadoc/com/hp/hpl/jena/rdf/model/Resource.html">Jena Resource</a> objects. I maintain this list to ensure that any subsequent data that I add to the datastore relates to a contributor that I&#8217;ve already processed.</p>
<p>This is necessary because updates to the AusStage data can, and do, happen at any time. To reduce the amount of memory consumed by my application I&#8217;ve switched to maintaining a set of integers, representing the unique identifiers or contributors, using the <a title="More information about this library" href="http://trove4j.sourceforge.net/">GNU Trove library</a>. Rather than store Jena Resource objects I use the <a title="More information about this method" href="http://openjena.org/javadoc/com/hp/hpl/jena/rdf/model/Model.html#createResource%28java.lang.String%29">Model.createResource()</a> method to create new resources and return instances of resources that have already been added to the RDF model.</p>
<p>I also optimised my SQL queries to return the exact datasets that I require, and with the assistance of one of my colleagues use <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/SQL">SQL</a> to de-duplicate the list of relationships between contributors. Trying to do this in my Java code was causing too much memory to be consumed.</p>
<p>Lastly, after <a title="Start of my message thread" href="http://tech.groups.yahoo.com/group/jena-dev/message/44513">sending a few messages</a> to the <a title="Main page for the list" href="http://tech.groups.yahoo.com/group/jena-dev/">Jena-dev mailing list</a> I tweaked some other aspects of the code and gained a great understanding of what was happening. Many thanks to those who responded.</p>
<p>All of this means that my code now successfully runs again, but this is not to say that it doesn&#8217;t require a large amount of memory. As can be seen from the instructions for the <a title="More information about the application" href="http://code.google.com/p/aus-e-stage/wiki/RdfExport">RdfExport application</a> stored in our Wiki on the <a title="Main page of the site" href="http://code.google.com/p/aus-e-stage/">Aus-e-Stage project</a> pages the application does require 2 <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Gigabyte">gigabytes</a> of heap space. This necessitated moving to the <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/64-bit">64-bit</a> version of <a title="Official Ubuntu Website" href="http://www.ubuntu.com/">Ubuntu</a> and away from the <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/32-bit">32-bit</a> version so that I could use the 64bit version of the <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Java_Virtual_Machine">Java Virtual Machine</a>.</p>
<p>The majority of this space is used when collaborations, the resources representing a collaboration between two contributors, are added to the datastore. At the moment there are in excess of 1.6 million unique collaborations represented in the AusStage database. An attribute of a collaboration resource is how many times the two contributors have collaborated together has occurred. This means that the actual number of times that two contributors have worked together is significantly more than this.</p>
<p>Using the <a title="More information about this application" href="https://visualvm.dev.java.net/">VisualVM</a> application as suggested by a message posted to the mailing list I can see when my code starts to add collaborations and the impact this has on heap space.</p>
<div id="attachment_1618" class="wp-caption aligncenter" style="width: 310px"><a href="http://techxplorer.files.wordpress.com/2010/07/heap-allocation-graph.png"><img class="size-medium wp-image-1618" title="heap-allocation-graph" src="http://techxplorer.files.wordpress.com/2010/07/heap-allocation-graph.png?w=300&#038;h=156" alt="Heap Allocation Graph" width="300" height="156" /></a><p class="wp-caption-text">Heap Allocation Graph  (Click Image for Larger Version)</p></div>
<p>On the right hand side you can see how the amount of allocated heap space and the amount of heap used increases as the collaboration resources are added. Once all of the resources are added the heap space usage drops off to just under half of the maximum. Without the 2GB of heap space my application would fail with an out of memory error related to the garbage collector spending too much time undertaking garbage collection for little or no heap recovery. This error was usually prefaced with my application slowing to a crawl with what I assume is more and more time spent by the JVM trying to do garbage collection.</p>
<p>The <code>-XX:-UseGCOverheadLimit</code> option was not helpful in this instance as it just prolonged the inevitable.</p>
<p><strong>[Update Monday 19 July 2010]</strong></p>
<p>I forgot to mention that the final change I made to my code was to develop and run it on 64bit operating systems. This is because of the 1.5 &#8211; 1.6GB <a title="More information about this limit" href="http://java.sun.com/docs/hotspot/HotSpotFAQ.html#gc_heap_32bit">heap space limit that the 32bit version</a> of the JVM has.</p>
<p><strong>[/Update]</strong></p>
<p>By undertaking these changes my application now compiles the dataset using the fields that are currently required. Further changes will need to be made to add some additional fields and manipulate other aspects of the data for use by the Navigating Networks Service.</p>
<p>Some interesting statistics of the current dataset are:</p>
<ul>
<li>81,680 contributors resources</li>
<li>101,984 contributor function properties</li>
<li>1,640,250 collaboration resources</li>
<li>43,706 event resources</li>
<li>283,617 contributor functions at events</li>
</ul>
<p>In total there are 15,924,406 triples in the datastore.</p>
<p>The photo &#8220;<a title="Direct link to the photo" href="http://www.flickr.com/photos/t0msk/3804126455/">244/365 Six Rocks</a>&#8221; photos was uploaded to <a title="Flickr homepage" href="http://www.flickr.com/">Flickr</a> by <a title="Flickr profile page" href="http://www.flickr.com/people/t0msk/">t0msk</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by-nc-sa/2.0/deed.en">Creative Commons License</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1617/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1617/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1617/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1617/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1617/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1617/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1617/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1617/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1617/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1617/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1617/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1617/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1617/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1617/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1617&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/07/12/jena-tdb-memory-issues-and-ausstage/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/07/pebble-stack.jpg" medium="image">
			<media:title type="html">pebble-stack</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/07/heap-allocation-graph.png?w=300" medium="image">
			<media:title type="html">heap-allocation-graph</media:title>
		</media:content>
	</item>
		<item>
		<title>Modelling Collaborations in AusStage Using RDF</title>
		<link>http://techxplorer.com/2010/07/06/modelling-collaborations-in-ausstage-using-rdf/</link>
		<comments>http://techxplorer.com/2010/07/06/modelling-collaborations-in-ausstage-using-rdf/#comments</comments>
		<pubDate>Tue, 06 Jul 2010 01:33:02 +0000</pubDate>
		<dc:creator>techxplorer</dc:creator>
				<category><![CDATA[Thoughts]]></category>
		<category><![CDATA[java]]></category>
		<category><![CDATA[ausstage]]></category>
		<category><![CDATA[jena]]></category>
		<category><![CDATA[rdf]]></category>

		<guid isPermaLink="false">http://techxplorer.com/?p=1605</guid>
		<description><![CDATA[My last few posts have focused on my use of Jena as a means of constructing a RDF based dataset using data sourced from the AusStage relational database for use with the Navigating Networks component of the Aus-e-Stage project. I&#8217;m taking this approach for two main reasons: It is faster to query than the relational [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1605&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></description>
			<content:encoded><![CDATA[<div id="attachment_1613" class="wp-caption alignright" style="width: 170px"><a href="http://www.flickr.com/photos/tym/269106139/"><img class="size-full wp-image-1613" title="contortion" src="http://techxplorer.files.wordpress.com/2010/07/contortion.jpg?w=160&#038;h=240" alt="" width="160" height="240" /></a><p class="wp-caption-text">By: Proggie</p></div>
<p>My last few posts have focused on my use of <a title="Official Jena website" href="http://openjena.org/">Jena</a> as a means of constructing a <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Resource_Description_Framework">RDF</a> based dataset using data sourced from the <a title="Official AusStage website" href="http://www.ausstage.edu.au/">AusStage</a> relational database for use with the <a title="Development documentation on the Navigating Networks Service" href="http://code.google.com/p/aus-e-stage/wiki/NetworkService">Navigating Networks</a> component of the <a title="Official Aus-e-Stage website" href="http://beta.ausstage.edu.au">Aus-e-Stage project</a>. I&#8217;m taking this approach for two main reasons:</p>
<ol>
<li>It is faster to query than the relational database</li>
<li>It gives us experience in using RDF and related technologies for use in other aspects of the AusStage project</li>
</ol>
<p>One of the primary key components of the dataset is the relationships between contributors. A contributor is someone who has in some way contributed to an event. A relationship is deemed to exist where two contributors have contributed to the same event. For the purposes of this discussion I&#8217;m calling this a collaboration.</p>
<p>We use the <a title="Official FOAF website" href="http://www.foaf-project.org/">FOAF ontology</a> to store basic biographical information about contributors and we were using the <a title="More information about this term" href="http://xmlns.com/foaf/spec/#term_knows">foaf:knows term</a> to model the relationships between contributors. Continued development of the <a title="More information about the goals" href="http://code.google.com/p/aus-e-stage/wiki/NavigatingNetworksGoals">Networking Service Goals</a> and the <a title="More information about the specification" href="http://code.google.com/p/aus-e-stage/wiki/NavigatingNetworksSpecification">Networking Service Specification</a> have shown that the foaf:knows term isn&#8217;t sufficient to model the complex data related to a collaboration. One of the properties of interest is the number of times that two contributors have collaborated on the same event and the FOAF ontology doesn&#8217;t provide a mechanism for modelling this type of information.</p>
<p>In conjunction with the <a title="More information about the dataset" href="http://code.google.com/p/aus-e-stage/wiki/NavigatingNetworksDataset">development of the dataset</a> I&#8217;ve been working on the basis of an <a title="More information about the ontology" href="http://code.google.com/p/aus-e-stage/wiki/AuseStageOntology">Aus-e-Stage ontology</a>. This ontology allows me to model the data required for the Navigating Networks service where existing ontologies are not available. <strong> </strong></p>
<p><strong>Please Note</strong> that this ontology is only meant as a stopgap measure and should in no way be seen as pre-empting the work that will be conducted as part of another AusStage related project in flattening the entire AusStage dataset for periodic export into a data repository.</p>
<p>The ontology provides four terms related to collaborations which are:</p>
<ol>
<li>collaboration &#8211; used to store details of a collaboration relationship</li>
<li>collaborator -  a reference to a contributor involved in this collaboration (appears twice, one for each contributor)</li>
<li>collaborationCount &#8211; the number of times the two contributors have collaborated</li>
<li>hasCollaboration &#8211; stores a reference to a collaboration with a contributor</li>
</ol>
<p>An example of two contributors and their collaboration is shown in the following snippet of <a title="Wikipedia article on this topic" href="http://en.wikipedia.org/wiki/Notation3">Notation3</a> code below.</p>
<pre class="brush: plain; auto-links: false;">

@prefix time:    &lt;http://www.w3.org/2006/time#&gt; .
@prefix tl:      &lt;http://purl.org/NET/c4dm/timeline.owl#&gt; .
@prefix foaf:    &lt;http://xmlns.com/foaf/0.1/&gt; .
@prefix event:   &lt;http://purl.org/NET/c4dm/event.owl#&gt; .
@prefix xsd:     &lt;http://www.w3.org/2001/XMLSchema&gt; .
@prefix ausestage:  &lt;http://code.google.com/p/aus-e-stage/wiki/AuseStageOntology#&gt; .
@prefix dcterms:  &lt;http://purl.org/dc/terms/&gt; .

&lt;ausstage:c:261085&gt;
      a       foaf:Person ;
      ausestage:collaboratorCount
              &quot;17&quot; ;
      ausestage:function &quot;Circus Style Performer&quot; ;
      ausestage:hasCollaboration
              &lt;ausstage:rel:261085-261086&gt; ;
      ausestage:nationality
              &quot;Australian&quot; ;
      foaf:familyName &quot;Ristic&quot; ;
      foaf:givenName &quot;Denjan&quot; ;
      foaf:name &quot;Denjan Ristic&quot; ;
      foaf:page &quot;http://www.ausstage.edu.au/indexdrilldown.jsp?xcid=59&amp;f_contrib_id=261085&quot; .

&lt;ausstage:c:261086&gt;
      a       foaf:Person ;
      ausestage:collaboratorCount
              &quot;17&quot; ;
      ausestage:function &quot;Circus Style Performer&quot; ;
      ausestage:hasCollaboration
              &lt;ausstage:rel:261085-261086&gt; ;
      ausestage:nationality
              &quot;Australian&quot; ;
      foaf:familyName &quot;Sanders&quot; ;
      foaf:givenName &quot;D'Nash&quot; ;
      foaf:name &quot;D'Nash Sanders&quot; ;
      foaf:page &quot;http://www.ausstage.edu.au/indexdrilldown.jsp?xcid=59&amp;f_contrib_id=261086&quot; .

&lt;ausstage:rel:261085-261086&gt;
      a       ausestage:collaboration ;
      ausestage:collaborationCount
              &quot;1&quot; ;
      ausestage:collaborator
              &lt;ausstage:c:261085&gt; , &lt;ausstage:c:261086&gt; .
</pre>
<p>In the above code there is:</p>
<ul>
<li>A <a title="More information about this term" href="http://xmlns.com/foaf/spec/#term_Person">foaf:person</a> with the unique identifier ausstage:c:261085</li>
<li>A second foaf:person with the unique identifier ausstage:c:261086</li>
<li>A ausestage:collaboration linked to the contributors</li>
</ul>
<p>The unique identifiers are encoded using <a title="More information about the way unique identifiers are encoded" href="http://code.google.com/p/aus-e-stage/wiki/RdfExportMoreInfo#AusStage_Unique_Id_Encoding_Scheme">this URI scheme</a> which provides a way to easily distinguish between identifiers that are used for contributors, events, and other internal objects such as these collaboration relationships.</p>
<p>Using this type of model to record collaborations is extensible and additional information can be included. One data element that is of particular interest is the time span of the collaboration relationship as determined by the dates associated by the events that form the collaboration. For example if the first event that was part of the collaboration started on 01/01/2010 and the last event finish on 01/06/2010 it could be said that the collaboration relationship lasted 6 months.</p>
<p>The dataset, and by extension the data model, continue to evolve as more is understood about RDF and the supporting technologies as well as the continued development of the Navigating Networks Service goals and specification.</p>
<p>The photo &#8220;<a title="Direct link to the photo" href="http://www.flickr.com/photos/tym/269106139/">Contortion</a>&#8221; was uploaded to <a title="Flickr homepage" href="http://www.flickr.com/">Flickr</a> by <a title="Flickr profile page" href="http://www.flickr.com/people/tym/">Proggie</a> and used under the terms of a <a title="Details of this license" href="http://creativecommons.org/licenses/by-nc-nd/2.0/deed.en">Creative Commons license</a>.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/techxplorer.wordpress.com/1605/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/techxplorer.wordpress.com/1605/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/techxplorer.wordpress.com/1605/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/techxplorer.wordpress.com/1605/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/techxplorer.wordpress.com/1605/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/facebook/techxplorer.wordpress.com/1605/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/techxplorer.wordpress.com/1605/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/twitter/techxplorer.wordpress.com/1605/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/techxplorer.wordpress.com/1605/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/techxplorer.wordpress.com/1605/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/techxplorer.wordpress.com/1605/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/techxplorer.wordpress.com/1605/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/techxplorer.wordpress.com/1605/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/techxplorer.wordpress.com/1605/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=techxplorer.com&amp;blog=193831&amp;post=1605&amp;subd=techxplorer&amp;ref=&amp;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://techxplorer.com/2010/07/06/modelling-collaborations-in-ausstage-using-rdf/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
	
		<media:content url="http://0.gravatar.com/avatar/e3e762252143800cc3b6efe3eaf8913a?s=96&#38;d=identicon&#38;r=G" medium="image">
			<media:title type="html">techxplorer</media:title>
		</media:content>

		<media:content url="http://techxplorer.files.wordpress.com/2010/07/contortion.jpg" medium="image">
			<media:title type="html">contortion</media:title>
		</media:content>
	</item>
	</channel>
</rss>