Three Utility Scripts Updated

"deBUG" by Franz & P

“deBUG” by Franz & P

Today I updated three of my utility scripts. More information on the scripts is available in the Techxplorer’s Utility Scripts repository on GitHub.

The three scripts that I updated are:

  1. JiraGitBridge – for acting as a bridge between JIRA and a Git repository
  2. MdlUserListCreator – for creating randomly generated lists of users for upload into Moodle
  3. ServiceMgmt – making it easier to automate services managed with launchctl

More information on the exact changes are available in the Change Log sections of the wiki pages for each of the scripts on GitHub. The main page of the wiki, available here, links to all of the individual wiki pages for each of the scripts.

I maintain a wiki with documentation on the scripts because I believe, documentation is just as important as the code.

If you’re using these scripts, I hope they prove useful.

New Utility Script: ServiceMgmt

"Launch Control" by wbeem

“Launch Control” by wbeem

Today I released a new script in the Techxplorer’s Utility Scripts repository on GitHub called ServiceMgmt.

The intention of the script is to make it easier to start, stop, restart and checkup on the services that I use on a daily basis. This includes the nginx web server, the PHP-FPM server for executing PHP scripts, and the Memcached server.

There are two things that each of these servers has in common, they are:

  1. Installed using homebrew
  2. Managed by OS X by launchd which can be accessed via launchctl

A list of services is stored in a JSON file. The script uses this file to determine a list of services that it needs to work with. More information on how to use the script is contained in the wiki page for the script.

If you find the script useful, please let me know.

Is MOOC Too Masculine?

"0046 Le Boss" by Artemedion

“0046 Le Boss” by Artemedion

This is an edited repost of an earlier post from an earlier version of my blog.

While working at a university nearly a year ago, I had a discussion with a colleague of mine in the computer science school tea room about online learning. In particular we were discussing the impact of MOOCs on the higher education sector, and their potential impact here in Australia. The discussion was in the context of what we’d want to see in a MOOC if we were in control of one.

For those that aren’t familiar with the term MOOC, which stands for Massive Open Online Course, is defined by Wikipedia as:

an online course aimed at large-scale interactive participation and open access via the web. In addition to traditional course materials such as videos, readings, and problem sets, MOOCs provide interactive user forums that help build a community for the students, professors, and teaching assistants

The key items from the point of view of the higher education sector are:

  1. They’re large scale
  2. They have content other than just slides, documents and video recordings of lectures
  3. They are accessible via the web
  4. Most importantly they’re open access, or to put it another way they’re free to access

At the time I had to cut the discussion short because I needed to make the trek across campus to the humanities school for a meeting with another colleague of mine. As an interesting side note, who knew working on a cross discipline project involved so much walking around campus?

While in the humanities schoolI happened to see a list of items on their whiteboard for discussion and further investigation I noticed that one of them was MOOC’s. I mentioned that I’d just been discussing it before our meeting. That’s when she said something quite interesting.

She said that she was a member of a feminist collective in the school and that the collective was against the term MOOC as it was deemed to be too masculine. I enquired how it was too masculine and she wasn’t sure, perhaps it has something to do with the term massive. We left the discussion there and moved on with the rest of the meeting

This comment stuck in my mind. A few days later I was back in the computer science tea room, involved in a discussion on online learning again, when I happened to mention the discussion I’d had in the humanities a few days earlier.

The idea that the term MOOC was too masculine for the feminist collective was met with laughter and a comment that they’d just have to get used to it. Now I should clarify at this point that both colleagues were women.

I felt this was entirely the wrong response. The key issue for me isn’t if MOOC is too masculine or not. They key issue for me, is that a not insignificant number of the user community where a MOOC may be deployed felt that the term was too masculine. My thoughts on the matter of the masculinity of MOOCs are immaterial, what is important is their thoughts on it.

It is important because if a MOOC was introduced, they will need to use the system. They will need to be the ones uploading the content, in many instances creating new content, and most importantly being advocates for the system.

If they don’t like the term MOOC, getting them to use and participate in the system is going to that much more difficult. Not only would their expectations of the new system need to be managed carefully, they’re less likely to engage as it uses a term that they don’t like.

The key issue here is that if they don’t engage with the system then the system is a failure. For in my mind if a system doesn’t meet the needs of users, they won’t use the system. Or if they do use the system it will be under duress and therefore they won’t use it to the fullest extent possible. Lastly, and perhaps most importantly, they won’t be good advocates for the system.

The response to the feminists shouldn’t be “just get over it” it should be “what term would be better?”. That way they’re involved in the development of the system and more importantly starting to engage with the system.

While this is an amusing anecdote the key message that I want to impart is this. When developing a new system getting the users to engage in the system is critical and if that means changing the language that is used to describe the system, then that’s what you’ve got to do.

Investigating Performance Issues in a Moodle Plugin

"deBUG" by Franz & P

“deBUG” by Franz & P

In an earlier post today I mentioned that I’d added a new script to the Techxplorer’s Utility Scripts repository on GitHub, the MakeRamDisk script. The intention of the script is to make it easier to to make a RAM Disk, also known as a RAM Drive, on OS X.

The reason I wrote this script, is that last week I had to do some work for a NetSpot client who was having timeout issues. The Moodle plugin that they were using wasn’t completing a task. It was taking too long and was being killed by PHP. There  were two issues that I identified, with the help of a colleague.

The first issue was that the plugin was calculating a grade and then writing that grade to the database. Basically the code did this:

  1. Retrieve all records related to grading of assignments, five records per assignment in this case
  2. Calculate the grades
  3. Use the records from step 1 as a list of assignment IDs to update

Can you see the problem?

Step 1 retrieves five records per assignment, and step 3 uses this as a list of assignment IDs to update. Therefore each assignment was updated 5 times with exactly the same grade. While this is certainly inefficient, this isn’t a big deal if have a small number of students in a course. But if you have 1,500 students in a course that means that there is going to be 7,500 database writes.

To fix this it was necessary to do two things:

  1. Store the assignment id, and a small number of other ids, only once
  2. Use this revised list when writing the grades to the database

Now instead of 7,500 database writes there were only 1,500 writes, one for each student. Additionally, rather than retaining the entire database record for later, only the small number of IDs per record were retained. This second change also reduced memory consumption of the script.

The second issue was tricker to resolve. As part of the investigation I needed to use the profiler of the Xdebug extension for PHP to profile the script and understand where the performance issues were. I also used the Xdebug helper extension for Google Chrome to help only profile the scripts that I needed to.

Profiling a script has a significant performance impact. Writing the output file to a RAM disk, which is significantly faster than even a SSD, helps speed up the process. Hence the development of the MakeRamDisk script which made it easier to make RAM disks of various sizes. The trick with RAM disks is to create one that is big enough for the task, which doesn’t deprive the rest of your system of the RAM that it needs to work.

Lastly I used the qcachegrind application, part of the KCachegrind suite, to analyse the file and identified a few areas that could be optimised. As always with applications like this it was easy to install it using Homebrew.

Now I have some new skills to add to my forensic programming toolkit.


Big Update to Techxplorer’s Utility Scripts

"Concentration V1.0" by Adam Ward

“Concentration V1.0″ by Adam Ward

Last night I pushed a big update to the Techxplorer’s Utility Scripts repository on GitHub. The intention of the scripts, which I write and maintain in my own time, is to help with my work at NetSpot. I’ve made them open source in the hope that they prove useful to others.

I’ve written more scripts that I initially thought I would when I started the project, and so I thought it was time to take it seriously. As such I have:

  1. Taken common code and broken it out into library functions that all the scripts can use
  2. Added PHPUnit tests for all of the library functions
  3. Made all of the code compliant with the default PHP_CodeSniffer rules
  4. Added a new script, MakeRamDisk, which is designed to make it easier to make a RAM Disk, also known as a RAM Drive, on OS X

In any sufficiently large code refactoring project there is the chance of bugs creeping in. If you use these scripts and notice something wrong, please let me know by adding an issue so that I can fix it. If you are using the scripts I’d appreciate it if you let me know what you think of them.

Investigating Issues with a Mobile App

"iPhone hacked" by Bwana McCall

“iPhone hacked” by Bwana McCall

Over the past few weeks I’ve been investigating issues reported by one of our NetSpot clients with the Moodle for Mobile app. To diagnose the issue, I needed to see the requests and responses between the app and a number of Moodle instances. Specifically, a development instance on my laptop and the clients test instance, located in the data centre. Additionally I needed to use the app installed on my iPhone.

The security model of the iPhone prohibited me from using the developer tools available in the Safari web browser to see what the app was doing. Even though the app was using an embedded web browser. So I needed to find a way to see what was going on without relying on anything on the device.

After a series of experimentations this is what I came up with.

First, I installed the Tinyproxy application which is a lightweight proxy server. I use the Homebrew package manager for OS X to install software like this. Installing the application was as simple as executing the following:

I chose Tinyproxy because I prefer small applications that do a small number of things well. Rather than applications that try to do everything. If for no other reason than smaller applications typically mean smaller configuration files.

Once Tinyproxy was installed, configured and running, I could configure my iPhone to use it as a proxy server. Now I could see the requests that the application made to the Moodle instances. Incidentally I could also see the requests that other background apps were making. To filter out these requests that I didn’t want to see, I combined the tail and grep commands, like this:

Replacing the hostname place holder with the server hostname that was of interest.

Now I could see, in real time, the requests being made by the mobile app on my iPhone.

Second, I used the Wireshark application to capture and analyse the network packets sent between the Tinyproxy server on my laptop and the iPhone. Using Wireshark made it easy to capture the packets, and more importantly reconstruct them so that I could see the contents of the requests and responses.

In summary, using both Tinyproxy and Wireshark I could see:

  1. The requests being made by the mobile app
  2. The responses from the Moodle instance
  3. The content of the requests and responses

Using this information I was able to determine what was causing the issue and investigate implementing a fix.

Forensic Programming – Implementing a Fix

"#53/366 Let Me Borrow" by Robert McGoldrick

“#53/366 Let Me Borrow” by Robert McGoldrick

After you have communicated with the client and understood the code, the last stage in the forensic programming process is implementing a fix. There are two important aspects to implementing a fix:

  1. Ensuring that the bug is squashed, and the client is happy
  2. Managing the risk of the fix

Making sure that the bug is squashed, is the self evident part of implementing a fix. You want to ensure that the code changes that you make fix the clients issue. The issue could be that the code doesn’t work as intended, or it could be that the code does work, but it doesn’t work the way the client wants. Either way the change that you make must solve the clients issue. 

What you don’t want is to implement what you believe is a fix, and then report back to the client. Only to have them test the code change to find that it doesn’t work. That just leads to embarrassment for you, and frustration for all.

The second element of implementing a fix isn’t as obvious. It’s all about managing the risk of the code change. There are a number of different types of risk. What you can’t do is yell something like ‘You’re doing it wrong!’ and then make a large number of code changes. This is because doing so creates too much risk. It also annoys the other software engineers in your vicinity who are quietly working away.

The more code changes that you make, the higher the risk that you’re introducing new bugs. The goal is implementing a fix for a bug or issue, not introducing new ones.

A high number of code changes also increases the risk of conflicts later. I don’t mean conflicts between developers, although this can happen, I mean conflicts reported by your version control software. You are running some sort of version control on your source code aren’t you?

Each change that you make has the potential to introduce code conflicts. For example if the upstream developers have fixed the bug in a different way. Or they may have made such significant changes to the code that your version control software can’t work out where in the file to make your change.

Solving conflicts in source code is not fun for anyone. Except perhaps if you’re one of those rare individuals that enjoy looking at diffs of source code and enjoy that sort of thing.

In summary, the key is to implement a fix which solves the clients issue with the minimum number of code changes. Adding code comments to indicate why you’re making the change can also be useful. Especially if you’re the person who is trying to solve a code conflict.

Forensic Programming – Understanding the Code

"Minifig Characters #5: Sherlock Holmes and Dr. Watson" by thom

“Minifig Characters #5: Sherlock Holmes and Dr. Watson” by thom

This is the third in a series of posts about forensic programming that started with the first post on my newly restarted blog. This post will focus on the task of understanding the code.

There are three aspects to understanding the code in any forensic programming task, and they are understanding:

  1. What the code is supposed to do
  2. What it actually does
  3. Where and how it is going wrong

This is arguably the hardest aspect of a forensic programming task, for two main reasons:

  1. You need to understand what the user is doing, so you can replicate the problem
  2. You need to understand the code

Using your communication skills it is possible to work with the user to understand the problem. Once the problem is understood you can develop a series of steps that can replicate the problem. If you can’t reliably replicate the problem, you have no idea if any fixes that you put in place, actually fix the root of the problem. Or if it just solves a symptom which may further mask the root cause.

The last thing that you want to happen is for you to develop a fix, have it tested by the client, and then have them report that it doesn’t work. That just leads to frustration for everyone.

Understanding the code is made more difficult by the actions of the people who have gone before you. One of my pet hates is code that isn’t sufficiently documented. It’s much easier to read and understand a series of comments, than to keep a series of function calls, variables and values straight in your head. If there was one phrase I’d ban from the vocabulary of people who write software it would be this:

The code is self documenting, so it doesn’t need comments

In my mind that’s an excuse uttered by a lazy programmer. Moreover, it is disrespectful to those that have to maintain their code after they’ve moved on to other things.

When writing code I like to include comments that:

  1. Describe what a function does
  2. The type and purpose of parameters
  3. The return type of a function, and what it means
  4. Any exceptions that I know may be thrown
  5. Decision points within the code

Basically the goal is to write enough comments to give someone coming after me an idea of what is going on, without having to read each line of code. Other developers may feel that I write too many comments. However I like to err on the side of too many comments, rather than not enough.

Adding debug statements can also be helpful when understanding how code works. They can be used to provide an insight into the contents of variables, as well as program flow. For example by dumping the contents of a variable or object. Or by having two debug statements, one after an if statement and another after the else.

Once you understand how the code works and where it is going wrong, you can implement a fix.

Two new Utility Scripts

This evening I’ve uploaded two new utility scripts to my techxplorer-utils repository in GitHub.

The first script is called JiraGitBridge and is designed to act as a bridge between JIRA and a Git. It automates Git related tasks using information in JIRA. At the moment the only action it undertakes is to create a list of commits in a Git repository based on the issues associated with a JIRA project and version. I intend to add additional actions as necessary.

The second script is called JiraListIssues and is designed to generate a list of issues, including their status, in JIRA associated with a project and version.

More documentation on these scripts, and the other scripts in the repository, is available in the repository wiki.

Forensic Programming – Communication is Key

"Communication Breakdown" by Stéfan

“Communication Breakdown” by Stéfan

Communication, I believe, is one of the things that separates software engineers from programmers / developers. It is also one of the defining characteristics of the best software engineers.

Communication is a key factor in the success of a forensic programming task. Without good communication skills, a software engineer can not achieve a positive outcome.

The first step is gaining an understanding of the issue that the client needs addressed. As I mentioned in the first post in this series the issue report usually goes something like this:

It doesn’t work! It is super critical, urgent, nay imperative, that this issue is fixed right now! In fact do you have a time machine? Because it should have been fixed ages ago! If you don’t fix it right now, I’m coming over and will rock your roof! 1

The use of your communication skills helps in asking questions of the client to determine exactly what the problem is. Often asking for a screen capture can be helpful. When asking for a screen capture be sure to be specific and ask for the entire browser window.

This can be helpful for two reasons. The first, is that the screen capture includes the URL of the page, which could provide vital information. The second, is that it gives you a clue about the browser and operating system that the client is using. Which may be something that they wouldn’t otherwise be able to tell you.

If necessary ask for steps to replicate the problem, and ask for as much detail as possible. It is important to remember, especially in the case of large enterprise systems, that your users may know more about an aspect of the system than you.

Keeping the client informed about your progress is very important. There is nothing more distressing to a client than having an issue that is affecting their users, which they don’t have any information from you about.

Remember that the person, or people, that your working with are a buffer between you and their users. A lack of communication puts them in a very awkward position.

Above all remain professional in your manner and use clear, concise and plain language. Limit the use of abbreviations and technical jargon as much as possible.

Good communication can go a long way to fostering a positive relationship between you and the client, which can only be a good thing.

Yes, I’m paraphrasing and taking dramatic liberties but you get the idea.