Debugging using a magical exploding class

'marvin' by photoguyinmo

‘marvin’ by photoguyinmo

Last week I was debugging an issue that I had in Moodle related to the use of a global variable in PHP. If you’re not familiar with global variables in PHP, conceptually they are a variable that can be accessed from any part of your script. Typically through the use of the ‘global’ keyword.

The issue I had was that a global variable was having a string property set to false somewhere during the execution of the script. This was causing errors in a completely unrelated part of the codebase.

I discussed this with a colleague of mine. They showed me a debugging technique which uses what they called a ‘magical exploding class’.

The first part of the technique is to define a class like this:

Once the class is defined, it is possible to use it like this:

The two key parts to this technique is that:

  1. The instance of the magical class automatically returns its value as a string automatically. This ensures that that other parts of the code don’t realise that the field in the global class is set to our class, and not a simple string
  2. If the content of the field is replaced with something else, the magical class is destructed which causes an exception to be thrown. In this way the code that is changing the field can be identified

Using this technique I was able to identify the troublesome code, and write a defect resolution. More importantly the issue wouldn’t have been possible if the global objects that Moodle uses were immutable objects. But that’s a discussion for another time.

Techxplorer’s Utility Scripts – Version 2.0

'number two' by mark dyer

‘number two’ by mark dyer

If there are two constants in the life of a software engineer they would be:

  1. You’re always learning. For example, learning new paradigms, programming languages, frameworks or just a different way of doing something you do regularly
  2. Software is never finished, there is always something to do.

Both of these constants came together this weekend when I decided to work on rewriting the scripts available in the Techxplorer’s Utility Scripts.

I’ve started rewriting the scripts that I use the most, taking advantage of the things I’ve learnt over since August 2013 when I started the project.

The three scripts that I’ve re-written so far are:

  1. DbAssist.php – Automate various database related tasks
  2. GerritPush.php – Make it easier to push changes to a Gerrit code review system
  3. GitFetchReset.php – Automate fetching changes from Git and reseting the current branch to the latest upstream version

Other scripts that I use regularly will be updated as time allows. All of the pre-existing scripts are still available in their own branch in the repository.

jQuery and Moodle. Just don’t do it.

'stop+think' by Caffeinatrix

‘stop+think’ by Caffeinatrix

Fair warning, this post is going to be a little bit of a rant.

My role at NetSpot / Blackboard is working as a Senior Software Engineer as the client development team for Enterprise Moodle. As part of my role I need to review plugins that our clients want to use as part of their Moodle instance.

The things we’re most in the lookout are potential issues that will have an adverse impact on our ability to provide our service to the high standard expected by our clients. An example is having very long timeouts on cURL requests, doing things in Moodle without appropriate considerations for security, or not using the standard Moodle libraries.

One thing I have noticed is that a number of plugins, and themes, use JavaScript that is built using the jQuery library. For reasons that I suspect are lost to history, and are not relevant to this discussion, Moodle HQ has settled on the YUI library.

Wether you agree with this decision or not, the fact remains that core Moodle uses YUI, and uses it extensively. Incorporating jQuery into your plugin or theme causes a number of issues for us and our clients.

One of the most severe are rendering issues of pages because the JavaScript code breaks in some way. Most often because loading both jQuery and YUI cause conflicts. This breaks the rendering of pages and takes us time to rectify.

The second issue is that by including jQuery the developer is, in my opinion, being disrespectful to the users of the plugin. This is because you’re forcing users to spend time and bandwidth downloading two JavaScript libraries.

In the event that, as a developer, you must use jQuery to achieve your goals, please adhere to the guidelines developed by Moodle HQ, which are available here. In particular don’t include your own copy of the jQuery library. It causes way too many issues for those of us who need to support users of your code.

If you feel strongly about the use of YUI, and want to contribute to the discussions of Moodle HQ about the future of YUI and Moodle, I encourage you to checkout out this Moodle tracker item, and this associated Moodle forum thread.

Here ends the rant.

Updated: January 26, 2015

A trusted colleague of mine has pointed out that discussions are underway at Moodle HQ on the possible switch to jQuery, and Require.js with possibly using Grunt as a build tool. More information is available in this Moodle forum thread, which I also linked to above. The most relevant forum post, so far, is available here.

This is particularly relevant as in August, 2014 it was announced that the YUI library is no longer actively maintained. You can read more about the decision here.

I believe that my original point still stands. As a developer, if you are writing JavaScript for Moodle, you should be using YUI and not jQuery unless there is a very serious and compelling reason to do so. As it causes complications for those of us that need to support users of your code, and is disrespectful to users of your code by forcing them to spend time and bandwidth on loading two JavaScript libraries.

This is because the decision on which JavaScript library Moodle will use in the future is yet to be decided. Until that decision has been made, and the version of Moodle that has made the switch reaches critical mass, the use of YUI is still going to be preferred.

Sending a password in a second email is not secure

'Broken Rusty Lock: Security (grunge)' by  Nick Carter

‘Broken Rusty Lock: Security (grunge)’ by Nick Carter

There are times when decisions made for reasons of ‘enterprise security’ perplex me. Take as an example this story related to me by a friend of mine the other day.

Every month the payroll department of the organisation my friend works for send them their pay slip via email. The payslip is a password protected PDF file. All good so far, right? Password protecting a file containing private and confidential information is a good thing.

Following the first email a second is sent by the payroll department.This second email contains the password required to open the PDF file. Both of these emails are sent in plain text. Additionally the password used to ‘secure’ the PDF file never changes.

I can see the chain of decisions that culminates in this process:

  1. The decision is made to send the payslip as a PDF file
  2. A payslip contains personal and confidential information, so it should be password protected
  3. To ensure the recipient can open the PDF file, the password needs to be shared with them
  4. Sending the PDF file and the password in the same email is not ‘secure’
  5. The conclusion is to send the password in a separate email
  6. To minimise the inconvenience to all parties the password should stay the same

The issue that I have with this process is that it isn’t very secure. In fact I would argue that it is not secure at all, and that it only provides the illusion of security to those who fundamentally don’t understand how email works.

The intention is to ensure that if the email with the PDF file is intercepted by a ‘bad actor’, they don’t get the password at the same time. However if one of the mail servers or networks that the email goes through is compromised, to the extent that bad actors can access an email, what is stopping them from intercepting the second email? The answer is nothing.

Additionally even if the second email is not intercepted, the password is always the same. Meaning that the bad actor only needs to intercept one of the emails with the password once, in order to gain access to the private and confidential data contained in the payslip.

This is an example of security theatre, designed to provide the illusion of security where none actually exists. As the emails are sent in plain text, there is no improved security here at all. Merely a small increase in annoyance for the recipients who need to use a password to access their payslip.

In short, this is not even a dubious example of security through obscurity. It is an example of the decision making process that occurs in enterprises. As it is a process which provides an illusion of increased security, with minimum impact on those following the process.

I wonder what it would take for this organisation to change this process?

Project Ara and the Paradox of Choice

’29/52 choice paralysis’ by Lauren Macdonald

’29/52 choice paralysis’ by Lauren Macdonald

Last week I was listening to one of my favourite podcasts, Daily Tech News Show (DTNS), when the topic of Project Ara from Google came up. The archived episode is available here. It was an interesting discussion, as they always are on DTNS, and I had a thought. I wondered what, if any, impact the paradox of choice would have on Project Ara.

For those not familiar with the paradox of choice, this is my understanding. It is ‘conventional wisdom’ that having more choice is good. Indeed, the more choices that you have the better. However this is not always the case. In particular:

  1. More choices increases anxiety about making the ‘best’ choice, by increasing the number of options and comparisons that you have to make. For example two choices mean that you have two comparisons to make in order to determine which one is ‘best’. If you have four choices you now have six comparisons to make and five choices means nine comparisons.
  2. More choices increases feelings of regret that you may not have made the ‘right’ or ‘best’ choice. The more choices you have, the greater the anxiety that the ‘best’ choice wasn’t made because one of the other options may have been better.

In short, the more choices that you have, the more anxious that you are during the decision making process, and the more regret you feel and after.

As a software engineer I try to keep this in mind when developing software that contains configuration screens. Particularly if there are a significant number of choices. Because the anxiety becomes so great that the user makes the easiest choice available to them. That is, leave the options at their defaults. A classic example related to Microsoft Word is available here. As such ensuring suitable defaults is paramount, but I digress.

I wondered if this type of thinking would apply to Project Ara. At its core, Project Ara is about giving consumers the ability to build their own mobile phone by selecting from a wide range of different modules and integrating them into a single frame. I can see that this type of paradigm means that a consumer will have many choices to make. Perhaps even too many choices.

If this is the case, I can’t help wondering if consumer satisfaction with Project Ara phones will suffer. Because of anxiety during the process of deciding on which modules to have, and anxiety afterward the purchasing decision has been made due to concern that a different combination may have been ‘better’.

I’m interested in mobile computing and will keep an eye on news of the project and see what the world makes of it.

Users are critical to an Enterprise Architecture

Late last year I gave a presentation to the Blackboard Service Delivery Manager (SDM) team in Adelaide about the Enterprise Moodle architecture. It was an adaptation of a session I’ve run twice now for final year computer science students at Flinders University for their Enterprise Systems topic. This was exciting for two reasons:

  1. I enjoy sharing information with colleagues and clients;
  2. It was the first opportunity that I’d had to share information in a cross team workshop.

I can’t share the full details of the presentation here. Below is a de-identified version of the final architecture slide to give you an idea of the complexity of the system that we discussed.

De-identified Enterprise Architecture overview  (Click for full size version)

De-identified Enterprise Architecture overview
(Click for full size version)

During the session I asked a question that I’ve also asked of the students, which is:

What is the most critical consideration when it comes to an n-tier architecture?

I always get a variety of answers including:

  • Money
  • Hardware
  • Software / programming language
  • Location of the datacenter

The answer that I’m looking for is that the users are the most critical consideration. This is because their requirements inform all of the considerations. Additionally users, and their requirements and needs, are the main reason for the system existing. Without users we may as well just go home and surf the interwebz.

It is easy to loose sight of this. In particular by those in technical roles who have little interaction with users. However I believe that users are critical to the overall success of the system. I’ve been involved in a number of enterprise system development and deployment projects. Those that struggled were the ones that users did not engage with, mainly because they didn’t see the system as meeting one of their needs.

Similarly their needs influence the architecture decisions. For example some users of an enterprise system, have a requirement that their data be stored in their country. It doesn’t matter if the system is housed in a state of the art datacenter, if it isn’t in their country they won’t use it.

A simpler example was a government department that made a significant investment in a Microsoft Excel based spreadsheet to collect data from the researchers at an institution where I worked. The spreadsheet was very complex. It included data validation and filtering, as well as restricting the information collected to that which they government department was interested in.

Unfortunately there were a significant number of researchers, who for a variety of reasons, didn’t have access to Microsoft Excel. Most notably those in computer science who preferred to use Mac OS X or a flavour of Linux. These users didn’t want, or simply couldn’t, use the spreadsheet. So they developed their own template and provided the information that they thought we needed. In some cases it was clear they had spent significant amounts of time collecting this data.

Unfortunately we then had to throw away most of that data as it didn’t fit within the specifications of what was required by the spreadsheet. In essence:

  • Key stakeholders didn’t engage in the process because the system, in this case the spreadsheet, didn’t meet their requirements
  • Users spent significant amounts of time collecting data that we couldn’t pass on
  • The government department missed out on data that could have been relevant
  • Users became disenfranchised with our efforts and didn’t engage in future activities

This is a long way of saying that in my personal opinion users are critical to the success of a system. With their needs and requirements informing decisions made at the strategic and architecture level, right through to bug fixing and communication.

Losing sight of this has adverse affects, from users losing trust in the system and those people working hard to support them, through to users becoming so disengaged with the system that they move elsewhere.

As someone working in a technical role, it can be useful to take a step back and try to see the problem from the users perspective.

Creswell Garden, Adelaide Australia

On the same day I took these photos of the Rotunda in Elder Park, I also took a couple of quick shots of the Creswell Garden sign on the eastern side of the Adelaide Oval.

Apparently the sign, and associated garden, was created to memorialise the work of John Creswell. It’s just a shame that with all of the money spent on the Adelaide Oval redevelopment in the last year that:

  1. The garden is now just a patch of lawn, the fountain that used to be there has gone
  2. No on spared any money to fix the finial on top of the right hand post

The Rotunda in Elder Park, Adelaide – Australia

Below is a gallery of photos that I took the other day on my way to work. I ended up running a little late, and even tweeted about it.

I hope to take more photos in the future and post them here.

New Utility Script: MdlDeprecatedCheck

'Extravaganza Deprecated' by Jan Kus

‘Extravaganza Deprecated’ by Jan Kus

For the past few weeks the focus of my work at NetSpot has been on Moodle upgrades. One of the many aims of an upgrade project is to ensure that there are minimal debugging messages being displayed during the testing process. One of the most common reasons for debugging messages being displayed is the use of functions that have been deprecated in the latest version of Moodle.

To help in fixing these issues before the client sees the message, I’ve developed a new utility script. The script is called MdlDeprecatedCheck and it is available in my GitHub repository. The script builds a list of functions, based on the deprecatedlib.php file, and compares it to the list of functions used in the target file or directory of files. If a match is found it is reported.

In this way I can identify the parts of the code that need updating to use the latest functions, and stop debugging messages from being displayed. As always, if you find any of my utility scripts useful, please let me know.

Using WordPress with HTTPS: Lessons Learned

'Learning' by Anne Davis 773

‘Learning’ by Anne Davis 773

A few days ago I wrote a post about why I switched my little part of the Interwebs to HTTPS. When I restarted my blog a few months ago I chose WordPress, mainly because I have been using it off an on for a number of years.

For example in the past I used WordPress to set up a centrally administrated blogging platform for the university that I was working for at the time. I even wrote a chapter on a small number of WordPress plugins that I wrote in a book on library mashups. The book was published in 2009.

I have learnt two new things from using WordPress with HTTPS.

Redirecting non HTTPS requests

The first lesson to learn was how to automatically redirect requests made using HTTP, to use the HTTPS version of the same URL. For example to automatically redirect from ‘http://techxplorer.com’ to ‘https://techxplorer.com’.

To do this I’ve added the following rewrite rules to my .htaccess file.

The important thing is to add these lines before the rules to make WordPress permalinks work. As an example, the relevant lines of my .htaccess file, including the rules for using permalinks, is as follows:

With this in place, any request for content using HTTP is automatically redirected to use HTTPS.

Fixing Absolute URLs to Post Content

The second lesson was that WordPress stores links to content within a post, such as the image above, as an absolute URL. That is, a URL that specifies the protocol, hostname and full path to the content. This is to ensure the link to the content works with the permalink format when the post was originally written, and in the future if the permalink format changes.

Including content loaded via HTTP when the page is loaded via HTTPS causes a mixed content warning to be displayed by the browser. This is because a page that contains mixed content is considered a security risk. More information on mixed content is available here.

WordPress uses a MySQL database to store post content. Therefore fixing the issue is a simple matter of using a SQL query like this:

Obviously if you’re intending to this with your own site you will need to:

  1. Take a backup of your database first, just in case
  2. Replace my domain name with your own

I remain pleasantly surprised by how easy this whole process has been. It has been so easy, that the cynical IT professional in me is just waiting for something bad to happen.

If you notice anything wrong with the site, please let me know.