NetMusician Labs

After several hours of tedious experimentation and research, I’ve finally come up with a solution to the common conundrum: how can pages be loaded via AJAX calls while still making this content accessible to the search engines?

The most common solution to this problem has usually and traditionally been one of the following:

1. Create separate pages for Javascript/AJAX compatible browsers and for the search engines
2. Don’t use AJAX at all, negotiate your content the old fashioned way - even if this results in a less user-friendly experience for your visitors
3. Use AJAX, accept the fact that your pages will not be crawlable by the search engines

For those of you that have played around with my media player on the NetMusician website you will have noticed that the beauty of this player is that the audio doesn’t cut in and out between page loads. Audio players that do this are a peeve of mine (especially ones that force audio on you with no way to turn them off!), I just think this is poor design.

The most common solutions to circumvent this problem have been:

1. Put the media player in a pop-up window
2. Put the media player in a frame
3. Load the content surrounding the media player as needed via an AJAX trigger

While solution #1 is often a workable solution, I think it is often rather awkward. Frames are a pain in the butt in general, and visitors can’t save direct URLs to the page they are on. AJAX is the nicest way to handle this, IMHO, but the search engine issue has always been a challenge.

Enter my Javascript function…

This function will read all of the HTML located within a div tag and will search for HTML hyperlinks. Internal, relative links are replaced with a Javascript link such as: javascript:ajax(’mypage’), external links and absolute links are left as is.

The result is that as long as you use internal links within your site, these can be converted into your AJAX/Javascript triggers on-the-fly via a Javascript onload event handler. For all non Javascript compatible browsers, this event handler will simply be ignored, leaving all of your links intact.

I believe this to be far easier than probing user agents and providing a separate set of links on the page for certain web browser user agents (or creating separate versions of the page). This makes it difficult to create links via a CMS, and is limited to the search engines you have provided user agent conditional statements for. This link rewriting technique, I believe, is a far more simplistic and effective approach.

I’ve uploaded a demo of this technique in action. Please leave a comment here if this is of use to you or if you have any questions… Enjoy!

Edit: modified function to work in IE

Search engine friendly AJAX demo

I have had great success and have really benefited from using the open source software RT (Request Tracker) as a trouble ticketing system for use with my clients, and also as a means to log time spent on projects and provide clients with access to the self-serve interface for providing read-only access their tickets so they can see exactly how their billable time was accounted for.

Since a number of organizations are using RT, I won’t go into detail as to how to set it up and use it for basic operation (unless you might find this guide written for my graphic designers detailing how to log time in RT of use). However, I will share some enhancements that have been quite useful.

One enhancement is creating a shared mailbox where RT ticket information is sent via its AdminCC function. I have configured myself as an AdminCC for each RT queue, and have setup mailfilter rules (to work with Courier IMAP, if you are using the Cyrus mail server you can create sieve rules) to file these messages into this shared mailbox based on whether the sender address includes the text “Firstname Lastname via RT”.

I have provided my designers with read access to these shared mailboxes so that all our correspondence on these projects is logged into this folder. As a bonus, we can append to these tickets simply by responding to them within our email out of this shared mailbox.

If you want to take this a step further, you can install the “CommandByMail” RT plug, which will allow you to resolve tickets and perform a number of other functions via email as well.

Since the time of writing my original post about using MacFUSE and SSHfs, an update to MacFUSE has been released. By default, this release deals with connection terminations more gracefully, bringing up a dialog asking you if you wish to still attempt to contact the server.

In my original set of instructions, I failed to include the flags necessary to deal with this gracefully (as it was able to do, but this wasn’t the software’s default behavior). I would update this document now, but this appears to no longer be necessary.

Apparently, a lot of users are using MacFUSE to handle NTFS file systems. Since I’m not a Windows person, I haven’t benefited from this feature, but this appears to be one of the compelling features of MacFUSE to a number of other users.

I really like the way that WordPress will write changes to its .htaccess file in order to generate nicer, mod_rewrite driven permalinks. I wanted a means for my customers to be able to create their own URLs that would correspond to WordPress generated Pages. This this feature doesn’t exist at time of writing, this is something I’ve been looking into writing myself and including within the Webkit.

Preface: I haven’t actually developed this solution yet, but I’m confident it will work. I will report back later with my findings…

My plan here is to read from the .htaccess file, and use pattern matching to filter out htaccess data unrelated to this task. This data will be dumped into an array line by line, and displayed within a form with a simple interface that would resemble something along the lines of:

Display WordPress page ID #: [ID] via URL: [URL]

where “ID” is a form field accepting input corresponding WordPress ID, and the URL is for typing in a URL such as “mypage.html”.

In addition, the array output will be sent along with form input as hidden form variables so that the form processor is able to do a search and replace of the old rewrite rule with the new one, using its own logic to generate the specific required pattern matching syntax. This output will be written back to the site’s .htaccess file, as long as the file is set with writable permissions.

For reference purposes, here is an example of an htaccess rewrite command that this Webkit component would manipulate:

RewriteRule ^mynewcd.html$ /wordpress/?page_id=34

Edit: the following instructions no longer seem to work. I have replaced these by writing to a file, picking up the contents of this file via a cron job, and using the included Mailman “add_members” binary to pick up the contents of this file.

As is the case with a number of web applications, one of the ongoing problems includes negotiating using a web interface as a bridge and functional interface into making system modifications securely, while acknowledging the fact that in most installs the Apache user will be different than the user(s) with the appropriate permissions to execute your Unix scripts and binaries over an SSH session. For commands and scripts that may affect a number of users on your system, typically only the root user may have executable permissions.

One such instance in which these restraints and challenges comes to the fore is in developing a more user-friendly, alternate Mailman web interface for users. Mailman is a superb mailing list with a robust plethora of features, but if you wish to make this available to a novice user simply to, say, subscribe users in bulk to an opt-in mailing, there are a lot of extraneous features the user will be forced to wade through. Additionally, I always thought that the unsubscribe process was a little unwieldy for a list configured to send out “one-way” mailings (i.e. a musician who wants to send out information to fans, but doesn’t want to invite responses to the entire list). I had envisioned it to be a simple form, preferably styled to match the look of the site. The subscriber in this instance does not need to be concerned with digest options and such, so there is no need for these options to even be displayed.

It seems like the two ways to go about customizing the web interface is to actually patch and markup the Mailman code base to your liking, or else simply piggyback upon the forms and binaries provided by Mailman, replacing the interface with your own.

Preface: I haven’t actually developed this solution yet, but I’m confident it will work. I will report back later with my findings…

I’ve developed my own subscribe and unsubscribe forms that mimic the ones provided by Mailman, setting up identical form fields that direct form output to the same Mailman pages. An example of my subscribe page can be found here. To enhance this, I setup a root cronjob to use the included Mailman list_members binary to generate a listing of members subscribed to the list, and save this output to a file that is inaccessible from the web. I then setup the subscribe email address field to trigger an AJAX request to grep for the email address in this list. If it is found, a “you are already subscribed to this list” message will be generated even before the form is submitted. You can use Javascript to hide the form submit button when the address provided has already been subscribed to the list.

Mailman’s normal form interface is designed to generate these errors, but I found them a little ugly and wanted to replace them with my own. I also wanted embed these feedback within my own site template. Mailman provides the means to modify the subscribe/unsubscribe feedback templates within its web interface, but I wanted to include PHP code in this template, and go beyond simple HTML. So, I decided to not rely on Mailman to generate this feedback, and simply replaced the Mailman subscribe feedback template with a meta redirect to a page of my own.

So, to summarize, I’ve basically replaced the logic behind examining an incoming subscription address with an AJAX call, and replaced the feedback that Mailman provides with pages of my own. The unsubscribe form can be designed in a very similar fashion.

What about providing mass subscription type functionality?

This is the part I’m still working on. I’ll need to find a script (or write one of my own) to convert vcard data to a list of addresses I can feed to Mailman, and I’ll need to design this web interface.

I intend for the web interface to work in a similar way as described above, where addresses are read from a flat file generated by the cronjob. Using this data, I can provide an AJAXey interface for manipulating the addresses in this list, and create another root cronjob to read this file an execute a remove_members and add_members to replace the list addresses with those in this file whenever this file is changed.

The one weakness of this approach is that there will always be a time delay and a lack of immediacy while the user waits around for the cronjob to run. This can be made a little more user-friendly by providing feedback that “your changes will take effect in 3 minutes”, or whatever, and having feedback emailed to that user’s email address.

Once the Webkit is released as an open source product, I may include this module for those that wish this functionality, in addition to my “Mailman HTML mailer” module.

I found the following guide explaining how Mediawiki wikis can be password protected with basic http/htaccess authentication. The Apache directives listed in step 5 of this guide will need to be wrapped in a <Directory> or <Location> tag, as warranted.

Unfortunately, there doesn’t seem to be a way at time of writing to password protect individual wiki pages, but you can use these instructions in conjunction with my “Managing multiple Mediawiki wikis from a single install” post instructions to separate out your wiki content into public and private areas.

Since I prefer to use the FreeBSD ports tree to install and update software on my server, it is a pain having multiple installs of different versions of the same software scattered across different directories. My goal has been to run all web-based software from a single code base, yet I want the data maintained by these multiple wikis to be separated.

Here is how this can be done in Mediawiki, allowing for separate wikis reading from separate databases:

1. Make a copy of a virgin Mediawiki database, perhaps calling the database “wikidb_template” (note, you’ll have to keep the schema stored within this DB up-to-date across Medawiki releases, this is explained below…)
2. Make a copy of Mediawiki’s LocalSettings.php file. Name it something like “LocalSettings-mywiki.php”, where “mywiki” is the name of your wiki.
3. Create symbolic links from your Mediawiki codebase to the directories you wish to be served by your Mediawiki wiki
4. Replace the original LocalSettings.php file with something like the following, matching “mydomain.com” with your domain, and the path names to paths matching your symbolic linked directories:

   if (preg_match('/(www\.)?yourdomain.com/', $_SERVER['SERVER_NAME']) && preg_match(’/^\/mywiki/’, $_SERVER['REQUEST_URI'])) {
// settings file for wiki in path starting with “/mywiki”
include ‘LocalSettings-mywiki.php’;
}
elseif (preg_match(’/(www\.)?yourdomain.com/’, $_SERVER['SERVER_NAME']) && preg_match(’/^\/anotherwiki/’, $_SERVER['REQUEST_URI'])) {
// settings file for wiki in path starting with “/anotherwiki”
include ‘LocalSettings-anotherwiki.php’;
}

5. Edit each LocalSettings file to modify the database connection information to match the database credentials you have established for each wiki
6. Create another LocalSettings file pointing to your wikidb_template database. This way, you’ll be able to update the schema to your virgin database as necessary using the Mediawiki web interface updater