The technical side of Commons v1.0

As Matt @admin announced earlier today, we’ve just tagged the CUNY Academic Commons “version 1.0”.

The 1.0 milestone represents a couple of big changes in our development processes.

  1. Versioning

    The most obvious change is the use of version numbers. It might seem at first like a relatively minor and superficial alteration in the way we operate. In truth, it’s a big change.

    Prior to the switch, our development cycles were haphazard at best. Bugs would be fixed and features implemented whenever someone on the development team happened to get around to it. That system (or lack thereof) works pretty well for a small team. But the team behind the Commons has been growing steadily over the past year, both in sheer number and in breadth (of geographical distribution, of areas of technical expertise, of responsibility). Greater distribution requires more thoughtful coordination and communication, and releasing discrete versions is a tool that we hope will help us achieve these goals.

    There are many benefits to discrete versioning. Here are a few prominent ones, off the top of my head:

    • Improved ticket management · Our ticketing system works best when tasks can be clearly assigned to individual developers, and organized according to priority. Version numbers provide a framework for this kind of management, as important tickets get assigned to earlier versions, and developer workload balance for a given release can be easily assessed.
    • Improved roadmapping · Roadmaps – software development language for “schedule of upcoming features” – are easier to understand and to implement when they’re organized by release numbers instead of thrown into a single pile.
    • Improved testing · When features and bugfixes are rolled out one-at-a-time, it can mean near-nonstop testing for the development and community team. Having controlled releases means that testing for a whole set of changes can be managed at the same time, in the days leading up to the release itself. That’s a better use of everyone’s time.
  2. Better distinction between development, staging, and production

    In those halcyon days early in the life of the Commons, nearly all development was done in the live production environment. When fixing a bug or launching a feature, I’d edit the site while it was up and running. Danger, Will Robinson! So many things can go wrong when you’re editing a live site, it’s hardly worth enumerating.

    The dev team eventually did set up a development environment. But it was running on a different server from the production site, and so you could never be guaranteed that a fix on the dev site would translate to the live site. Moreover, all those little fixes that would get sneaked directly onto the production site meant the dev site and the production site got progressively more out of synch as time went on, until the dev site was sufficiently different from the live site to be next to useless as a development site.

    Our new setup is much more sophisticated and, we hope, more conducive to quality development. We have a staging environment running in exactly the same operating environment as the production environment. Development environments are all local, which is to say that each member of the development team has a functional copy of the site on his personal computer. To whatever extent possible, development takes place in those local environments, is tested in our staging environment to look for environment-specific bugs, and only then pushed to the production environmnet. The (lofty) goal is that we’ll never have to make tweaks directly to the production site.

  3. Version control

    By “version control” I mean something a bit different from (though related to) “versioning” as mentioned above. Version control is a way to keep track of every change made to the files. It used to be that, when I tweaked a template or plugin file, unless I made a backup, the previous version would be overwritten and lost. Now that we’re using Git, we can always revert changes if need be (or browse the source out of mere historical curiosity!).

    The need for version control is all the more pressing as our development team grows. It’s important to keep track of who is changing what. And when you have local, independent development environments, it’s imperative that you have a way of making sure that the changes that I make on my version are not overwritten by the changes made by another member of the team when we merge them together in the staging environment. Git is designed for this kind of distributed development, as it keeps careful track of who edited what (‘git blame’, appropriately enough) and allows graceful merging of disparate edits.

    Using Git with WordPress and MediaWiki posed something of a technical challenge. For one thing, we wanted to track the entire codebase, but not files that users had uploaded to the system. Furthermore, MW and WP store crucial configuration information in the database, which is not tracked by Git. Finally, in order for local development environments to work properly, some environment-specific variables (database passwords, for instance) needed to be abstracted out of tracked files like wp-config.php. Zach at Cast Iron Coding forged a path through this jungle, and we’ve come up with a system that allows us all to keep our versions in sync using Git. In the future, I hope to write in a little more techinical detail just how this works.

    We use Git not just in our local development environments, but on the staging and production servers as well. For instance, when a release candidate has been vetted in the staging environment and is ready to go live, it’s extremely easy to launch the version on the production site: I simply shell into the production server and git pull or checkout the tagged version from the repository.

Adding an “email to members” checkbox to the BuddyPress group activity stream

During the recent upgrade from BuddyPress 1.1.x to BuddyPress 1.2.x, and the subsequent move away from group wires to interactive group activity streams, one thing that some users on the CUNY Academic Commons missed was the “Notify members by email” checkbox of the old wire.

This morning I wrote a bit of code to add that kind of functionality to group activity streams. There are three functions, each of which goes in your plugins/bp-custom.php file.

First, adding the checkbox to the activity box. Notice that it only shows up when you’re on a group page.

function cac_email_activity_checkbox() {
	if ( !bp_is_groups_component() )
add_action( 'bp_activity_post_form_options', 'cac_email_activity_checkbox' );

Second, handling the data when it gets to the server and sending the emails. Obviously, you’ll want to change the text of the email to match your own site and your own preferences.

function cac_email_activity_handler( $activity ) {
	global $bp;
	if ( $_POST['mailme'] == 'mailme' ) {
		$subject = sprintf('[CUNY Academic Commons] New update in the group "%s"',  $bp->groups->current_group->name );
		$message = strip_tags($activity->action);
		$message .= '

		$message .= strip_tags($activity->content);
		$message .= '

		$message .= sprintf('You recieved this message because you are a member of the group "%s" on the CUNY Academic Commons. Visit the group: %s', $bp->groups->current_group->name, $bp->root_domain . '/' . $bp->groups->current_group->slug . '/' . $bp->groups->current_group->slug . '/' );
		if ( bp_group_has_members( 'exclude_admins_mods=0&per_page=10000' ) ) {
			global $members_template;
			foreach( $members_template->members as $m ) {
				wp_mail( $m->user_email, $subject, $message );
	remove_action( 'bp_activity_after_save' , 'ass_group_notification_activity' , 50 );
add_action( 'bp_activity_after_save', 'cac_email_activity_handler', 1 );

Finally, you’ll need some Javascript to make the AJAX activity submission work correctly. This is really just a copy of what’s in the bp-default JS file, with a few added lines to make it work.

function cac_email_activity_js() {
	if ( !bp_is_groups_component() )

	var jq = jQuery;
	jq(document).ready( function() {
			/* New posts */
	jq("input#aw-whats-new-submit").click( function() {
		var button = jq(this);
		var form = button.parent().parent().parent().parent();

		form.children().each( function() {
			if ( jq.nodeName(this, "textarea") || jq.nodeName(this, "input") )
				jq(this).attr( 'disabled', 'disabled' );

		jq( 'form#' + form.attr('id') + ' span.ajax-loader' ).show();

		/* Remove any errors */

		/* Default POST values */
		var object = '';
		var item_id = jq("#whats-new-post-in").val();
		var content = jq("textarea#whats-new").val();
		var mailme = jq("#cac_activity_mail:checked").val();

		/* Set object for non-profile posts */
		if ( item_id > 0 ) {
			object = jq("#whats-new-post-object").val();

		jq.post( ajaxurl, {
			action: 'post_update',
			'cookie': encodeURIComponent(document.cookie),
			'_wpnonce_post_update': jq("input#_wpnonce_post_update").val(),
			'content': content,
			'object': object,
			'mailme': mailme,
			'item_id': item_id
			jq( 'form#' + form.attr('id') + ' span.ajax-loader' ).hide();

			form.children().each( function() {
				if ( jq.nodeName(this, "textarea") || jq.nodeName(this, "input") )
					jq(this).attr( 'disabled', '' );

			/* Check for errors and append if found. */
			if ( response[0] + response[1] == '-1' ) {
				form.prepend( response.substr( 2, response.length ) );
				jq( 'form#' + form.attr('id') + ' div.error').hide().fadeIn( 200 );
				button.attr("disabled", '');
			} else {
				if ( 0 == jq("ul.activity-list").length ) {
					jq("div.activity").append( '
    ' ); } jq("ul.activity-list").prepend(response); jq("ul.activity-list li:first").addClass('new-update'); jq("li.new-update").hide().slideDown( 300 ); jq("li.new-update").removeClass( 'new-update' ); jq("textarea#whats-new").val(''); jq("#cac_activity_mail").removeAttr('checked'); /* Re-enable the submit button after 8 seconds. */ setTimeout( function() { button.attr("disabled", ''); }, 8000 ); } }); return false; }); }); <?php } add_action( 'bp_activity_post_form_options', 'cac_email_activity_js', 999 );

New BuddyPress plugin: BP Include Non-Member Comments

By default, BuddyPress does not include comments from non-members (or non-logged-in users more generally) in the sitewide activity stream. For some communities, this default behavior is probably just fine. But in a community like the CUNY Academic Commons, which aims to attract readership and conversation from all sorts of folks, whether or not they’re signed in, the sitewide activity stream will be much more meaningful if it includes comments from everyone.

My new plugin, BP Include Non-Member Comments, was built with this purpose in mind. The plugin is activated here on the CUNY Academic Commons, so that all comments on your blog – whether from logged-in users or not – will show up on the news feed.

BP Include Non-Member Comments

The plugin has been tested on version 1.1.3 of BP, as well as the 1.2 release candidate. If you want to use the plugin for 1.1.3 or lower, you will need to uncomment the first few add_action and add_filter lines in the plugin file.

Technical caveat: Non-logged-in commenters have BP user_id 0. When BP creates the activity stream, it decides whether or not to show the Delete button by checking to see whether the user_id for the currently logged in user is the same as the user_id of the person to whom the comment belongs. Presumably, though, you don’t want non-logged-in viewers of the activity stream to be able to delete items from the activity stream at all. BP’s core code is not currently set up to make it easy to remove these buttons, so I employed an ugly fix. If you have changed your theme significantly from the default, you might have to adjust the filter bp_nonmember_comment_content (near the end of the plugin) to remove the button properly.

Download the plugin here.

Upgrading from BuddyPress 1.0 to 1.1

The upgrade from BuddyPress 1.0.x to BuddyPress 1.1 on the CUNY Academic Commons posed a number of challenges. Between the big changes to theme structure, the one-click installation of bbPress forums, and ongoing plugin compatibility issues, a lot of planning and testing was required. This week, I and the rest of the CAC development team finally made the switch. Over the course of a few practice upgrades in development environments, followed by an upgrade on the production site, I developed a pretty detailed set of instructions for myself. I thought I’d write up an annotated version of those instructions, for the benefit of anyone who might be facing a similar upgrade.

The Academic Commons had previously been set up roughly along the same lines as Jim Groom describes here: with bbPress storing its data in a separate database, but through the cunning use of various clever plugins, sharing user data, cookies, and BP group info with the WPMU installation. BP 1.1 does things differently: the one click installation of BP forums expects to store its data in tables that exist in the BP/WP database. In order to save the old data, we had two choices: move the old data into the new structure, or make BP 1.1 recognize the old db setup. Fellow dev team member Chris Stein and I decided that the former choice would be the easiest and most future proof. Here’s how it was done.

    Dealing with bbPress forums

  1. Move orphan forums into groups. Strictly speaking, this doesn’t have to be done first, but if you’ve got bbPress forums that aren’t attached to groups, you’ve got to get them in a group in order for them to display in BP. To the best of my knowledge, there’s no really way to do this from the bbPress or BP admin panels, but it’s easy to do manually. Set up a new group for each forum (a “Help” forum might get a “Help” group, for example), make sure the group has discussion forums enabled, and then find the line in table wp_bp_groups_groupmeta corresponding to the group_id of the new group and meta key `forum_id`, and change the meta_value to match the forum number of the orphan group. Rinse and repeat.
  2. Export your bbPress database to a local file.
  3. Replace HTML markup. bbPress (my installation at least) stored formatting in HTML. The most common markup has to do with line breaks: bbPress interprets a single line break in a forum post as <br />, and double line breaks as </p> and <p>. Opening the local version of the bbPress db in a plain text editor, I did a couple big search-and-replaces:
    • </p>\\n<p> became \n\r\n\r (redundant UNIX newlines)
    • Some <p> and </p> tags still remain (single paragraph posts), which I replaced with an empty string
    • <br /> became \n\r

    You’ll also need to make sure that all references to tables in the SQL dump are replaced with the proper names. For me, that meant replacing things like bb_posts with wp_bb_posts, but it might be different depending on the prefixes you use for bbPress and WP. If you’re not sure about how BP will name the tables, do the one-button forum setup in BP 1.1 (as a new installation) and post a test forum item, to make sure the forums get created. Then check out the database to see how BP formats table names on your install.

  4. Upgrading

    The instructions at buddypress.org are quite helpful. Here’s a short version:

  5. Deactivate BP-dependent plugins. I once neglected to do this. If you do, and then deactivate BP, there is a very good chance that your entire site will go white-screen-of-death, as the plugins will be making reference to BP functions that do not exist. Check your plugin list three or four times to make sure you’ve deactivated each one that has to do with BuddyPress. I like to make a list, so that I remember which ones to activate after the upgrade (though some, like the bbPress integration plugins, won’t be necessary).
  6. Back up all your data. Always! I like to take a snapshot of the whole database, and then an individual export of the wp_sitemeta table. That way, if I screw anything up (say, by deactivating plugins in the wrong order!) I can bring the site back to life by reverting to a previous state that is known to work.
  7. Upgrade BuddyPress. Deactivate, upload, reactivate.
  8. Reactivate BP dependent plugins. I didn’t have too many issues with plugin compatibility, but you might, depending on what you use. Some plugins will not be necessary anymore, while others (like Forum Attachments for BuddyPress) become useful.
  9. Themes

    The CUNY Academic Commons was in the midst of a redesign, so it made sense to have the new design implemented in BuddyPress’s new, parent-child theme environment. Buddypress.org has some instructions that you might find helpful. Our theme consists of a highly customized home page, header, and footer, with interior BuddyPress pages that are more or less similar in terms of markup to the BuddyPress parent theme. Thus it made sense for us to create a child theme in wp-content/themes (by copying and renaming the bp-default folder that comes with BuddyPress), make our CSS changes, and fill our child theme with only those templates where our markup had to differ from bp-sn-parent. For us that means home.php, header.php, footer.php, functions.php (which does not override but adds to bp-sn-parent/functions.php), and a few other miscellaneous files. How to implement:

  10. Rename bp-themes. If BuddyPress sees the bp-themes folder, it will assume that you’re going to use the older, deprecated theme structure. I renamed it to bp-themes-old, and will remove it altogether once we’re 100% done with the migration.
  11. Move bp-sn-parent, bp-default, and your child theme to the main WP themes folder. In most cases this is wp-content/themes.
  12. Activate the themes in Dashboard > Site Admin > Themes. No need to activate bp-sn-parent, since it’s not used directly.
  13. Apply the child theme of your choice (either bp-default or your custom theme) to the main blog.
  14. Setting up the forums

    You’ve exported and modified the forum data from the bbPress standalone version, but you’ll still need to get them into BP.

  15. Rename /forums. In our case, our previous installation of bbPress just happened to live at [BP install dir]/forums – exactly where the BP 1.1 forum directory goes. I don’t think it’d hurt anything to leave that folder named the way it is (because of the order in which BP handles URL arguments), but just to be safe I changed it to forums-old.
  16. Set up BP forums. In Dashboard > BuddyPress > Forum Setup, set up forums as a new installation.
  17. Drop the empty forum tables from the BP database. I can’t remember whether BP creates the forum tables upon installation or upon the first forum post, but in any case you won’t be able to import your old data properly without first dropping any table prefixed with wp_bb_.
  18. Import the modified local version of your forum data. I did it in PHPMyAdmin, which was easy because the file had been exported in the form of a huge INSERT command. If all goes well – and it should! – you should now be able to navigate to your groups and see the proper forum posts for that group.

There are kinks we’re still trying to work out of the system. Some strange things happened to some users’ avatars after the upgrade. And tweaking the theme is turning out to be a hassle in a lot of ways. But overall, the migration of data – which I had lost a bit of sleep over – went extremely smoothly. Good luck!

New WordPress plugin: Sitewide Tag Suggestion

WordPress’s Post Tags box, which appears on each Edit Post and Add New Post screen, has a great feature called “Choose from the most used tags in Post Tags”. This link allows you to see which tags you’ve used frequently in the past and apply them to your current blog post. The feature gives rise to two nice effects. One is that you avoid ending up with a lot of close-but-not-quite-identical tags on your blog (like “WordPress”, “WP”, “Word Press”, and so on) that, because they aren’t grouped together, tend to defeat the organizing purpose of tags. Another benefit is that it reminds you of the things you found important in past posts, and encourages you to mark your current post with the same tags, if relevant.

Sitewide Tag Suggestion
Sitewide Tag Suggestion

This plugin (and set of hacks) that I’m calling Sitewide Tag Suggestion allows you to glean the same benefits from the tag collection not just of a single blog, but of all the blogs on a WPMU installation. In short: STS adds another link to the Post Tags box, this one allowing authors choose from the entire community’s most popular tags when composing a post.

If you’ve got a blog on the CUNY Academic Commons, you can take advantage of Sitewide Tag Suggestion today. Just check out the Post Tags box when you’re writing a blog entry.

You can download the plugin here: sitewide-tag-suggestion.php.zip.

Detailed instructions for installation – which, partially as a result of my own ineptitude, involves a few hacks to WPMU’s core code – are contained inside of the file. Please note that WPMU 2.8+ is required, as is Donncha’s Sitewide Tags plugin.

Version 0.2 of BuddyPressActivity

Since releasing my MediaWiki extension BuddyPressActivity, which puts wiki edits into your BuddyPress activity stream, I’ve noticed a bug or two. In particular, the cool feature that filters out multiple edits of a page by a given author within a 24 hour period wasn’t working right. I’ve made the necessary fixes in v0.2.

Get the updated code today.

Missed Connections

sharing Boone Gorges, who is familiar to many of you as the homegrown rising star of our development team, recently traveled to Vancouver to take part in the Open Ed 2009 conference. From what I’ve gathered, Boone’s flights to Canada and back went relatively smoothly.

And indeed, far from missing connections, Boone was able to make some valuable ones, particularly with the development team at the University of British Columbia’s Office of Learning Technology. Knowing that UBC is working on projects that are similar to ours, Boone starting thinking about ways in which our Dev teams could do a better job of sharing things with one another — something that make sense, given the commitment we all have to open education. Boone speculated on this in a blog post titled “Sharing Hacks“:

Communication about code is a hard thing. On one end of the spectrum is internal communication. The gang at OLT keeps internal notes of the small hacks they do on their system, as do we at the CUNY Academic Commons. On the other end is end-user documentation, meant for a broad and largely non-technical audience. The kind of communication that’s missing here is the stuff in the middle, between groups doing similar sorts of work.

Boone ended that post by stating his commitment to writing more frequently on this development blog about the smaller issues of development. But his larger point — that those of us developing open-source educational platforms need to communicate more regularly with one another — is one that can’t be emphasized strongly enough.

And so, I will close this post with one of Boone’s central questions: “What are some good ways to get this kind of sharing moving?” His post, along with this one, represents a start, as does the revival of UBC’s development blog. But we need to build more lasting channels of communication soon, lest we miss some important connections.

Photo credit: Flickr user o.blaat

New plugin: Custom Profile Filters for BuddyPress

Custom Profile Filters for BuddyPress
Custom Profile Filters for BuddyPress - click picture to download plugin

If you’ve set up a profile here on the Commons (or on some other site run on BuddyPress), you may have noticed that some of the words and phrases in your profile have turned into links that, when clicked, lead you to other profiles where those words appear. This tagging feature is a great way to find out about people in the community who share your interests, but the algorithm that BuddyPress uses to create links can be somewhat finnicky. I built this plugin to allow users to customize these tags, choosing for themselves which phrases should be linked by surrounding them in square brackets.

Here’s an example. Let’s say that, in a profile field called Academic Interests, I said the following:

I’m interested in philosophy, chewing gum, and mariachi bands.

What I really want here is for the phrases “philosophy”, “chewing gum”, and “mariachi bands” to become links. So I’ll surround them in brackets like this:

I’m interested in [philosophy], [chewing gum], and [mariachi bands].

I’m submitting the plugin to the WordPress repository for versioning, but for now you can download version 0.1 here. Comments are welcome – have fun!

Custom Profile Filters for BuddyPress