[development] Drupal Administration survey II -- looking volunteers to do interviews

[Gary Feldman]

Kieran Lal wrote:
> On Jul 31, 2006, at 10:54 AM, Gary Feldman wrote:
> ...
>> Can you state some more specific goals for the survey?
> To understand Drupal administrators situation when they are 
> administering, their goals, and the tasks they are trying to complete.
That's still pretty vague.  It could be addressed by a single question 
that just asks administrators to describe such things.  But in order to 
come up with more specific (and useful) questions, it's good to have 
more specific goals.

If you're really just trying to improve your abstract understanding and 
appreciation of Drupal administrators, contextual inquiry would be 
better than a survey, and better than a scripted interview (though both 
could be done in one visit).
> The survey was part of a larger effort to improve Drupal 
> administration. The survey specifically helped to identify tasks 
> Drupal administrators were trying to accomplish so we could improve 
> their ability to complete those tasks.
Forgive my nitpicking, but unless there's something more complicated 
about the way the survey was done than indicated by the results, the 
survey started out with its own list of identified tasks.  There are 
about thirty in question 7.  Or did the survey actually ask those 
questions with no (or just a couple) of tasks listed, and then somebody 
took the prose results and organized them into a manageable number of 
tasks?

Question 8 appears to have identified a handful of tasks not on the 
original list, although it's not clear to me whether those are all tasks 
that administrators do or things they want (e.g. does "group tasks 
logically" mean that administrators have their own tasks that somehow 
need to be grouped?  or more likely, do they want Drupal's 
administration tasks to be grouped logically?)  That's good as far as it 
goes, as long as it's understood that they're giving their own 
perceptions and conclusions, which don't necessarily reflect reality. 
> Surveys serve a narrow purpose.  They allow broad participation from 
> the community as a whole and they help provide feedback to the Drupal 
> core development process.  I would use different user experience 
> techniques to evaluate some of these measures.  For example, I use 
> analysis of search terms on Drupal.org and comments in the Drupal 
> handbook to track what people are interested in and what they are 
> having trouble learning.
Narrow in the sense that they need to be focused?  Or in the sense that 
they're limited in what they can do?  I agree with both, which is why 
I'd like to see good, focused goals.  I also agree that a combination of 
various types of user data is good.
>
>> Another way to phrase my question is what decisions do you hope to 
>> make based on the results of the survey?
> I can't make decisions for the larger community, but I would hope that 
> developers, consulting firms, and Drupal site owners would choose to 
> put their resources to improving the most difficult and important 
> tasks identified in the survey.
Ah, finally some specific questions:  What's difficult to do?  What's 
important to work on? 

For the stuff that's difficult, the next question is what makes it 
difficult?  For some specific tasks, there might be useful survey 
questions.  For example, terminology was difficult for 30% of the 
respondents,  so it might be interesting to ask how hard is it to find a 
definition and once you've found it, how hard is it to understand.  But 
for something like administering the structure of a site (32% found it 
difficult), is it because people have hard things they want to do with 
the structure?  Or is it because they didn't structure it well in the 
first place?  Or is it because they're unaware of features that would 
make it easier?  There might be some good survey questions to ask for 
those, especially if you bring to bear the collective knowledge about 
Drupal administrators (which is much greater than my own), but I can't 
think of any such questions. 
> I would use the results to direct where CivicSpace makes it's 
> investments in improving the user experience of administering Drupal.  
> I would encourage and validate others efforts to do likewise.  For 
> example, in the last survey we identified that making your theme work 
> across all browsers was the most difficult Drupal administration 
> task.  If that result was validated again in this survey I'd probably 
> post emails and contact consulting firms and customers encouraging 
> them to fund Drupal theming improvements.
Is that the item that reads "Manage inconsistency in themes"?  Should 
that be "inconsistency in browsers"? 

Regardless, this is a good example of the limits of a survey (if I 
understand it correctly).  This may well be something that's high on 
their list, but it's not obvious how that applies to Drupal (especially 
when the quickest solution might be to just wait for IE 7 to become 
popular, assuming MS fully and correctly supports CSS 2.1 with it).  It 
might be that the most productive thing Drupal could do here would be to 
recommend some other open source tools that focus on this particular 
problem.
> In the survey I identified categorization as being the third most 
> "Very Difficult task".  I didn't understand why. I conducted a small 
> follow up survey for a dozen people to understand why categorization 
> was important.  What I learned was the for non-profits and advocacy 
> groups it was very important that they are able to communicate the 
> structure of their organization and the goals of their organization 
> through their website categorization.  I also learned that these users 
> treated categorization as three distinct tasks: managing categories, 
> navigating by categories, organizing by categories.    That lead to a 
> review of over 20 taxonomy modules and we built a taxonomy garden to 
> make it easier for Drupal administrator to understand how to use 
> categories and the available modules.  You can see the results of that 
> work here:
>
> http://drupal.org/handbook/modules/taxonomy
> http://drupal.org/node/47822  Managing categories
> http://drupal.org/node/47623 Navigating by categories
> http://drupal.org/node/47527 Organizing content by categories
>
I'm not really sure what to make of these.  From the titles, I was 
expecting something more in the way of strategies than module 
descriptions.  In this context, it might be most helpful to understand 
the mappings between the user data and the modules.  For example, how 
does Taxonomy XML fit in?  Are there specific problems in the user data 
that it solves?  Or is the point merely that the taxonomy modules were 
organized in a way that corresponds to the user's distinctions? 

My conclusions:

In trying to put this all together in my mind, I'm wondering is the need 
for a second survey to assign priorities to user tasks and problems 
(which is the way I would describe the results of the first survey, 
instead of the more general "get a better understanding").  If so, then 
what were the problems or deficiencies of that survey? 

Or would it be more valuable, to pick the top 3-5 issues from the first 
survey, and collect data around those?

These are the sorts of questions that would help me construct a survey 
(or decide to use another method for collecting user data at the 
moment).  And no, I'm not pretending to be a survey statistician or 
other expert; I'm coming from a background in requirements gathering and 
management.

Gary