[development] Drupal administration survey draft based on 10 interviews

[Kieran Lal]

On Aug 9, 2006, at 7:23 AM, Gary Feldman wrote:

> Kieran Lal wrote:
>> ...
>> How would you describe yourself as a Drupal administrator?
>> (select all that apply)
> This, and several others, seem to be "pick the best one" questions,  
> not "select all that apply" questions.

ok.
>> New user
>> Non-technical user developing community or social change web site  
>> forced to become "accidental technologist"
> The nature of the web site doesn't seem relevant.  How about:  "  
> Non-technical user developing a web site, and becoming adept at the  
> technology"

Yes, it's very relevant.  See interviews and response to Dries.

>> Inexperienced with Drupal but curious. Learning fast based on  
>> other web site building experience
> Should this just be "Inexperienced with Drupal but learning fast,  
> based on ..."?

Ok, better.

>> Experienced user, comfortable with configuring site rapidly
> Perhaps "Experienced administrator, ..." would be better.  I expect  
> most participants won't have the benefit of the recent discussions  
> on terminology for users, administrators, developers, etc.
>> ...
>> How frequently do you administer you your Drupal site?
>> (select all that apply)
> Again, pick the best one.

Let me review this again.  Many people see themselves as wearing  
several hats(particularly because they run more than one site), so I  
am hesitant to stick them into a single category.

>> ...
>> How long do you administer your Drupal site in a single sitting  
>> approximately?
>> (select all that apply)
> Ditto

>> ...
>> How does Drupal help you accomplish your goals as a web site  
>> administrator?
>> (select all that apply)
>> Rapid deployment of features and ability to meet customer  
>> requirements saves time
> Strike "saves time", it's implied by "rapid".

ok.

>> Web based content publishing is easy and allows for end user  
>> contributions
>> Learnable website that can be taught to users and allows website  
>> developers to leverage existing learning when building new sites
> This seems to be two separate items.

You are right.  Drupal administrators learning curve, and Drupal site  
developer learning curve.  I had hoped to cluster into one response.   
See response to Dries.


>> Allows for monitoring and logging of the website
>> Module configuration, extensibility with new modules, and clean  
>> code make it easier to configure the website as  you need it
> How about "easy to customize", just to read better?

Ok.

>> Dynamic and exciting community allows for rapid and fun learning
> Does community mean "Drupal user community"?  Or the community that  
> the person is creating?

Drupal developer community (core + contribs + consulting + groups +  
mailing lists + forums + themes).  I'll clarify.

>> ...
>> How does Drupal help your users?
>> ...
>> Cost effective and winning over non-technical decision makers
> I can't parse this.  Should it be "Cost effective, easy to convince  
> non-technical decision makers"?

Agreed. I'll re-do it.

>> ...
>> What are some common Drupal administration tasks? (select all that  
>> apply)
> Should this be "select all", "select no more than 3", or something  
> else?  In this case, "select all" is meaningful, but I don't think  
> it's the right question.

I'll use Dries suggestion.

> I'd also change it to read "What are _your_ ...", so it's clear  
> it's about what they do, not what they think Drupal administrators  
> in general do.
>> ...

Ok, good feedback.

>> What are some infrequent Drupal tasks?
>> (select all that apply)
> Ditto, but see next comment.
>> Add new features by adding modules or coding new features
>> ...
>> Modify site navigation
>> Other_______________
> The fact that this list differs from the preceding list suggests  
> that you already believe these to be infrequent.  But does that  
> mean that if the respondent doesn't check something, it means that  
> it's a frequent task, or that it's not a task at all?  Or more to  
> the point, what is this question trying to learn?

Think about the 400* options in Drupal's administration interface  
today.  Some should easily accessible as they are routinely  
accessed.  Think about the hundreds of professional Drupal  
consultants who live in the Drupal administration interface day in an  
day out.  For example logs or spam seem to be the most frequently  
accessed.   However, a task like clean URLs is only usually set once,  
but it has prominent real estate at administer >> settings.   
Infrequent tasks should be tucked away in favor of more common tasks,  
but made easy to find during site set-up.

The first set of questions are about understanding the users.  The  
second set of questions are about the situation the user is in when  
they are doing administration.  In general I would say that the  
notion of administration configuration and administration settings  
need to give way to: Site set-up, Site development, site testing and  
maintenance.  At least this is what the interviews reveal about how  
Drupal is actually being used.

> Should it be something like "These are some Drupal tasks that are  
> done less frequently.

infrequent, less frequently.  Potato , Potato, Tomato, Tomato.  Pick  
the one that is grammatically correct and let me know!

> Which of these do you consider important?"

That question is asked below.
>>
>> When you administer your site you find it easy to:
>> (1 is not easy, 5 is very easy)
>> ...
>> When you administer your site you find it hard to?
>> (1 is not hard, 5 is very hard)
> The duality here is a bit more meaningful, but still confusing.   
> Why not combine these into one question?

Response rate for long questions falls off significantly.   I want to  
get the response rates consistent and high to avoid skewed results.

> Regardless, having the first one be "1 is not easy" and the second  
> one be "1 is not hard" will cause people to mess up.  If you want  
> to keep these as two separate questions, then perhaps it would work  
> better if you asked "Do you agree or disagree that the following  
> are easy tasks?", and "Do you agree or disagree that the following  
> are difficult tasks?", with 1 being strongly agree and 5 being  
> strongly disagree.  I'm not positive this would be better, but it  
> feels better to me.

Ok, I'll consult with Charlie Lowe, who taught audience research at  
Purdue, and get his input on your suggestion.

> Another alternative would be to just move the questions away from  
> each other, so that the respondent doesn't trip over the flipping  
> of the scale.

Ok, one of the reasons I use SurveyMonkey is that it jumbles the  
order of questions to avoid leading bias.  Insert your favorite joke  
here!


>> What other important tasks did not fit into the categories above  
>> for you?
> Which categories?  There's frequent/infrequent and easy/hard.  If  
> something isn't checked, does that mean it's not important, or that  
> it's neither easy nor hard, or what?

I don't think we can infer anything from a non-selection.

> My expectation when I read "What other important tasks" is that  
> there will be a text area for me to insert arbitrary items.

Yes.  The part that says Othere________________  will be a text field.


> If these have already been identified as possible tasks, why aren't  
> they just included above?

It's a challenge.  As the interviews progressed the responses got  
more and more varied.  By the time I was clustering the answers for  
the important tasks there were 20+ unique important tasks for 10  
interviewees!

>> ...
>> Setting up a testing process including creating a test role,  
>> soliciting feedback from users, configuring corrections, and  
>> working with module development process to get bugs resolved
> Should this be split into a testing item and a bug fix item?

Yes, it's a casualty of trying to cluster responses.  I am trying to  
keep the number of responses short so that we keep the survey  
response rates high, at the same time get enough granularity in the  
responses so that we can take action .   It's a tough balance.

>
> I hope this is helpful.  It it's not, let me know and I'll shut up.
>
It's absolutely helpful.  The more review, the clearer the survey  
questions, the more accurate the responses, the clearer the analysis,  
and the greater capacity the developer community has to address the  
needs of their users.

Cheers,
Kieran
> Gary
>
>