Conducting a split-sample (split ballot) experiment

Sampling & Quotas
1

Hi,

Is there a simple way in SurveyEngine to implement a split-sample whereby 50% of respondents receive one experiment (DCE) and the other 50% receive a very similar experiment (e.g., the same experimental design) but that is worded slightly different (to test whether the specific wording has an impact on respondents preferences)?

The only required difference between the two samples would be that one of the attributes is worded slightly differently. From my understanding, data from the two experiments could be pooled and a binary variable introduced to differentiate the samples?

I understand that this is possible from the survey administration end (i.e., by creating two experiments in SurveyEngine and then sending to half of the respondents and pooling manually) but would like to know if this is possible in SurveyEngine itself?

Thanks in advance,

Rob

2

Hi Rob,

there are multiple solutions to control a split sample.

I’ll assume the following from your post to select a mechanism here:

  • The only difference you have between two versions of a DCE is a small change in wording within a text label (e.g. a level)
  • You want to have only one DCE in your survey (ie. only one DCE dataset in the end)
  • You want a control variable in your covariates datafile indicating what wording was shown
  • You want a random, but balanced 50/50 split in your data

Solution:

  • on a page before the experiment starts, add a ‘Deck’ element (How do decks work? with the value 2, generating a random without replacement variable [1;2] that will be equally distributed in your completed survey responses; let’s call that variable $label_group

  • add two ‘Derived’ elements (let’s call them $label_one and $label_two), which contain the two different text labels you want to use; and use a ‘display condition’ based on the deck element so only store one label for each group; (eg. $label_group == 1)

  • add a third derived value (let’s call it $label_shown), where we select (and store in the dataset) the actual label which is used. The content of that derived would be: $label_one || $label_two.
    Now, as all respondents get one Deck value and based on that only one of the two labels will be “set” the final Derived variable will always only contain the label for the split assigned.

  • finally, you need to call your $label_shown in the DCE level by changing to the code view in the textbox and adding the stored variable by using the following code: <span class="expression">$label_shown</span> either on its own or as part of existing text.
    While this can be achieved in the DCE Specification screen for levels, if you need to change a DCE attribute label, this needs to be done in the layout editor of the experiment.

This solution is not very elegant - but it does the job and does not require writing if/then logic or advanced perl coding. Applying those techniques can save adding a few of the ‘Derived’ fields explained above.

If your edits between the versions are more extensive, you might consider adding 2 DCEs to your survey and controlling which is to be displayed on each page by using the ‘Deck’ mentioned above and a ‘Display Condition’ for each scenario in the page editor. This solution has other implications, however, as your design rows might not be balanced automatically between the two DCEs for example.

As this is a customised response, please reach out to our team directly if you have additional questions.

Cheers,
Ludwig