Allow LiteFarm to access private SurveyStack data

Use case

For many reasons, farm data can be sensitive. Part of our commitment to our users is that their data should default to private unless they explicitly share it. One use case where this is particularly important, is when the users answers additional survey questions for their certifying body via the SurveyStack iFrame via the certification export flow. The expectation of the farmer in this case is that data is only shared with:

• LiteFarm

• their certifying body

To ensure this, we need to make sure data transmitted to their certifying body in SurveyStack is only visible to LiteFarm and the certifier in transit and at rest.

Current situation

As of February 2023, the connection with SurveyStack (see https://lite-farm.atlassian.net/browse/LF-1558 for more information) only works if all of the questions and attributes of the survey are public. The moment a question is set to “Private”, we’re no longer able to access the question or answer.

Open

Desired state

Moving forward, we need to support the ability to:

• View private surveys and survey fields in LiteFarm (via the iFrame)

• Create export documents with private SurveyStack fields

Solutions

The only individuals that are able to view private question submissions in SurveyStack are:

1. The submitter

2. An administrator on the group that owns the survey

Since surveys will be anonymously submitted from LiteFarm, that leaves only the second option as viable. Via the UI, administrative access is proven by logging in and having an auth token stored in the browser local storage (see auth_header in the image below):

Open

However, since we’ll be interacting with SurveyStack via the API, we’ll need to pass this as a “bearer token” to SurveyStack instead:

Within this solution, there are 2 paths we could take:

1. As recommended by Greg Austic of Our-Sci (the makers and maintainers of SurveyStack): Have all certifiers create their surveys with sub-groups under the LiteFarm group (see image). With this approach, administrators within the LiteFarm group would automatically have administrator access to all sub-groups and all surveys (and fields) created within each sub-group. Using this approach, there would be a single auth_token used to access all SurveyStack survey submissions for LiteFarm - regardless of the survey.

2. Alternatively, we could allow each certifier to have their own SurveyStack instance. In this case, they would just need to invite (someone at) LiteFarm as an administrator. We would then log in, grab the auth_token associated with that user and augment our certifiers table with an auth_token column (as shown in the image below). Using this approach, there would be a single auth_token for each certifying body.

Shape of the query

The query to retrieve the JSON for a particular submission appears as follows:

https://app.surveystack.io/api/submissions?survey={survey_id}&match={"_id":{"$oid":"{submission_id}"}}

Example:

Work to be done

Two pieces of work here:

1. A decision: decide which solution (of those outlined in Allow LiteFarm to access private SurveyStack data | Solutions or proposed as an alternative) to move forward with

2. To augment the current communication between LiteFarm and SurveyStack to provide authenticated movement of data. This may potentially include augmenting the schema as well.