Sensors

Sensors are a generic term for any device that takes measurements from its environment and records them for observation. There are numerous use cases on a farm where sensors can be useful. Perhaps most top of mind is measuring rainfall in order to better understand how much water your crops are getting.

As an open source project, we will endeavour to be as open to integrating various types of sensors as possible.

CSV upload format

As of May 2023, users can upload X sensors on the farm map, by attaching a csv file with the following columns. The csv headers can be in English, Spanish, Portuguese, or French.

Sensor	Name	Latitude	Longitude	Reading_types	External_ID	Depth	Brand	Model

Sensor	Name	Latitude	Longitude	Reading_types	External_ID	Depth	Brand	Model
Description	How the sensor will be labelled in LiteFarm.	The latitude for the sensor.	The longitude for the sensor.	Comma delimited list of sensor reading types. Valid values are: soil_water_potential temperature soil_water_content	The sensor’s unique external id. This is the identity that will be used to register the sensor with external services and receive readings associated with this sensor.	Assumed to be in cm, only 1 depth per sensor.	The brand of sensor.	The model of sensor.
Data type	String	Decimal	Decimal	Specific strings	String	Decimal	String	String
Required?	Required	Required	Required	Required	Optional	Required	Optional	Optional
Example	“Sensor 1”	-31.3624425	-64.2104754448	soil_water_potential, temperature	“WERKTX”	10	“Ensemble Scientific”	“Model ABC”
Upload Validation	1 <= name.length <= 100	-85 <= lat <= 85	-180 <= long <= 180	At least 1 of the following: soil_water_potential temperature soil_water_content	1 <= id.length <= 40	0 <= depth <= 1000	String: < 100 length	String: < 100 length
Error	“Invalid sensor name, must be between 1 and 100 characters”	“Invalid latitude value, must be between -90 and 90. and fewer than 10 decimals”	“Invalid longitude value, must be between -180 and 180. and fewer than 10 decimals”	“Invalid reading type detected, valid values include: soil_water_potential temperature soil_water_content	“Sensor with external id: <id> already exists.” “Invalid external id, must be between 1 and 20 characters”	“Invalid depth, must be a decimal value between 0 and 1000.”	“Invalid brand, must be fewer than 100 characters.”	“Invalid model, must be fewer than 100 characters.”
Notes						May need to group sensors with same / similar lat / long on the map. This should be default behaviour since sensors are points. For this project depths can be 10, 20, 30, 50.	If this is “Ensemble Scientific”, it should activate the need to register this sensor during the bulk creation endpoint.

Definition of a sensor: To be decided at stand-up on Monday, May 8th @Lite Farm

Current implementation: farm_id, partner_id, external_id

Other approaches:

A device with a single Name, Brand, and external_id, and farm_id.

Sensors must have a depth, but a single sensor might have more than one depth (TBD). Must have at least one reading parameter, but may have several. Optionally has a brand and model.

Sensors are moved from season to season, so the same name with a different lat / long is a likely scenario.

Validation should be performed via the API and through the UI during bulk upload as well.

Architecture

As of May 2023, sensors have the following tables that support their operations:

farm_external_integration: Mapping of how LiteFarm and external partners refer to the same
farm.
integrating_partner: Sensor companies LiteFarm integrates with. Currently “0” for non-supported and “1” “Ensemble Scientific”.
partner_reading_type: The types of sensor readings supported by each integrating partner.
sensor_reading_type: A junction table holding a reference to the sensor (location), partner reading type, and sensor reading type. For a sensor with multiple reading types, multiple entries will exist.
sensor: The attributes describing the physical device and it’s location that aren’t held in the location table.
sensor_reading: Readings from the sensor.

farm_external_integration	farm_id	partner_id	organization_uuid	webhook_id

farm_external_integration	farm_id	partner_id	organization_uuid	webhook_id
Description	Foreign key to farm table.	Foreign key to integrating_partner table.	The integrating partner’s UUID for the same farm.	The webhook for the integrating partner to provide updates to this farm specifically. Incrementing integer
Required?	Yes; LiteFarm generated	Yes	Yes	No
Example	“d58e64d2-f72e-11ec-a4f4-0242ac120003”		“f42cde0d-7c14-4a1d-8eab-c367ce8cfe16”	174
Notes

integrating_partner	partner_id	partner_name	access_token	refresh_token	root_url	deactivated

integrating_partner	partner_id	partner_name	access_token	refresh_token	root_url	deactivated
Description	Incrementing integer representing the unique identifier for this partner.	The name of the integrating partner	Access token passed to the foreign API when communicating.	Access token passed to the foreign API to renew the access token if it has expired.	Root URL to prefix all communication with this integrating partner
Required?	Yes; set by LiteFarm	Yes	No	No	No	Yes
Example	0	“Ensemble Scientific”			“https://api.esci.io/”	FALSE
Notes	Currently “0” for non-supported and “1” “Ensemble Scientific”.

partner_reading_type	partner_reading_type_id	partner_id	raw_value	readable_value

partner_reading_type	partner_reading_type_id	partner_id	raw_value	readable_value
Description	UUID	Foreign key to integrating_partner table	Not sure	Supported reading type for this partner.
Required?	Yes; LiteFarm generated	Yes	No	Yes
Example	“d58e64d2-f72e-11ec-a4f4-0242ac120003”	“0”
Notes			Could be how the reading_type is represented from the partner, if for example there is a difference between that value and how the user uploads the sensor csv - e.g. “temp” vs. “temperature”.	For MVP, supported values are temperature and soil_water_potential.

sensor_reading_type	sensor_reading_type_id	partner_reading_type_id	location_id

sensor_reading_type	sensor_reading_type_id	partner_reading_type_id	location_id
Description	UUID	UUID	UUID
Required?	Yes; LiteFarm generated	Yes; LiteFarm generated	Yes; LiteFarm generated
Example	“d58e64d2-f72e-11ec-a4f4-0242ac120003”	“d58e64d2-f72e-11ec-a4f4-0242ac120003”	“d58e64d2-f72e-11ec-a4f4-0242ac120003”
Notes

sensor	location_id	partner_id	external_id	Depth	Elevation	Model	Depth_unit

sensor	location_id	partner_id	external_id	Depth	Elevation	Model	Depth_unit
Description	The location associated with this sensor.	Linkage to the integrating_partner table.	The unique identifier with the integrating partner for this particular sensor.	Assumed to be in cm, comma delimited if multiple depths.	The elevation of ground level at the location where the sensor is mounted.	The model of the sensor.	The depth of the sensor.
Required?	Connection is LiteFarm generated	Connection is LiteFarm generated	Required	Optional	Optional	LiteFarm generated	LiteFarm generated
Example		0	“WERKTX”				“cm”
Notes	Note that the name, lat / long, and other physical attributes are held in the location table.	At time of writing “0” for “All others” and 1 for “Ensemble Scientific”.			Not implemented in MVP.	Not implemented in MVP.	Always “cm”.

Sensor_reading	reading_id	location_id	Read_time	created_at	reading_type	value	unit	valid

Sensor_reading	reading_id	location_id	Read_time	created_at	reading_type	value	unit	valid
Description	A unique identifier for this reading.	The location that represents the sensor that created this reading.	The timestamp when this reading was captured on the integrated device.	The timestamp when this reading was created in the LiteFarm database.	The reading type which (should) corresponds with a partner_reading_type	The numerical value of the reading.	The unit that when combined with the value represents the reading.	Boolean value on whether this reading has passed some sort of external QA process.
Required?	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Example	“ce7c5f2c-c77c-11ed-86ec-0242ac120002”	“2ec78a8a-a993-11ed-ac8f-0242ac120002”	2023-03-20 23:55:21.739103+00	2023-03-21 00:10:37.243388+00	“temperature”	20.191444	“Celsius”	FALSE
Notes					Why doesn’t this reference a partner_reading_type?

The above architecture could probably be streamlined.

Sensor registration flow

https://lucid.app/lucidchart/57b61ad1-0fa5-4fb4-bdb8-e753c2cd549f/edit?viewport_loc=-1716%2C-915%2C4305%2C2168%2C0_0&invitationId=inv_b24cabbe-67d9-457d-853e-30e351438487

Testing and the bulk unclaim script

The Ensemble team has provided several dummy sensors to us for testing purposes. You can also find documentation on their API here:

https://api.esci.io/docs/

Each of these sensors generates synthetic data on a regular basis to help with testing. For each, the brand should be “Ensemble Scientific” and the External_id as follows:

QLCD8Q - Will register, but no data feed
5GFLSS - Will register, but no data feed
D36B32 - Will register, but no data feed
SHJ44V - Will register, but no data feed
7XZPYL - Will register and generate a data feed
LTEXWC - Will register and generate a data feed
MBZ69B - Would generate a feed, but is being used by Ensemble for testing and so can’t be registered
JRVALF - Will register and generate a data feed (claimed by Joyce / on beta)
BDCDKN - Will register and generate a data feed
TEE2TG - Will register and generate a data feed (Claimed by Denis on BETA)
9MP6V9 - Will register and generate a data feed (claimed by Joyce / local)
9XF8Z5 - Will register and generate a data feed
GSUU4U - Will register and generate a data feed (claimed by Joyce / local)
AW72KG - Will register and generate a data feed(claimed by Mwaya)

Please note that a sensor can only be registered to ONE ORGANIZATION at a time, so you’ll need to coordinate which sensors should exist on each farm.

When you’re done with a sensor, you can use the following unclaim script to unregister that sensor from your farm and return it to the pool of sensors that can be used for testing. You’ll need the username and password for the Ensemble API and to modify the esid’s (External_id’s) listed.

import requests as re

url = "https://api.esci.io"


def get_tokens():
    payload = {
        "username": "ASK_KEVIN",
        "password": "ASK_KEVIN"
    }
    headers = {"Content-Type": "application/json"}

    response = re.post(f"{url}/token/", json=payload, headers=headers)

    response_data = response.json()

    return response_data['refresh'], response_data['access']


def get_device_org_uuid(access_token, esid):
    headers = {"Authorization": f"Bearer {access_token}"}
    response = re.get(f"{url}/devices/{esid}", headers=headers)
    response_data = response.json()
    if "owner_organization" not in response_data:
        return None, True
    return response_data['owner_organization']['uuid'], False


def unclaim_sensor(access_token, esid):
    headers = {"Authorization": f"Bearer {access_token}"}
    payload = {"esid": esid}
    org_uuid, unclaimed = get_device_org_uuid(access_token, esid)
    if not unclaimed:
        response = re.post(f"{url}/organizations/{org_uuid}/devices/unclaim/", json=payload, headers=headers)
        print(response.text, f"device: {esid}")


def bulk_unclaim(esid_list):
    _, access_token = get_tokens()
    for esid in esid_list:
        unclaim_sensor(access_token, esid)

if __name__ == "__main__":
    bulk_unclaim(["7XZPYL", "MBZ69B"])

Sensor failure cases

To be documented by @Denis Dovganyuk and @Mwayanjana Bolokonya (Unlicensed).

Case	Response

Case	Response
No validation error when the user uploads the soil_water_content sensor	200 - Successful upload
Although the user is not able to upload a sensor with depth more than 1000 via API , the user can manually change the depth of the sensor to 10000 using the APP.
The user is able to upload the sensors with the longitude / latitude consists more than 10 decimals.

Questions:

See Sensor Q&A

Requirements

https://lite-farm.atlassian.net/browse/LF-2338

Summary	Description	Fix Version

Summary	Description	Fix Version
Sensor model work	Build out database to support tables for: Sensors (location) Sensor reading types Sensor readings	MVP
Bulk sensor upload		MVP
Individual sensor upload		TBD
Upload screen with template and front-end validation		MVP
Bulk sensor creation and outcome splash		MVP
Support for soil_water_content reading_type	This is a derived value that can be calculated based on the soil texture and soil water potential.	V1
Support for multiple depths within 1 sensor		V1

Future enhancements

Support for ‘soil_water_content’ reading type. This is a derived value that can be calculated based on the soil texture and soil water potential.

Approach for appropriately clustering sensors that have the same lat / long but different depths. Potential approaches:

Only display 1 sensor and have readings tab show selector for which depths should be shown (We would need to allow duplicates of the same external id to enable this).

To investigate

Ensemble requires a part_number and hardware_version to register it with their service.