Proposed CSV upload format
This is a proposed format for bulk uploading sensors. To be vetted and verified with Ensemble folks.
...
Sensor
...
ESID
...
Name
...
Lat_long
...
Reading_types
...
Depth
...
Brand
...
Model
...
Part_number
...
hardware_version
...
Description
...
The sensor’s unique external id. This is the identity that will be used to register the sensor with external services and receive readings associated with this sensor.
...
How the sensor will be labelled in LiteFarm.
...
The latitude and longitude for the sensor, comma separated.
...
Sensors are a generic term for any device that takes measurements from its environment and records them for observation. There are numerous use cases on a farm where sensors can be useful. Perhaps most top of mind is measuring rainfall in order to better understand how much water your crops are getting.
As an open source project, we will endeavour to be as open to integrating various types of sensors as possible.
CSV upload format
As of May 2023, users can upload X sensors on the farm map, by attaching a csv file with the following columns. The csv headers can be in English, Spanish, Portuguese, or French.
Column
Valid inputs
Possible errors
ID
1 <= id.length <= 20
“Sensor with id <id> already exists.”
“Sensor id invalidSensor | Name | Latitude | Longitude | Reading_types | External_ID | Depth | Brand | Model | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Description | How the sensor will be labelled in LiteFarm. | The latitude for the sensor. | The longitude for the sensor. | Comma delimited list of sensor reading types. Valid values are:
| The sensor’s unique external id. This is the identity that will be used to register the sensor with external services and receive readings associated with this sensor. | Assumed to be in cm, only 1 depth per sensor. For this project depths can be 10, 20, 30, 50. | The brand of sensor. | The model of sensor. | |||||||
Required?Data typeRequired | String | RequiredDecimalRequired | DecimalRequired | Specific strings | RequiredString | Optional | OptionalDecimal | String | String | ||||||
Required? | Required | Required | Required | Required | Optional | Required | Optional | Optional | |||||||
ExampleWERKTX | “Sensor 1” | -31.362442522541148, 3624425 | -64.210475444875952104754448 | soil_water_content, soil_water_potential, temperature | “WERKTX” | 10 | “Ensemble Scientific” | “Model ABC” | “ESA-12345” | “B” | Notes | Where the sensor has been placed in the ground. | Will need to be grouped on the map. This should be default behaviour since sensors are points. | ESCI Add device will fail without this attribute. | ESCI Add device will fail without this attribute. |
Definition of a sensor: A device with a single id, name, lat / long, and depth. Must have at least one reading parameter, but may have several. Optionally has a brand and model. Ensemble requires a part_number and hardware_version to register it with their service.
When uploading, LiteFarm needs to perform validation on the uploaded content.
Upload Validation | 1 <= name.length <= 100 | -85 <= lat <= 85 | -180 <= long <= 180 | At least 1 of the following:
| 1 <= id.length <= 40 | 0 <= depth <= 1000 | String: < 100 length | String: < 100 length |
Error | “Invalid sensor name, must be between 1 and 100 characters” | “Invalid latitude value, must be between -90 and 90. and fewer than 10 decimals” | “Invalid longitude value, must be between -180 and 180. and fewer than 10 decimals” | “Invalid reading type detected, valid values include:
| “Sensor with external id: <id> already exists.” “Invalid external id, must be between 1 and 20 characters” |
Name
1 <= name.length <= 100
“Sensor name invalid, must be between 1 and 100 characters”
Lat_long
-90 <= lat <= 90
-180 <= long <= 180
“Invalid format for lat_long, must be: 1.2345, 6.7890”
“Invalid lat entry, must be between -90 and 90.”
“Invalid long entry, must be between -180 and 180.”
Reading_types
At least 1 of the following:
soil_moisture_content, water_potential,
temperature
“Invalid type detected, must be: soil_moisture_content, water_potential, or temperature.”
Depth
0 <= depth <= 500; must be a decimal value. Assumed to be cm.
“Invalid depth, must be a decimal value between 0 and 500.”
Brand
String: < 100 length
“Brand must be fewer than 100 characters.”
Model
String: < 100 length
“Model must be fewer than 100 characters.”
Validation should be performed at least via the API and potentially through the UI as well.
Architectural guidance
Please take this as guidance rather than instruction. The following tables seem to make sense to allow users to document sensors and to integrate them with those external services.
Sensor: The attributes describing the physical device and it’s location. This should somehow tie to a point location for that sensor.
sensor_parameter: The parameters that instruct LiteFarm on how to parse data read by the sensor but don’t change from reading to reading and also aren’t related to the physical nature of the sensor. May just be part of the sensor table!
sensor_reading: Readings from the sensor.
...
Sensor
...
ID
...
Name
...
Lat_long
...
Type
...
Depth
...
Elevation
...
…
...
Description
...
The sensor’s unique id.
...
How the sensor will be labelled in LiteFarm.
...
The latitude and longitude for the sensor, comma separated.
Valid values to begin are:
...
Assumed to be in cm, comma delimited if multiple depths.
...
The elevation of ground level at the location where the sensor is mounted.
...
Required?
...
Required
...
Required
...
Required
...
Required
...
Optional
...
Optional
...
Example
...
“Sensor 1”
...
-31.362442522541148, -64.21047544487595
...
sensor_parameter
...
ID
...
Sensor_id
...
parameter_number
...
Type
...
Parameter_measured
...
Unit
...
Min_value
...
Max_value
...
Description
...
Unique identifier
...
Reference to a specific sensor.
...
For sensors with multiple types, this calibration would define 1 specific type.
...
Required?
...
Example
...
Sensor_reading
...
ID
...
Read_time
...
Transmit_time
...
Sensor_ID
...
sensor_parameter_id
...
Value
...
Description
...
A unique identifier for this reading. May have some meaning or traceability for Ensemble.
...
The timestamp when this reading was captured.
...
The timestamp when this reading was transmitted.
...
The sensor reporting the reading.
...
The calibration which instructs on how to interpret the value. Required since each sensor can have multiple types of readings, e.g. “Temperature” and “Moisture”.
...
The numerical value of the reading.
...
Required?
...
Required
...
Required
...
Required
...
Required
...
Required
...
Example
Questions:
How does LiteFarm get a non-expiring token from the Ensemble API?
Probably manually enter this into an “integrations” table to similar to how certifiers have a surveystack link. Ultimately up to ESCI and the owner of LF-2483.
Will LiteFarm need a different token for each organization or will one token allow us to access multiple organizations?
Ideally one token per farm, but may have one universal instead. To verify with ESCI team.
What are the valid types of sensor (reading) types to start with?
soil_water_content (derived)
soil_water_potential
temperature
...
parameter
...
parameter_number(s)
...
explanation
...
soil_water_content
...
Ask Morten!
...
soil_water_potential
...
Ask Morten!
...
temperature
...
Ask Morten!
Mark says that parameter_number is dynamic from device to device. Will need to develop a strategy to make this consistent.
How do we link up sensor_reading_types with a depth via csv upload?
Each sensor will only have 1 depth. We may choose to display sensors with very similar lat / long and different depths as the same sensor via the UI - but they are different in terms of sensor reading types and readings.
How does ensemble tie a reading to a parameter when posting new sensor readings?
It appears to be based on “parameter_number”. Need to explore what exactly this means to ESCI.
Code Block |
---|
"node_esid":"123",
"sensor_data":[
{
"sensor_esid":"124",
"sensor_values":[
{
"parameter_number":1,
"raw_value":2.910,
"timestamp":"2022-01-10T17:41:00+0800"
},
{
"parameter_number":1,
"raw_value":2.110,
"timestamp":"2022-01-10T17:41:00+0800"
},
{
"parameter_number":2,
"raw_value":0.001938,
"timestamp":"2022-01-10T17:41:00+0800"
}
]
... |
What is the format for registering an organization with ESCI?
https://documenter.getpostman.com/view/18529342/UVXeqchj#b3a26ae9-be68-42a4-ac92-ca8025e7008c (see POST Create an organization)
Code Block |
---|
curl --location --request POST 'api.esci.io/organizations/' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "tes_org",
"uuid": "76df1f72-6fb0-11ec-90d6-0242ac120003",
"website_url": "http://test.com",
"email": "orgemail@hotmail.com",
"phone": "123-4567",
"street_number": "12345",
"street": "Easy Street",
"city": "Vancouver",
"postal": "A1B2C3",
"state": "British Columbia",
"country": "Canada"
}' |
Authenticated users can create an organization.
"name" is the only required field
The creator will automatically become the admin and member of organization
Preferred UUID for organization can be optionally specified. Otherwise it will be autogenerated
Returns an error if UUID incorrect or non-unique
What is the format for registering a callback address with ESCI?
https://documenter.getpostman.com/view/18529342/UVXeqchj#b3a26ae9-be68-42a4-ac92-ca8025e7008c (assumed to be POST Create organization webhook)
Code Block | ||
---|---|---|
| ||
curl --location -g --request POST 'api.esci.io/organizations/{{organization_uuid}}/webhooks/' \
--data-raw '{
"url": "http://www.example.com",
"authorization_header": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX3BrIjoxLCJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiY29sZF9zdHVmZiI6IuKYgyIsImV4cCI6MTIzNDU2LCJqdGkiOiJmZDJmOWQ1ZTFhN2M0MmU4OTQ5MzVlMzYyYmNhOGJjYSJ9.NHlztMGER7UADHZJlxNG0WSi22a2KaYSfd1SAuT7lU",
"frequency": 15
}
// Example json that webhook will post to the url
// [
// {
// "node_esid": "123",
// "sensor_esid": "124",
// "parameter_number": 1,
// "unit": "ºC",
// "time": [
// "02-09-2022, 22:53:52",
// "02-09-2022, 22:54:07"
// ],
// "value": [
// 1,
// 2
// ],
// "validated": [
// false,
// false
// ]
// },
// {
// "node_esid": "123",
// "sensor_esid": "124",
// "parameter_number": 2,
// "unit": "kPa",
// "time": [
// "02-09-2022, 22:54:23"
// ],
// "value": [
// 3
// ],
// "validated": [
// false
// ]
// }
// ]' |
Create webhook for organization
Will send a confirmation request in the form of an empty array to the url (with authorization if any) before being created
Webhooks automatically send any organization data between current time and the webhook's last transmit time based on the webhook's frequency as long as it is active. Will send an empty array if no new data
authorization_header is optional
frequency is in minutes
Errors
400_BAD_REQUEST
Invalid json, incorrect url
received Response other than 200_OK from url during creation
Errors when trying to send confirmation request to the url
Only organization admins can call
Do we need a different callback address for each organization or can the callback specify org as part of the hierarchy?
Does the callback only send what has happened since the last update? Is there a failsafe if a callback fails to communicate several rounds of data?
Yes, ESCI will keep track of what has been shared and only send updates.
What is the format for registering sensors with ESCI?
https://documenter.getpostman.com/view/18529342/UVXeqchj#b3a26ae9-be68-42a4-ac92-ca8025e7008c (assumed to be POST Add a new device)
Code Block |
---|
curl --location --request POST 'api.esci.io/devices/' \
--data-raw '{
"part_number": "ESA-12345",
"hardware_version": "B"
}
// example response.data
// {
// "esid": "GLMCHJ",
// "device_type": {
// "id": 1,
// "name": "Temperature Pressure Sensor",
// "part_number": "ESA-1001",
// "device_category": "S"
// },
// "hardware_version": "B",
// "parameters": [
// {
// "parameter_number": 1,
// "category": "Temperature",
// "unit": "ºC",
// "label": "Area 1"
// },
// {
// "parameter_number": 2,
// "category": "Soil Water Tension",
// "unit": "kPa",
// "label": "Area 2"
// }
// ]
// }
' |
Adds device to cloud with the specified POST data:
part_number
hardware_version
Device ESID will automatically be generated and any device parameters will be created based on the DeviceType's default parameter list if it exists.
Any generated parameters will have a 1:1 calibration generated with it as well
If device is interfacer, sets configurable to TRUE, so users can modify the device with other APIs
Errors:
400_BAD_REQUEST
Device type with part_number does not exist
Any fields missing
Only ESci admins can call
What depths are the sensors at (for this project)?
10, 20, 30, and 50cm. Duplicates at 10 and 30.
Are there any attributes above that are missing?
What data types can we have for sensor reading values?
soil_water_content: Derived from water_potential and temperature
water_potential: Sensed
temperature: Sensed
Requirements
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Summary
Description
Fix Version
Sensor model work
Build out database to support tables for:
Sensors (location)
Sensor reading types (see ESCI doc)
Sensor readings
MVP
Bulk or individual?
v2
Upload screen with template and front-end validation
v1
Bulk sensor creation and outcome splash
v1“Invalid depth, must be a decimal value between 0 and 1000.” | “Invalid brand, must be fewer than 100 characters.” | “Invalid model, must be fewer than 100 characters.” | ||||||
Notes | May need to group sensors with same / similar lat / long on the map. This should be default behaviour since sensors are points. For this project depths can be 10, 20, 30, 50. | If this is “Ensemble Scientific”, it should activate the need to register this sensor during the bulk creation endpoint. |
Info |
---|
Definition of a sensor: To be decided at stand-up on Monday, May 8th Lite Farm Current implementation: farm_id, partner_id, external_id Other approaches:
Sensors must have a depth, but a single sensor might have more than one depth (TBD). Must have at least one reading parameter, but may have several. Optionally has a brand and model. Sensors are moved from season to season, so the same name with a different lat / long is a likely scenario. |
Validation should be performed via the API and through the UI during bulk upload as well.
Architecture
As of May 2023, sensors have the following tables that support their operations:
farm_external_integration: Mapping of how LiteFarm and external partners refer to the same
farm.integrating_partner: Sensor companies LiteFarm integrates with. Currently “0” for non-supported and “1” “Ensemble Scientific”.
partner_reading_type: The types of sensor readings supported by each integrating partner.
sensor_reading_type: A junction table holding a reference to the sensor (location), partner reading type, and sensor reading type. For a sensor with multiple reading types, multiple entries will exist.
sensor: The attributes describing the physical device and it’s location that aren’t held in the location table.
sensor_reading: Readings from the sensor.
farm_external_integration | farm_id | partner_id | organization_uuid | webhook_id |
---|---|---|---|---|
Description | Foreign key to farm table. | Foreign key to integrating_partner table. | The integrating partner’s UUID for the same farm. | The webhook for the integrating partner to provide updates to this farm specifically. Incrementing integer |
Required? | Yes; LiteFarm generated | Yes | Yes | No |
Example | “d58e64d2-f72e-11ec-a4f4-0242ac120003” | “f42cde0d-7c14-4a1d-8eab-c367ce8cfe16” | 174 | |
Notes |
integrating_partner | partner_id | partner_name | access_token | refresh_token | root_url | deactivated |
---|---|---|---|---|---|---|
Description | Incrementing integer representing the unique identifier for this partner. | The name of the integrating partner | Access token passed to the foreign API when communicating. | Access token passed to the foreign API to renew the access token if it has expired. | Root URL to prefix all communication with this integrating partner | |
Required? | Yes; set by LiteFarm | Yes | No | No | No | Yes |
Example | 0 | “Ensemble Scientific” | FALSE | |||
Notes | Currently “0” for non-supported and “1” “Ensemble Scientific”. |
partner_reading_type | partner_reading_type_id | partner_id | raw_value | readable_value |
---|---|---|---|---|
Description | UUID | Foreign key to integrating_partner table | Not sure | Supported reading type for this partner. |
Required? | Yes; LiteFarm generated | Yes | No | Yes |
Example | “d58e64d2-f72e-11ec-a4f4-0242ac120003” | “0” | ||
Notes | Could be how the reading_type is represented from the partner, if for example there is a difference between that value and how the user uploads the sensor csv - e.g. “temp” vs. “temperature”. | For MVP, supported values are temperature and soil_water_potential. |
sensor_reading_type | sensor_reading_type_id | partner_reading_type_id | location_id |
---|---|---|---|
Description | UUID | UUID | UUID |
Required? | Yes; LiteFarm generated | Yes; LiteFarm generated | Yes; LiteFarm generated |
Example | “d58e64d2-f72e-11ec-a4f4-0242ac120003” | “d58e64d2-f72e-11ec-a4f4-0242ac120003” | “d58e64d2-f72e-11ec-a4f4-0242ac120003” |
Notes |
sensor | location_id | partner_id | external_id | Depth | Elevation | Model | Depth_unit |
---|---|---|---|---|---|---|---|
Description | The location associated with this sensor. | Linkage to the integrating_partner table. | The unique identifier with the integrating partner for this particular sensor. | Assumed to be in cm, comma delimited if multiple depths. | The elevation of ground level at the location where the sensor is mounted. | The model of the sensor. | The depth of the sensor. |
Required? | Connection is LiteFarm generated | Connection is LiteFarm generated | Required | Optional | Optional | LiteFarm generated | LiteFarm generated |
Example | 0 | “WERKTX” | “cm” | ||||
Notes | Note that the name, lat / long, and other physical attributes are held in the location table. | At time of writing “0” for “All others” and 1 for “Ensemble Scientific”. | Not implemented in MVP. | Not implemented in MVP. | Always “cm”. |
Sensor_reading | reading_id | location_id | Read_time | created_at | reading_type | value | unit | valid |
---|---|---|---|---|---|---|---|---|
Description | A unique identifier for this reading. | The location that represents the sensor that created this reading. | The timestamp when this reading was captured on the integrated device. | The timestamp when this reading was created in the LiteFarm database. | The reading type which (should) corresponds with a partner_reading_type | The numerical value of the reading. | The unit that when combined with the value represents the reading. | Boolean value on whether this reading has passed some sort of external QA process. |
Required? | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
Example | “ce7c5f2c-c77c-11ed-86ec-0242ac120002” | “2ec78a8a-a993-11ed-ac8f-0242ac120002” | 2023-03-20 23:55:21.739103+00 | 2023-03-21 00:10:37.243388+00 | “temperature” | 20.191444 | “Celsius” | FALSE |
Notes | Why doesn’t this reference a partner_reading_type? |
The above architecture could probably be streamlined.
Sensor registration flow
Testing and the bulk unclaim script
The Ensemble team has provided several dummy sensors to us for testing purposes. You can also find documentation on their API here:
Each of these sensors generates synthetic data on a regular basis to help with testing. For each, the brand should be “Ensemble Scientific” and the External_id as follows:
QLCD8Q - Will register, but no data feed
5GFLSS - Will register, but no data feed
D36B32 - Will register, but no data feed
SHJ44V - Will register, but no data feed
7XZPYL - Will register and generate a data feed
LTEXWC - Will register and generate a data feed
MBZ69B - Would generate a feed, but is being used by Ensemble for testing and so can’t be registered
JRVALF - Will register and generate a data feed (claimed by Joyce / on beta)
BDCDKN - Will register and generate a data feed
TEE2TG - Will register and generate a data feed (Claimed by Denis on BETA)
9MP6V9 - Will register and generate a data feed (claimed by Joyce / local)
9XF8Z5 - Will register and generate a data feed
GSUU4U - Will register and generate a data feed (claimed by Joyce / local)
AW72KG - Will register and generate a data feed(claimed by Mwaya)
Note |
---|
Please note that a sensor can only be registered to ONE ORGANIZATION at a time, so you’ll need to coordinate which sensors should exist on each farm. |
When you’re done with a sensor, you can use the following unclaim script to unregister that sensor from your farm and return it to the pool of sensors that can be used for testing. You’ll need the username and password for the Ensemble API and to modify the esid’s (External_id’s) listed.
Code Block |
---|
import requests as re
url = "https://api.esci.io"
def get_tokens():
payload = {
"username": "ASK_KEVIN",
"password": "ASK_KEVIN"
}
headers = {"Content-Type": "application/json"}
response = re.post(f"{url}/token/", json=payload, headers=headers)
response_data = response.json()
return response_data['refresh'], response_data['access']
def get_device_org_uuid(access_token, esid):
headers = {"Authorization": f"Bearer {access_token}"}
response = re.get(f"{url}/devices/{esid}", headers=headers)
response_data = response.json()
if "owner_organization" not in response_data:
return None, True
return response_data['owner_organization']['uuid'], False
def unclaim_sensor(access_token, esid):
headers = {"Authorization": f"Bearer {access_token}"}
payload = {"esid": esid}
org_uuid, unclaimed = get_device_org_uuid(access_token, esid)
if not unclaimed:
response = re.post(f"{url}/organizations/{org_uuid}/devices/unclaim/", json=payload, headers=headers)
print(response.text, f"device: {esid}")
def bulk_unclaim(esid_list):
_, access_token = get_tokens()
for esid in esid_list:
unclaim_sensor(access_token, esid)
if __name__ == "__main__":
bulk_unclaim(["7XZPYL", "MBZ69B"]) |
Sensor failure cases
To be documented by Denis Dovganyuk and Mwayanjana Bolokonya (Unlicensed).
Case | Response | |
---|---|---|
No validation error when the user uploads the soil_water_content sensor | 200 - Successful upload | |
Although the user is not able to upload a sensor with depth more than 1000 via API , the user can manually change the depth of the sensor to 10000 using the APP. | ||
The user is able to upload the sensors with the longitude / latitude consists more than 10 decimals. | ||
Questions:
Requirements
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Summary | Description | Fix Version |
---|---|---|
Sensor model work | Build out database to support tables for:
| MVP |
Bulk sensor upload | MVP | |
Individual sensor upload | TBD | |
Upload screen with template and front-end validation | MVP | |
Bulk sensor creation and outcome splash | MVP | |
Support for soil_water_content reading_type | This is a derived value that can be calculated based on the soil texture and soil water potential. | V1 |
Support for multiple depths within 1 sensor | V1 |
Future enhancements
Support for ‘soil_water_content’ reading type. This is a derived value that can be calculated based on the soil texture and soil water potential.
Approach for appropriately clustering sensors that have the same lat / long but different depths. Potential approaches:
Only display 1 sensor and have readings tab show selector for which depths should be shown (We would need to allow duplicates of the same external id to enable this).
To investigate
Ensemble requires a part_number and hardware_version to register it with their service.