Ingestion of Acartia data by the WRAS (Whale Report Alert System)

[scottveirs]

What might be the most efficient way to ingest near real time data from the Acartia data cooperative into the WRAS (Whale Report Alert System)?

Let's bandy about all possible solutions, for both short-term cooperation via the phase 1 MVP, and longer-term resilience, including through refinement of the Acartia data scheme, new features, and documentation.

[scottveirs]

I think the path of least resistance for integrating a subset of Acartia's real time data into WRAS would involve the following technical steps:

1. Follow the Acartia.io CONTRIBUTING guidance to register and obtain a token
2. Query the Acartia API every N minutes (e.g. N=5) for data with these attributes: 

2a. created datetime (currently with format "2021-11-19 19:54:37" and UTC time zone) is less than the age of useful data ] for ingestion

2b. data_source_name = "Spotter-API" (this constrains you to just data provided only through conserve.io apps, not e.g. an Excel upload source and not acoustic detections from Orcasound hydrophones)

2c. trusted = "1" (only accepted moderated / confirmed sightings from Conserve.io's Spotter server

Then filter the query result for only the species that WRAS wants to transmit to end-users (i.e. not birds, only a subset of the marine mammals in Acartia), mapping Acartia species names into the WRAS ones.

Please, Ali and Virgil, don't hesitate to recommend a different path, or modify/discuss any of the steps I've proposed.

Other thoughts:

If you want to implement a workflow ASAP, you could just use the existing API call retrieve-current-sightings and then implement filter for 2a, 2b, and species of interest.

An alternative to 2a for obtaining recent data could be to use the ssemmi_id field. This is a unique sequential ID for the Acartia record. In your case, I think you could store the most recent value obtained, e.g. ssemmi_id = "SPOTTER138809", and then query for any new records that have numerical values greater than 138809. This could be very efficient in the current situation (i.e. with mostly real time data coming in, primarily from trusted sighting network moderators), but inefficient if someone uploads a large archive of old whale locations, e.g. via .csv upload.

[scottveirs]

Feel free to offer thoughts here @aalaydrus . I'll invite others to experiment with the Github Discussion, tool, too!