-
-
Notifications
You must be signed in to change notification settings - Fork 25
Data Sets
All of the Bgeigie log files are uploaded to api.safecast.org. Additional pointcast fixed radiation sensors publish data into this database.
This service has a public rest API. Examples are available at https://api.safecast.org/home and Swagger definition is available at https://api.safecast.org/api_docs.json
Current Safecast Radiation Dataset
Complete Radiation dataset (zipped), updated daily. This is a large download.
The Imbrogno Dataset contains data out of the following cities:
Uppsala, Hong Kong, Seoul, Taipei, Amsterdam, Perth, Sydney, Honolulu and Munich
The ingest.safecast.org endpoint was developed for the new Solarcast project to support a much more flexible data model. The data stores for this app contain all the Solarcast data as well as data from pointcast and various fixed sensors.
Data is exported every hour to s3 available via standard S3 tools. For example with the aws cli:
❯ aws s3 ls s3://safecastdata-us-west-2/ingest/prd/s3raw/2017-10-01/
The format is in lines of JSON, delimited by two line feeds. The JSON objects are wrapped first with a versioning envelope, then the payload key in that envelope which is the payload that was posted to ingest.safecast.org.
Example:
❯ aws s3 cp s3://safecastdata-us-west-2/ingest/prd/s3raw/2017-10-01/00/12 - | head -n4
{"version":1,"payload":{"device":10004,"when_captured":"2017-09-30T23:12:38Z","loc_lat":34.4883,"loc_lon":136.165,"loc_alt":535,"loc_olc":"8Q6RF5Q7+8XH","lnd_7318u":2100,"dev_test":true,"service_uploaded":"2017-09-30T23:12:38Z","service_transport":"pointcast:210.142.99.156","service_md5":"f0b668c27bcd19ad316cfff6308918b6","service_handler":"i-0c65ac97805549e0d"}}
{"version":1,"payload":{"device":178,"when_captured":"2017-09-30T23:12:54Z","loc_lat":37.05676,"loc_lon":-122.138336,"loc_olc":"849V3V46+PM3","lnd_7318u":46,"service_uploaded":"2017-09-30T23:12:54Z","service_transport":"ngeigie:107.161.164.166","service_md5":"94a54e3e2fc510030bc2d3a38b4206c2","service_handler":"i-051a2a353509414f0"}}
Each device is permitted to publish their own fields but here are some of the fields expected from our Solarcast device fleet:
-
when_captured: Device timestamp when measurement was taken -
service_uploaded: Device timestamp when measurement was uploaded -
loc_lat: Device latitude -
loc_lon: Device longitude -
opc_pm01_0: Alphasense OPC-N2 air sensor, PM 1.0 (units: μg/m³) -
opc_pm02_5: Alphasense OPC-N2 air sensor, PM 2.5 (units: μg/m³) -
opc_pm10_0: Alphasense OPC-N2 air sensor, PM 10.0 (units: μg/m³) -
pms_pm01_0: Plantower PMS5003 air sensor, PM 1.0 (units: μg/m³) -
pms_pm02_5: Plantower PMS5003 air sensor, PM 2.5 (units: μg/m³) -
pms_pm10_0: Plantower PMS5003 air sensor, PM 10.0 (units: μg/m³) -
pms2_pm01_0: Plantower PMS5003 air sensor (secondary), PM 1.0 (units: μg/m³) -
pms2_pm02_5: Plantower PMS5003 air sensor (secondary), PM 2.5 (units: μg/m³) -
pms2_pm10_0: Plantower PMS5003 air sensor (secondary), PM 10.0 (units: μg/m³) -
lnd_712u: Unshielded LND 712 radiation sensor (units: μSv/h) -
lnd_712c: Shielded LND 712 radiation sensor (units: μSv/h) -
lnd_7318u: Unshielded LND 7318 radiation sensor (units: μSv/h) -
lnd_7318c: Shielded LND 7318 radiation sensor (units: μSv/h) -
lnd_7128ec: Energy-compensated LND 7128 radiation sensor (units: μSv/h) -
lnd_78017u: Unshielded LND 78017 radiation sensor (units: μSv/h) -
lnd_78017c: Shielded LND 78017 radiation sensor (units: μSv/h) -
lnd_78017w: Water-attenuated LND 78017 radiation sensor (units: μSv/h) -
env_temp: External temperature (units: °C) -
env_humid: External humidity (units: RH%) -
env_press: External air pressure (units: hPa)
Data that comes into ingest.safecast.org is also available in kibana, available anonymously at:
Or authenticated at:
- https://03a7d5abd1d836ffe2772e95abc7dd25.us-west-2.aws.found.io:9243 - kibana UI
- https://40ad140d461d810ac41ed710b5c7a5b6.us-west-2.aws.found.io:9243 - elasticsearch rest endpoint
Authenticated access requires an account to access but anyone is welcome. Please email [email protected] to request an account.
Data that comes into ingest.safecast.org is also available in grafana as well.
This is publicly browsable, or if you're added to the Safecast github organization you can authenticate with your github account to be given write access.
We also have a publicly accessible postgres replica available at ingest-replica1.prd.safecast.cc
This system requires an account to access but anyone is welcome. Please email [email protected] to request an account.
Once you have a login you can access the database via:
❯ psql -h ingest-replica1.prd.safecast.cc -U your_user_name safecast
Password for user your_user_name:
See https://github.com/Safecast/ingest/blob/master/db/structure.sql for schema information. All measurements are initially stored into the measurements table.
All ingest data also feeds through this AWS SNS topic:
-
arn:aws:sns:us-west-2:985752656544:ingest-measurements-prd.
This topic is set to allow public subscriptions so simply attach an HTTP endpoint or SQS queue from your AWS console or CLI to receive points at they come in.