I'm pretty deep into the arrow ecosystem and have been dreaming about something like this for a while now. Nice to see!
I'm also imagining what this would look like in other languages (Java, Go, Python) - you can't do as much at compile time of course, but there are still marginal improvements to be made over the dynamism of the standard arrow libraries, when the use case benefits from it.
Cool, curious if there is any willingness or interest in adding a geoarrow feature, or generally some form of integration with that crate? (Though to be honest I'm not currently clear up to date on the stability/usability of geoarrow at this current point in time)
My project hasn't really gotten to the level of actually exporting data to any database yet, but basically I'm just using standard openstreetmap data exported to geoparquet format.
That is then combined with a bunch of time series sensor readings combined with a gps data, (I'm basically still working on the device firmware so I don't have any actual data export happening yet)...
But it is a pretty basic combination of timeseries and geospatial data.
I'm pretty deep into the arrow ecosystem and have been dreaming about something like this for a while now. Nice to see!
I'm also imagining what this would look like in other languages (Java, Go, Python) - you can't do as much at compile time of course, but there are still marginal improvements to be made over the dynamism of the standard arrow libraries, when the use case benefits from it.
Cool, curious if there is any willingness or interest in adding a geoarrow feature, or generally some form of integration with that crate? (Though to be honest I'm not currently clear up to date on the stability/usability of geoarrow at this current point in time)
Sure I'd love to, could you share your practice or what you are doing on top of Parquet?
My project hasn't really gotten to the level of actually exporting data to any database yet, but basically I'm just using standard openstreetmap data exported to geoparquet format.
That is then combined with a bunch of time series sensor readings combined with a gps data, (I'm basically still working on the device firmware so I don't have any actual data export happening yet)...
But it is a pretty basic combination of timeseries and geospatial data.
I had no idea what Arrow is: https://arrow.apache.org or arrow-rs: https://github.com/apache/arrow-rs
How is this different from https://github.com/Swoorup/arrow-convert?
typed-arrow avoids any kinds of runtime side effects: cost, errors, etc.
I congratulate you. This is a clever extension of a great framework.