[P1] Run tool over batch of input addresses

alistairmcintyre commented

2022-10-17 10:43:28 +13:00

(Migrated from gitlab.catalyst.net.nz)

Wilfred wants to run the tool for a large number of input addresses (thousands) and any other required input data.

alistairmcintyre commented

2022-10-17 10:46:42 +13:00

(Migrated from gitlab.catalyst.net.nz)

changed title from [1] Run tool over batch of input addresses to [{+P+}1] Run tool over batch of input addresses

changed title from **[1] Run tool over batch of input addresses** to **[{+P+}1] Run tool over batch of input addresses**

alistairmcintyre commented

2022-10-17 11:21:03 +13:00

(Migrated from gitlab.catalyst.net.nz)

There's a few options here:

Management script or admin panel action that takes in data in a format (assuming CSV) and runs the scripts on those inputs
Access controlled REST API endpoint that also takes in a CSV file as input and generates output.

Both solutions if they must return PDFs could end up returning a large number of files, or a single very large archive. Either way this could use up a lot of data.

Also some performance considerations may need to be made depending on how the PDF is generated. @mattn to check into this before an estimate.

There's a few options here: 1. Management script or admin panel action that takes in data in a format (assuming CSV) and runs the scripts on those inputs 2. Access controlled REST API endpoint that also takes in a CSV file as input and generates output. Both solutions if they must return PDFs could end up returning a large number of files, or a single very large archive. Either way this could use up a lot of data. Also some performance considerations may need to be made depending on how the PDF is generated. @mattn to check into this before an estimate.

mattn commented

2022-10-17 12:08:17 +13:00

(Migrated from gitlab.catalyst.net.nz)

using the built in library, generates a CSV file from a queryset/iterable of Plant instances and saves it to disk (becomes downloadable)
using Pandas, reads the CSV file, converts it to an HTML <table>, and saves it to disk
using pdfkit, reads the HTML file, converts it to PDF, and saves it to disk
using PyPDF2, reads a static user guide PDF, reads the generated PDF, concatenates them, and saves the result to disk

This does too many disk IO operations and should probably use Python's buffer objects to save intermediate data that won't be served to the end user, i.e. the HTML file and generated PDF.

Still need to consider if we want a CSV+PDF per-address or if there should be a bulk output format. Should check with Wilfred.

1. using the built in library, generates a CSV file from a queryset/iterable of `Plant` instances and saves it to disk (becomes downloadable) 2. using Pandas, reads the CSV file, converts it to an HTML `<table>`, and saves it to disk 3. using pdfkit, reads the HTML file, converts it to PDF, and saves it to disk 4. using PyPDF2, reads a static user guide PDF, reads the generated PDF, concatenates them, and saves the result to disk This does too many disk IO operations and should probably use Python's buffer objects to save intermediate data that won't be served to the end user, i.e. the HTML file and generated PDF. Still need to consider if we want a CSV+PDF per-address or if there should be a bulk output format. Should check with Wilfred.

mattn commented

2022-10-19 10:43:35 +13:00

(Migrated from gitlab.catalyst.net.nz)

After speaking with Wilfred, it sounds like we can do this quite easily. It is also his number one priority. What he wants is to generate a CSV and PDF for each row of an input CSV containing:

An address or geometry (probably a point with a projection; EWKT would work)
Soil type
Habitat
Zone

which is essentially a filled out form from the frontend.

After speaking with Wilfred, it sounds like we can do this quite easily. It is also his number one priority. What he wants is to generate a CSV and PDF for each row of an input CSV containing: 1. An address or geometry (probably a point with a projection; EWKT would work) 2. Soil type 3. Habitat 4. Zone which is essentially a filled out form from the frontend.

mattn commented

2023-01-25 11:24:03 +13:00

(Migrated from gitlab.catalyst.net.nz)

Spoken to Wilfred and he now wants this to become a front end form where a user inputs

An address (assuming a replacement for step 1 of the questionnaire)

require LINZ ingest into Postgres
API endpoint for address search
address search and autocomplete algorithm (previously implemented in FWV)

Multiple zones/habitats, each appearing as a new query (CSV row/database record in a a new table) for the same address

On submit, this creates one or more records in a new table. This will require a new Django model (StoredQuery?) that will have options on the admin interface to generate a PDF per query, a functionality described in #29

Spoken to Wilfred and he now wants this to become a front end form where a user inputs 1. An address (assuming a replacement for step 1 of the questionnaire) * require LINZ ingest into Postgres * API endpoint for address search * address search and autocomplete algorithm (previously implemented in FWV) 2. *Multiple* zones/habitats, each appearing as a new query (CSV row/database record in a a new table) for the same address On submit, this creates one or more records in a new table. This will require a new Django model (`StoredQuery`?) that will have options on the admin interface to generate a PDF per query, a functionality described in #29

mattn commented

2023-02-02 15:03:43 +13:00

(Migrated from gitlab.catalyst.net.nz)

assigned to @mattn

mattn commented

2023-02-03 09:30:18 +13:00

(Migrated from gitlab.catalyst.net.nz)

mentioned in issue #44