Syntool installation - syntool-meta2sql problem

Syntool installation - syntool-meta2sql problem - Printable Version

+- ODL Forums (https://forum.oceandatalab.com)
+-- Forum: Syntool (https://forum.oceandatalab.com/forum-5.html)
+--- Forum: Questions (https://forum.oceandatalab.com/forum-10.html)
+--- Thread: Syntool installation - syntool-meta2sql problem (/thread-48.html)

Syntool installation - syntool-meta2sql problem - petwa - 2021-04-28

Hi,
Not sure if this is the correct place to ask installation questions. Please redirect me if it is not.

I have downloaded (from https://ftp.odl.bzh/odl/syntool/) and built the syntool code with no obvious errors. syntool-converter and syntool-ingestor run OK and produce a json file. When I run syntool-meta2sql on the json I get the error:
NotImplementedError: Don't know how to literal-quote value u'POLYGON((0.356966801331357.......

It looks like the version of SQLAlchemy I am using does not know how to deal with WKT. Have I missed something in the config?

Full output from command attached.

Thanks,
Pete

RE: Syntool installation - syntool-meta2sql problem - Sylvain Herlédan - 2021-04-28

Hi Pete,

syntool-metadata should be able to handle any metadata.json file produced by syntool-ingestor, so I share your suspicion about the SQLAlchemy version.

Could you please tell us which version of SQLAlchemy is installed in your environment? You can get this information using the following command:

Code:
pip show SQLAlchemy

If possible, could you also include the metadata.json file in the attachments so we can check its content and try to reproduce the problem?

Cheers,

Sylvain

RE: Syntool installation - syntool-meta2sql problem - petwa - 2021-04-29

Hi Sylvain,

Thanks for the quick response. I have attached the json file.

The SQLAlchemy version is below:

root@a0d0024659e5:~# pip show SQLAlchemy
Name: SQLAlchemy
Version: 1.4.11
Summary: Database Abstraction Library
Home-page: http://www.sqlalchemy.org
Author: Mike Bayer
Author-email: mike_mp@zzzcomputing.com
License: MIT
Location: /usr/local/lib/python2.7/dist-packages
Requires: importlib-metadata

Cheers,
Pete

RE: Syntool installation - syntool-meta2sql problem - Sylvain Herlédan - 2021-04-29

Hi Pete,

The problem is indeed the version of SQLAlchemy: the Syntool packages found in https://ftp.odl.bzh/odl/syntool have been published around 2018 if I remember correctly, so they worked with the latest SQLAlchemy version available at that time but incompatible changes have been introduced with later versions.

A simple workaround would be to downgrade SQLAlchemy to a version that doesn't contain these changes:

Code:
pip uninstall -y sqlalchemy

pip install "sqlalchemy<1.3.0"

Alternatively, you can install the syntool-exporter package available in the attachments of this message, it is an unreleased version but it should be compatible with the 1.4.11 version of SQLAlchemy.

Please let me know if one of these methods solved the problem.

Cheers,

Sylvain

RE: Syntool installation - syntool-meta2sql problem - petwa - 2021-04-29

Hi Sylvain,

SQLAlchemy 1.3.0 worked, thank you. I will continue testing with that as it is easy to downgrade (all in a Docker container). Would you recommend using the newer exporter package?
We were hoping to use Syntool as a quick way of publishing/viewing a number of project related datasets so I am trying to get a simple workflow going for the process.

Thanks again for your help.

Cheers,
Pete

RE: Syntool installation - syntool-meta2sql problem - Sylvain Herlédan - 2021-04-29

Glad to hear that it worked and that Syntool will be useful to you Smile

Your approach sounds quite good: once you have the processing tools installed and working in a container, you can scale by spawning or destroying containers depending on you needs. All you have to keep online permanently are the elements required to run the Web portal (database, webservice, web server and the files produced by syntool-ingestor).

Compatibility with newer versions of SQLAlchemy is really the only advantage of the newer version of syntool-exporter, so if your environment is already set up and everything works I think it's okay to continue with the official release.

Cheers,

Sylvain

RE: Syntool installation - syntool-meta2sql problem - petwa - 2021-04-29

Is the developer manual the best guide to setting up data ingestion or should I look elsewhere?

RE: Syntool installation - syntool-meta2sql problem - Sylvain Herlédan - 2021-04-29

There is no other documentation at the moment, so the developer manual is the most complete source of information regarding the data ingestion process. Don't hesitate to ask your questions here if there is something missing in the manual.

RE: Syntool installation - syntool-meta2sql problem - petwa - 2021-04-29

(2021-04-29T10:04 AM)Sylvain Herlédan Wrote: There is no other documentation at the moment, so the developer manuel is the most complete source of information regarding the data ingestion process. Don't hesitate to ask your questions here if there is something missing in the manual.

Great. I am sure I will Smile

Thanks
Pete

RE: Syntool installation - syntool-meta2sql problem - jiayoupack - 2022-07-21

I found this post very interesting and informative. Thank you for sharing your special thoughts with us. I definitely share this with my peeps.