ffdn-db/TODO.org

TODO Test current V0.2 implementation [25%]

DONE Find a way to validate generated JSON according to the JSON scheme.

Looks like everything is done in isbformat.validator.validate_isp, no automatic validation process for the form.

I think we don't really care about that as long as we test extensively this in some automated tests.

TODO Test the remote fetching scrapper for the 0.1 format [60%]

There's an automatic validation for the crawler though.
• [X] Find how everything has been tested so far.
Everything is basically stored in a single file: =test_ffdnispdb.py=.

2 classes in this file.
• TestCase, the first one, seems to be a kind of test case configuration.
All the db setup/teardown lies there.
• TestForm, is basically a test case testing the ISP creation form.
• TestAPI, another test case tesing the API.
• A mysterious load_tests. Looks like a doc test suite. I don't really get how this is
loaded/executed. A quick silver searcher run does not find any occurence for this function.

It's probably a weird python trick used by unittest to make some kind of ocult ceremony.

After a quick search, it's indeed a doctest, see https://docs.python.org/2/library/doctest.html

[X] How can we run the tests?

python manage.py test
[X] Is the 0.1 format properly tested?

yup
• [-] Does this particular test fails.
yup
• [-] There's a problem with the covered-areas table. Did I break something? We need to investigate.
#+BEGIN_CODE

Traceback (most recent call last): File "/ffdn/test_ffdnispdb.py", line 126, in setUp super(TestAPI, self).setUp() File "/ffdn/test_ffdnispdb.py", line 89, in setUp db.create_all() File "/usr/local/lib/python2.7/dist-packages/flask_sqlalchemy/__init__.py", line 856, in create_all self._execute_for_all_tables(app, bind, 'create_all') File "/usr/local/lib/python2.7/dist-packages/flask_sqlalchemy/__init__.py", line 848, in _execute_for_all_tables op(bind=self.get_engine(app, bind), tables=tables) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/schema.py", line 2848, in create_all tables=tables) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1479, in _run_visitor conn._run_visitor(visitorcallable, element, **kwargs) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1122, in _run_visitor **kwargs).traverse_single(element) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 122, in traverse_single return meth(obj, **kw) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/ddl.py", line 70, in visit_metadata self.traverse_single(table, create_ok=True) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/visitors.py", line 122, in traverse_single return meth(obj, **kw) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/ddl.py", line 97, in visit_table _ddl_runner=self) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/event.py", line 409, in __call__ fn(*args, **kw) File "/usr/local/lib/python2.7/dist-packages/geoalchemy/geometry.py", line 131, in after_create self('after-create', target, connection) File "/usr/local/lib/python2.7/dist-packages/geoalchemy/geometry.py", line 119, in __call__ spatial_dialect.handle_ddl_after_create(bind, table, c) File "/usr/local/lib/python2.7/dist-packages/geoalchemy/spatialite.py", line 112, in handle_ddl_after_create bind.execute("VACUUM %s" % table.name) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 805, in _execute_text statement, parameters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context context) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception exc_info File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 867, in _execute_context context) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 324, in do_execute cursor.execute(statement, parameters) OperationalError: (OperationalError) unknown database covered_areas 'VACUUM covered_areas' () #+END_CODE
• [-] Try to add a geodb contour using the form and see whether this wails.
• [ ] Understand how these contours are created? DOING: asked Sacha
• [X] Retrieve a contour from the live FDN DB.
#+BEGIN_CODE {"coveredAreas": [{"area": {"coordinates": [[[[3.24056113717041, 43.21280863308406], [3.228989263448332, 43.21915735757924], [3.201526184107128, 43.24776294578855], [3.193228839121154, 43.25040474369601], [3.18402303757751, 43.25012397227937], [3.173125708392068, 43.24383627394244], [3.159182900749729, 43.24322805223474], [3.146507526450182, 43.24738382359277], [3.24056113717041, 43.21280863308406]]]], "type": "MultiPolygon"} #+END_CODE
• [X] Try to insert that in the DB
• [ ] Check whether this geoJSON has also been inserted in its own table.
• It has its own table called CoveredAreas
• [ ] How did stuff end up in here?
This data is duplicated in the DB. It belongs both in the JSON model of IPS and its own CoveredAreas.

TODO Test the remote fetching scrapper for the 0.2 format [0%]

TODO Test the form for the 0.2 format

TODO Documentation [0%]

Update application documentation.

Everything seems to be synchronized though a =pre_save= hook on the ISP model. =_sync_covered_areas= is in charge of synchronising both the covered areas and the Office Location.
• [ ] How is this database created?
Is it the same sqlite database as the main one?
• [ ] Fix the test, if needed.
• [ ] Write a test using an example coming from the fdn json format repo.
• [ ] Make this test succeed.
We don't really care about the 0.1 format: we probably don't want to generate anything new using that format. Apparently, the documentation is coming from the ispformat package, =ispformat.specs=.

TODO Squash and write proper commit logs.

Seems like getting this up to date is a bit more tricky than just increasing the format version release.