Skip to content

Commit 334d8b6

Browse files
eywalkereywalker
authored
Merge pull request #681 from guzman-raphael/master
Add Loading Classes Docs, Fix errors/typos, Bump version, Update Release Notes, Fix tests
2 parents 198885d + f8d4b1a commit 334d8b6

15 files changed

+874
-10
lines changed

‎CHANGELOG.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
## Release notes
22

3-
### 0.12.0 -- October 1, 2019
3+
### 0.12.0 -- October 31, 2019
44
* Dropped support for Python 3.4
55
* Support secure connections with TLS (aka SSL) PR #620
66
* Convert numpy array from python object to appropriate data type if all elements are of the same type (#587) PR #608

‎datajoint/version.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
__version__ = "0.12.dev9"
1+
__version__ = "0.12.0"
22

33
assert len(__version__) <= 10 # The log table limits version to the 10 characters

‎docs-parts/admin/5-blob-config_lang1.rst

+2-2
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
dj.config['stores'] = {
44
'external': dict( # 'regular' external storage for this pipeline
55
protocol='s3',
6-
endpoint='https://s3.amazonaws.com',
6+
endpoint='s3.amazonaws.com:9000',
77
bucket = 'testbucket',
8-
location = '/datajoint-projects/myschema',
8+
location = 'datajoint-projects/lab1',
99
access_key='1234567',
1010
secret_key='foaf1234'),
1111
'external-raw'] = dict( # 'raw' storage for this pipeline

‎docs-parts/admin/5-blob-config_lang4.rst

+5-5
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11

2-
To remove only the tracking entries in the external table, call `delete`
3-
on the external table for the external configuration with the argument
4-
`delete_external_files=False`.
2+
To remove only the tracking entries in the external table, call ``delete``
3+
on the ``~external_<storename>`` table for the external configuration with the argument
4+
``delete_external_files=False``.
55

66
.. note::
77

@@ -23,7 +23,7 @@ on the external table for the external configuration with the argument
2323
2424
.. note::
2525

26-
Setting `delete_external_files=True` will always attempt to delete
26+
Setting ``delete_external_files=True`` will always attempt to delete
2727
the underlying data file, and so should not typically be used with
28-
the `filepath` datatype.
28+
the ``filepath`` datatype.
2929

‎docs-parts/existing/1-Virtual-Modules_lang1.rst renamed to ‎docs-parts/existing/0-Virtual-Modules_lang1.rst

File renamed without changes.

‎docs-parts/existing/1-Loading-Classes_lang1.rst

+239
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
2+
This section describes how to work with database schemas without access to the
3+
original code that generated the schema. These situations often arise when the
4+
database is created by another user who has not shared the generating code yet
5+
or when the database schema is created from a programming language other than
6+
Python.
7+
8+
.. code-block:: python
9+
10+
import datajoint as dj
11+
12+
13+
Working with schemas and their modules
14+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
15+
16+
Typically a DataJoint schema is created as a dedicated Python module. This
17+
module defines a schema object that is used to link classes declared in the
18+
module to tables in the database schema. As an example, examine the university
19+
module: `university.py <https://github.com/vathes/db-programming-with-datajoint/blob/master/notebooks/university.py>`_.
20+
21+
You may then import the module to interact with its tables:
22+
23+
.. code-block:: python
24+
25+
import university as uni
26+
27+
*Connecting dimitri\@localhost:3306*
28+
29+
.. code-block:: python
30+
31+
dj.Diagram(uni)
32+
33+
.. figure:: virtual-module-ERD.svg
34+
:align: center
35+
:alt: query object preview
36+
37+
.. .. raw:: html
38+
.. :file: virtual-module-ERD.svg
39+
40+
Note that dj.Diagram can extract the diagram from a schema object or from a
41+
Python module containing its schema object, lending further support to the
42+
convention of one-to-one correspondence between database schemas and Python
43+
modules in a DataJoint project:
44+
45+
``dj.Diagram(uni)``
46+
47+
is equvalent to
48+
49+
``dj.Diagram(uni.schema)``
50+
51+
.. code-block:: python
52+
53+
# students without majors
54+
uni.Student - uni.StudentMajor
55+
56+
.. figure:: StudentTable.png
57+
:align: center
58+
:alt: query object preview
59+
60+
.. .. csv-table::
61+
.. :file: Student_Table.csv
62+
.. :widths: 5, 5, 5, 5, 5, 5, 5, 5, 5, 5
63+
.. :header-rows: 1
64+
65+
Spawning missing classes
66+
~~~~~~~~~~~~~~~~~~~~~~~~
67+
Now imagine that you do not have access to ``university.py`` or you do not have
68+
its latest version. You can still connect to the database schema but you will
69+
not have classes declared to interact with it.
70+
71+
So let's start over in this scenario.
72+
73+
You can may use the ``dj.list_schemas`` function (new in DataJoint 0.12.0) to
74+
list the names of database schemas available to you.
75+
76+
.. code-block:: python
77+
78+
import datajoint as dj
79+
dj.list_schemas()
80+
81+
*Connecting dimitri@localhost:3306*
82+
83+
*['dimitri_alter','dimitri_attach','dimitri_blob','dimitri_blobs',
84+
'dimitri_nphoton','dimitri_schema','dimitri_university','dimitri_uuid',
85+
'university']*
86+
87+
Just as with a new schema, we start by creating a schema object to connect to
88+
the chosen database schema:
89+
90+
.. code-block:: python
91+
92+
schema = dj.schema('dimitri_university')
93+
94+
If the schema already exists, dj.schema is initialized as usual and you may plot
95+
the schema diagram. But instead of seeing class names, you will see the raw
96+
table names as they appear in the database.
97+
98+
.. code-block:: python
99+
100+
# let's plot its diagram
101+
dj.Diagram(schema)
102+
103+
.. figure:: dimitri-ERD.svg
104+
:align: center
105+
:alt: query object preview
106+
107+
.. .. raw:: html
108+
.. :file: dimitri-ERD.svg
109+
110+
You may view the diagram but, at this point, there is no way to interact with
111+
these tables. A similar situation arises when another developer has added new
112+
tables to the schema but has not yet shared the updated module code with you.
113+
Then the diagram will show a mixture of class names and database table names.
114+
115+
Now you may use the ``schema.spawn_missing_classes`` method to spawn classes into
116+
the local namespace for any tables missing their classes:
117+
118+
.. code-block:: python
119+
120+
schema.spawn_missing_classes()
121+
dj.Di(schema)
122+
123+
.. figure:: spawned-classes-ERD.svg
124+
:align: center
125+
:alt: query object preview
126+
127+
.. .. raw:: html
128+
.. :file: spawned-classes-ERD.svg
129+
130+
Now you may interact with these tables as if they were declared right here in
131+
this namespace:
132+
133+
.. code-block:: python
134+
135+
# students without majors
136+
Student - StudentMajor
137+
138+
.. figure:: StudentTable.png
139+
:align: center
140+
:alt: query object preview
141+
142+
Creating a virtual module
143+
~~~~~~~~~~~~~~~~~~~~~~~~~
144+
Now ``spawn_missing_classes`` creates the new classes in the local namespace.
145+
However, it is often more convenient to import a schema with its Python module,
146+
equivalent to the Python command
147+
148+
.. code-block:: python
149+
150+
import university as uni
151+
152+
We can mimick this import without having access to ``university.py`` using the
153+
``create_virtual_module`` function:
154+
155+
.. code-block:: python
156+
157+
import datajoint as dj
158+
159+
uni = dj.create_virtual_module('university.py', 'dimitri_university')
160+
161+
*Connecting dimitri@localhost:3306*
162+
163+
Now ``uni`` behaves as an imported module complete with the schema object and all
164+
the table classes.
165+
166+
.. code-block:: python
167+
168+
dj.Di(uni)
169+
170+
.. figure:: added-example-ERD.svg
171+
:align: center
172+
:alt: query object preview
173+
174+
.. .. raw:: html
175+
.. :file: added-example-ERD.svg
176+
177+
.. code-block:: python
178+
179+
uni.Student - uni.StudentMajor
180+
181+
.. figure:: StudentTable.png
182+
:align: center
183+
:alt: query object preview
184+
185+
``dj.create_virtual_module`` takes optional arguments.
186+
187+
First, ``create_schema=False`` assures that an error is raised when the schema
188+
does not already exist. Set it to ``True`` if you want to create an empty schema.
189+
190+
.. code-block:: python
191+
192+
dj.create_virtual_module('what', 'nonexistent')
193+
194+
.. code-block:: python
195+
196+
---------------------------------------------------------------------------
197+
DataJointError Traceback (most recent call last)
198+
.
199+
.
200+
.
201+
DataJointError: Database named `nonexistent` was not defined. Set argument create_schema=True to create it.
202+
203+
204+
The other optional argument, ``create_tables=False`` is passed to the schema
205+
object. It prevents the use of the schema obect of the virtual module for
206+
creating new tables in the existing schema. This is a precautionary measure
207+
since virtual modules are often used for completed schemas. You may set this
208+
argument to ``True`` if you wish to add new tables to the existing schema. A
209+
more common approach in this scenario would be to create a new schema object and
210+
to use the ``spawn_missing_classes`` function to make the classes available.
211+
212+
However, you if do decide to create new tables in an existing tables using the
213+
virtual module, you may do so by using the schema object from the module as the
214+
decorator for declaring new tables:
215+
216+
.. code-block:: python
217+
218+
uni = dj.create_virtual_module('university.py', 'dimitri_university', create_tables=True)
219+
220+
.. code-block:: python
221+
222+
@uni.schema
223+
class Example(dj.Manual):
224+
definition = """
225+
-> uni.Student
226+
---
227+
example : varchar(255)
228+
"""
229+
230+
.. code-block:: python
231+
232+
dj.Di(uni)
233+
234+
.. figure:: added-example-ERD.svg
235+
:align: center
236+
:alt: query object preview
237+
238+
.. .. raw:: html
239+
.. :file: added-example-ERD.svg

‎docs-parts/existing/StudentTable.png

46.9 KB
Loading

0 commit comments

Comments
 (0)