Skip to content

Commit 8f789d1

Browse files
wkliaowkliao
authored
Merge pull request #17 from Parallel-NetCDF/developer_notes
add developer notes
2 parents 76bac61 + ac35bbb commit 8f789d1

File tree

1 file changed

+358
-0
lines changed

1 file changed

+358
-0
lines changed

‎docs/DEVELOPER_NOTES.md

Lines changed: 358 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,358 @@
1+
## Notes for PnetCDF-Python developers
2+
3+
This file contains instructions for PnetCDF-Python developers.
4+
Users of PnetCDF-Python please refer to the
5+
[PnetCDF-Python User Guide](https://pnetcdf-python.readthedocs.io/en/latest/).
6+
7+
* [Making A New Release](#making-a-new-release)
8+
* [Library Packaging And Publishing](#library-packaging-and-publishing)
9+
* [Library installation](#library-installation)
10+
* [User Guide](user-guide)
11+
12+
---
13+
### Making A New Release
14+
15+
Below is a list of tasks to be done immediately before making a new release
16+
(must run in the following order).
17+
18+
1. Update the release version string
19+
* Edit file [src/pnetcdf/__init__.py](src/pnetcdf/__init__.py) and update
20+
the following string to the new version number.
21+
```
22+
__version__ = "1.0.0"
23+
```
24+
<ul>
25+
<li> <details>
26+
<summary>GNU guideline for updating version numbers(click to expand)</summary>
27+
<li> For libtool Application Binary Versioning (ABI) versioning rules see:
28+
http://www.gnu.org/software/libtool/manual/libtool.html#Updating-version-info
29+
</li>
30+
<li> Update the version information only immediately before a public release.
31+
</li>
32+
<li> Quote from the GNU guide: Here are a set of rules to help you update
33+
your library version information:
34+
<ul>
35+
<li> Start with version information of '0:0:0' for each libtool library.
36+
</li>
37+
<li> Update the version information only immediately before a public
38+
release of your software. More frequent updates are unnecessary, and
39+
only guarantee that the current interface number gets larger faster.
40+
</li>
41+
<li> If the library source code has changed at all since the last update,
42+
then increment revision ('c:r:a' becomes 'c:r+1:a').
43+
</li>
44+
<li> If any interfaces have been added, removed, or changed since the last
45+
update, increment current, and set revision to 0.
46+
</li>
47+
<li> If any interfaces have been added since the last public release, then
48+
increment age.
49+
</li>
50+
<li> If any interfaces have been removed or changed since the last public
51+
release, then set age to 0.
52+
</li>
53+
</li>
54+
<ul>
55+
</details></li>
56+
</ul>
57+
58+
59+
2. Update release note
60+
* Edit file [./RELEASE_NOTES.md](./RELEASE_NOTES.md) to include notes about
61+
new features and other noticeable changes.
62+
* Edit and update the release date.
63+
64+
3. Commit all changes to github repo
65+
* Run command below to upload changes to github.com.
66+
```
67+
git push upstream main
68+
```
69+
70+
4. Create a tag and push it to the github repo
71+
* Run commands below to do so. Replace the proper version string.
72+
```
73+
git tag -a pnetcdf-python.1.0.0 -m "PnetCDF-Python package version 1.0.0"
74+
git push upstream pnetcdf-python.1.0.0
75+
```
76+
77+
5. Create a new release from the tag on github.com.
78+
* Visit PnetCDF-Python github repo's
79+
[tags](https://github.com/Parallel-NetCDF/PnetCDF-Python/tags)
80+
* Click the name of tag just pushed.
81+
* Click "Create release from tag".
82+
* Fill in "Release title"
83+
* Fill in "Description field"
84+
* Optionally, click "Set as a pre-release", if this is a pre-release.
85+
* Optionally, click "Generate release notes", which will add a list of all
86+
PRs and new contributors into the Description field..
87+
* Click "Save draft" to save the changes. It can be modified later.
88+
* See all options from [github docs](https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes)
89+
90+
6. If a new commit were added or pushed to the main branch after a draft
91+
release was created, make sure the local main branch is up-to-date and then
92+
run commands below to force-update the tag on both local and remote.
93+
```
94+
git tag -fa pnetcdf-python.1.0.0
95+
git push -f --tags
96+
```
97+
98+
---
99+
### Library Packaging And Publishing
100+
* Currently, pip-install via build distribution is disabled. No wheel files
101+
are uploaded to PyPI. Lastest PnetCDF-Python package on PyPI can be found
102+
in: https://pypi.org/project/pnetcdf/
103+
* Packaging: build source distribution and wheel distribution on a local
104+
machine by following these steps.
105+
1. Install PnetCDF-C library
106+
2. Make sure a python virtual env is created and all python dependencies are
107+
installed as the developer installation.
108+
```
109+
cd $HOME
110+
python -m venv tmp_env
111+
source $HOME/tmp_env/bin/activate
112+
pip install --upgrade pip setuptools wheel packaging
113+
pip install numpy Cython
114+
CC=/path/to/mpicc pip install mpi4py
115+
```
116+
3. Download the release tarball of PnetCDF-Python. For example, use commands
117+
as shown below.
118+
```
119+
wget https://github.com/Parallel-NetCDF/PnetCDF-Python/archive/refs/tags/pnetcdf-python.1.0.0.tar.gz
120+
tar -xfv pnetcdf-python.1.0.0.tar.gz
121+
cd PnetCDF-Python
122+
python3 -m pip install --upgrade build twine
123+
```
124+
4. Build the package and generate distribution.
125+
```
126+
CC=/path/to/mpicc PNETCDF_DIR=/path/to/pnetcdf/dir python3 -m build
127+
```
128+
A successful run of this command will generate 2 new files:
129+
+ `dist/pnetcdf-x.x.x.tar.gz` and
130+
+ `dist/pnetcdf-x.x.x-cp39-cp39-linux_x86_64.whl`
131+
5. Leave the python environment, tmp_env, by running command
132+
```
133+
deactivate
134+
```
135+
136+
* Test distribution
137+
1. For testing purpose, we must publish the package built above on
138+
[TestPyPI](https://packaging.python.org/en/latest/guides/using-testpypi),
139+
by only uploading source distribution archive, since the wheel file
140+
(dist/pnetcdf-x.x.x*.whl) works exclusively for your own system and python
141+
version and is not useful for the users.
142+
2. Create TestPyPI account.
143+
* Optionally, update file `$HOME/.pypirc` on local machine to skip
144+
the log-in credentials
145+
3. Publish source distribution on TestPyPI:
146+
```
147+
python3 -m twine upload --repository testpypi dist/pnetcdf-x.x.x.tar.gz
148+
```
149+
The command will first check if the same name and version number have
150+
already existed. The command will error out if the same names have been
151+
found. After uploading, open your browser, go to https://test.pypi.org
152+
and search for package name: `pnetcdf` to verify.
153+
4. Next, test the uploaded distribution on a local machine
154+
* Create a new empty folder. E.g.
155+
```
156+
mkdir pypi_test
157+
cd pypi_test
158+
```
159+
* Make sure PnetCDF-C and mpich are installed.
160+
* Create and activate a new vanilla python env for testing.
161+
* This env shouldn't be the same env used in developer install, i.e. the
162+
environment does not contain any pre-installed PnetCDF-Python library
163+
```
164+
python -m venv test_env
165+
source test_env/bin/activate
166+
```
167+
* Do a quick install via the distribution on TestPyPI (no python
168+
dependencies required). Note that `-i` redirects pip-install to search
169+
PnetCDF-Python in testpypi index and `--extra-index-url` redirects
170+
pip-install to search dependency libraries (e.g. numpy) in official
171+
pypi index.
172+
```
173+
CC=/path/to/mpicc PNETCDF_DIR=/path/to/pnetcdf/dir \
174+
pip install -i https://test.pypi.org/simple/ \
175+
--extra-index-url https://pypi.org/simple pnetcdf==x.x.x
176+
```
177+
This command will download the package from testpypi and install
178+
PnetCDF-Python library in the current env.
179+
5. Run PnetCDF-Python test programs. For example, using test programs
180+
available in folder `test` of PnetCD-Python package.
181+
```
182+
wget https://github.com/Parallel-NetCDF/PnetCDF-Python/archive/refs/tags/pnetcdf-python.1.0.0.tar.gz
183+
tar -xfv pnetcdf.1.0.0.tar.gz
184+
cd PnetCDF-Python
185+
cd test
186+
make check
187+
make ptests
188+
```
189+
Test programs available in folder `examples`, which requires pytorch.
190+
```
191+
cd ../examples
192+
pip install torch torchvision
193+
make check
194+
make ptests
195+
```
196+
Leave the python environment, test_env, by running command
197+
```
198+
deactivate
199+
```
200+
201+
* Publish official release on PyPI
202+
1. Official releases are published on [PyPI](https://pypi.org/)
203+
2. WARNING: after upload to pypi, one cannot overwrite or change the
204+
published package on pypi!
205+
3. Create PyPI account and update `.pypirc` per instructions above.
206+
4. Publish source distribution on PyPI
207+
```
208+
python3 -m twine upload dist/pnetcdf-x.x.x.tar.gz
209+
```
210+
5. For testing, just create a new virtual env and quick install (using
211+
default PyPI index) without source code repo.
212+
```
213+
CC=/path/to/mpicc PNETCDF_DIR=/path/to/pnetcdf/dir pip install pnetcdf
214+
```
215+
6. To verify the official published verion, open the browser and go to
216+
https://pypi.org/project/pnetcdf/ to check the latest release.
217+
218+
219+
---
220+
### Library installation
221+
* Quick install
222+
* `MANIFEST.in` controls files to be included in source distribution
223+
(sdist), which will be eventually uploaded to PyPI if we enables quick
224+
install in the future. After modifications to `MANIFEST.IN` file, here are
225+
steps to check if the files included are valid to build the library.
226+
1. Make sure pnetcdf.egg-info folder is deleted. Otherwise it will first
227+
cache previous versions of `MANIFEST.IN` requirement. This can be done
228+
by running command below at the root folder.
229+
```
230+
make build-clean
231+
```
232+
2. Build the source distribution
233+
```
234+
CC=/path/to/mpicc PNETCDF_DIR=/path/to/pnetcdf/dir \
235+
python setup.py sdist
236+
```
237+
3. Check source distribution content if the list matches `MANIFEST.IN`
238+
```
239+
tar -tzf dist/package-<version>.tar.gz
240+
```
241+
4. Install the library from sdist
242+
```
243+
CC=/path/to/mpicc PNETCDF_DIR=/path/to/pnetcdf/dir \
244+
pip install dist/package-<version>.tar.gz
245+
```
246+
5. Check installation by testing programs in folders test and examples
247+
248+
* Additional notes about quick install
249+
1. Source distribution(.tar.gz)
250+
* Pip-install command’s second go-to option if no matching build
251+
distribution is found on PyPI. It will build its own wheel file for
252+
the current system. For python libs with C-extension (like
253+
PnetCDF-Python netcdf4-python, h5py), wheel installation require
254+
their C bindings.
255+
* Invariant to platforms/versions, usually each version of python
256+
library has one single source distribution.
257+
2. Build distribution (.whl)
258+
* Pip-install command’s first go-to option by default. For python libs
259+
with C-extension (like PnetCDF-Python netcdf4-python, h5py), wheel
260+
installation does not require their C bindings.
261+
* Wheels are platform-specific and python-version specific. Different
262+
system used for building and uploading will generate different
263+
versions of wheel files on PyPI. To cover most mainstream python
264+
versions and operating systems, python libraries (like
265+
[numpy 1.25](https://pypi.org/project/numpy/#files)) has 20+ wheels
266+
files to cover most mainstream systems (e.g. Linux (x86_64), MacOS
267+
(x86_64), and Windows (32-bit and 64-bit)) and recent python versions.
268+
* General procedures of building and uploading python library wheels
269+
(build distributions) for MacOS and Linux systems
270+
+ Python libs with C-extension (like PnetCDF-Python, netcdf4-python,
271+
h5py) requires shared object (.so in Linux and .dylibs in mac)
272+
files collected from C software installation. When making python
273+
library wheels (build distribution), an extra post-processing step
274+
(Delocate tool for Mac, auditwheel Tool in linux) is usually
275+
performed to copy and store these files in the python package to
276+
remove these dependencies. That’s why the user’s pip install with
277+
build distribution does not require a C installation as a
278+
prerequisite.
279+
+ For MacOS wheels (such as
280+
netCDF4-1.6.4-cp311-cp311-macosx_10_9_x86_64.whl) do the
281+
followings.
282+
1. Build the package to create the wheel,
283+
2. Use Delocate tool to fix wheels
284+
3. Upload to PyPI.
285+
+ For Linux wheels, do the followings.
286+
1. Pull manylinux docker image
287+
2. Build the package to create the wheel in this container
288+
3. Use auditwheel Tool to fix wheels
289+
4. Upload to PyPI.
290+
+ For some python libraries (numpy, netCDF4), a
291+
[dedicated github repo](https://github.com/MacPython/netcdf4-python-wheels)
292+
is used to automate building wheels for every release.
293+
294+
295+
* Developer install on a local machine
296+
* Developer installation is mainly managed by files `setup.py` and
297+
`pyproject.toml`. The former is the core file to build the package
298+
and the latter manages dependencies required by `pip install` command.
299+
* Command below builds and installs the python library based on the current
300+
python environment.
301+
```
302+
python setup.py install
303+
```
304+
The command will error out if any dependency (e.g. mpi4py) is missing.
305+
This command will soon be deprecated and is not recommended for modern
306+
python library installation.
307+
* Command below works as a wrapper command for the above but does further
308+
to automatically handle and install any dependencies listed in file
309+
`pyproject.toml`. Need to pay special attention to the dependencies
310+
listed in the filed of `requires` under `[build-system]` and
311+
`dependencies`.
312+
```
313+
pip install .
314+
```
315+
+ Field `dependencies` in file `pyproject.toml` defines python libraries
316+
required for running the project and will first check if requirement
317+
already satisfied in current environment before installing the latest
318+
qualified version of the library.
319+
+ Field `requires` in file `pyproject.toml` defines libraries required
320+
for building the project. Command `pip install` by default creates and
321+
uses isolated building env for building stage which completely ignores
322+
current user env. For example, if user already installed mpi4py==3.1.6,
323+
setting this field to "mpi4py>=3.1.4" will automatically install a
324+
mpi4py 4.0.0 in the building env and thereafter use syntax from 4.0.0
325+
to build PnetCDF-Python This causes version mismatch issues between
326+
building and running envs when numpy 2.0 and mpi4py 4.0.0 are released.
327+
To address this issue, use command below.
328+
```
329+
pip install --no-build-isolation -e .
330+
```
331+
332+
---
333+
### User Guide
334+
335+
Note on configuring Read-the-Docs tool for generating PnetCDF-Python user guide.
336+
* Read the Docs settings
337+
+ User guide is automatically generated when new commits are pushed to the
338+
main branch and a new PR is created. Modify this in Read the Docs dashboard
339+
if needed.
340+
341+
* Doc files
342+
+ `.readthedocs.yaml` -- is a script file controls commands that run before
343+
installation of PnetCDF-Python (under `pre-install:`) and installation
344+
(under `python:`)
345+
+ `/docs/requirements.txt` -- lists all Python dependencies required for doc
346+
generation, including `sphinx`
347+
348+
* Important environment variables
349+
+ User guide generation requires environment variables set by Read the Docs
350+
dashboard (`Admin` -> `Environment Variables`). Delete and add a new
351+
variable to modify (remember to select `Expose this environment variable in
352+
PR builds` if PR auto-build is enabled)
353+
+ Current environment variables set (only effective solution found to set env
354+
variable at installation):
355+
* `CC`: mpicc.mpich
356+
* `PNETCDF_DIR`: /home/docs/checkouts/readthedocs.org/user_builds/pnetcdf-python/checkouts/latest/_readthedocs/PnetCDF
357+
* `PNETCDF_VER`: 1.14.0
358+

0 commit comments

Comments
 (0)