History¶
This section summarizes all my steps on the way to a working micc, including dead-ends.
https://sdss-python-template.readthedocs.io/en/latest/
Using poetry on the cluster¶
Installed poetry with pip install poetry
. poetry install` for ``ET-dot
seems to update e.g.. Numpy (it is also found in .venv
’s site-packages
), but
import numpy
nevertheless uses the preinstalled python, which is almost what
we want, except for Numpy to appear in the site-packages
, which wastes disk
space.
I must still be verified how we can install poetry just once for several python versions. Suspectedly, the system python (2.7.5 on Leibniz) is not sufficient. I jugded this wrong. I installed poetry on Leibniz, using the recommended method:
> curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
with the system python, and i was able to run poetry install
on a project.
v0.10.25¶
It’s been a while that this file has been updated …
v0.9.6 (2019-11-21)¶
Keeping version number between micc and micc-build consistent is error-prone. Each has to know the other’s version number because dependencies must be set. Et-micc-build depends on et-micc. So et-micc-build/pyproject.toml must know the version number of et-micc. On the other hand, when et-micc adds a binary extension to a project it must add the dependency on et-micc-build to that project’s pyproject.toml file, and thus et-micc must know the version number of et-micc-build. bumpversion has a lot more in petto than we used so far.
if they have the same version number it becomes much simpler.
v0.9.5 (2019-11-19)¶
- OOP version of micc, code much better organised now.
- fixed Github issue #19
v0.8.0 (2019-11-11)¶
MICC exists on PyPI * project renamed to et-micc
v0.7.5 (2019-11-11)¶
- Fixed wrong location of
tests
directory inmodule-py
template - Fixed Github issue #17 micc is no longer compatible with itself
- Fixed Github issue #12
v0.7.4 (2019-10-23)¶
- Github issue #3
- created branch issue_3
- implemented a workaround in
micc dev-install
.
v0.7.3 (2019-10-22)¶
- micc –save/load: add .json extension if omitted, and test for filepath.
- tutorial
- micc build –clean: removes also binary extension, not only build directory
v0.7.2 (2019-10-22)¶
- issue #6 micc build –save/load <build options file>.json
- issue #5 micc build –clean option
- fixed setting compile options in cpp modules.
v0.7.1 (2019-10-21)¶
Issue2 providing f2py command line arguments through CMake seems like a lot of details
while in fact there is no reason to use CMake to call f2py in the first place. Instead
we try to use the numpy.f2py
module. works well.
Which there was a python interface to cmake…
v0.7.0 (2019-10-18)¶
we are writing our issues on github now, and we are working inside a seprate git branch for every issue. merging implies a version bump. * more intuitive micc subcommands
v0.6.8 (2019-10-17)¶
- fixed bug in
micc app
documentation generation - wip on tutorial
- bug fix in
micc app
- if github_username is empty do not add remote origin
- add
--license
option tomicc create
- simplified test template.
- bug fix
--clear-log
v0.6.6 (2019-10-10)¶
- refactored logging tools into
logging.py
- fixed indentation with loglevel
- wip tutorial-1
micc create
now creates a module structure by default. In this way the interface ofmicc create
andmicc module
is the same.- factored out template helpers in
expand.py
- some code cleanup
- documentation
- log warnings if executing commands fails (rather than debug messages) so the user sees them always
v0.6.5 (2019-10-08)¶
- indentation in logs
- log time of a command
- bug fix in template expansion when adding a second application
v0.6.4 (2019-10-05)¶
- improve documentation templates
- replaced
--structure
option with--module
flag and--package
flag
v0.6.3 (2019-10-04)¶
- started tutorial writing. They will help the user to learn how to use micc, and help me in improving the functionality of micc.
Using micc myself encouraged me to
command line interfaces are best non-interactive., i.e. they better not ask extra information. If information is missing, they better quit with a message. So,
- don’t ask for project name,
- don’t ask for short description
This approach allows the cli to be used in scripts, and it eases the testing.
Note that this principle should make us revisit the
micc.json
template parameter approach. see also issue #12, which was closed, but not entirely satisfactory.modify
micc create
as to create a project in an empty directory.Fixed: return codes from commands are now propagated as exit code of the application.
Fixed: issue #12.
Fixed: issue #23.
v0.6.2 (2019-10-02)¶
- Sofar, we added the dependencies manually, by editing pyproject.toml. We should
normally add dependencies through poetry. That seems to work well, though
you need to be connected to the internet for this. There was one issue:
sphinx-click
addspytest="^4.4.2"
as a dependency. This prevents the use ofpytest ^5.2.0
. Conclusion: do not add pytest as a dependency. In additionsphinx-click
depends onsphinx
, but seems to be request withsphinx-2.1.2
rather thansphinx-2.2.0
. Of course it slips in withsphinx-click
- While trying to run the micc tests with
poetry 1.0.0b1
installed, the tests fail because this poetry version depends oncleo-7.5.0
rather thancleo-6.8.0
. Conclusion: we do not want our code to depend on poetry, or any of its dependencies except for its command line interface because it is too volatile. - fixed issue #15
- fixed issue #16
- fixed issue #17
- fixed issue #18
- fixed issue #19
- fixed issue #20
- fixed issue #21
micc now no longer needs poetry and can run in a conda environment.
- poetry install is problematic and should not be used in a conda environment.
- poetry build seems to work well and the Makefile commands for installing and reinstalling locally work well too.
v0.6.1 (2019-09-27)¶
modified dependencies of micc. Created empty conda environment:
> make reinstall > pytest tests
All tests succeed.
v0.6.0 (2019-09-27)¶
- looks as most things are working smoothly now…, so we move to 0.6.0
- code review by codacy
v0.5.17 (2019-09-26)¶
modified assertions
v0.5.16 (2019-09-25)¶
- adding a
micc info
command that lists information on the project - improved the documentation.
v0.5.15 (2019-09-25)¶
It would be nice to have an automatic way of generating documentation from our click
apps. Howver, the output of micc --help
is not in .rst format…
Here is an interesting stackoverflow issue
for documenting click commands with sphinx
(see the answer on sphinx-click
works like a charm
v0.5.14 (2019-09-25)¶
- Improved logging, added function
micc.utils.get_micc_logger()
to get the logger from everywhere, as well as a functionmicc.utils.create_logger()
to create a new logger. - Added a function for executing and logging a list of commands.
- Added a micc command for documentation generation
- improved testing
- improved output of micc version
- refactoring: the project_path is now a global_option accepted by all subcommands
this changes the commands:
> micc create foo
becomes
> micc -p path/to/foo create
v0.5.13 (2019-09-23)¶
improving tests. I am still rather uncomfortable with testing cli’s. Most of the code is tested by manual inspection. This unlucky patch number is for improving this situation.
- step 1: study the click documentation on testing. Interesting topic: File system isolation, runs a scenario in an empty directory, e.g. /private/var/folders/rt/7h5lk6c955db20y1rzf1rjz00000gn/T/tmpa12gc_p9 Good idea but the location of that directory is a bit hard to trace. (Note, that a temporary directory inside micc doesn’t work, because micc refuses to create a project inside another project. a flag –allow-nesting is added (feature
refactoring: still not happy with the --simple
argument for micc
how does python refer to its project structure?
module.py
vsmodule/__init__.py
module.py
is called a modulemodule/__init__.py
is called a package
Thus, our modules may be modules or sub-level packages, and our package is actually the top-level package, which may contain other lower-level packages, as well as modules.
fixed issue #13 [feature] os.path -> pathlib fixed issue #12 common items in micc.json files fixed issue #14 [feature] –allow-nesting flag for creating projects implemented #6 [feature] decomposition fixed issue #10 micc files are part of the template
v0.5.12 (2019-09-19)¶
Add logging to micc commands as we tried out in micc build
v0.5.11 (2019-09-18)¶
#2 [feature] simple python project
add --simple
flag to micc create
to create a simple (=unnested) python module <package_name>.py
instead of the nested <package_name/>__init__.py
After a bit of thinking the most practical approach would be to make a copy of the cookiecutter
template micc-package, rename directory {{cookiecutter.package_name}}
to src
and rename
file {{cookiecutter.package_name}}/__init__.py
to src/{{cookiecutter.module_name.py}}
.
The big advantage of that is that the directory structure is almost completely the same. Other
approaches would need to relocate the docs
folder, the tests
folder and a bunch of other
files. This will most certainly limit the number of changes that is necessary.
However, this breaks the test``tests/test_<module_name>.py`` as the module is in src
and it
cannot be imported without adding src
to sys.path
. To cure the problem we can add a test
to see if the project is simple and add src
to sys.path
if it is. The less code, the better,
i think, thus i replace src/<module_name>.py
with <module_name>.py
and remove the src
directory.
This works fine. The next consideration that comes to my mind is that as a simple python package
grows, it might be useful to be able to convert it to a general python package. To do that,
the missing .rst
files must be added, with their references to cookiecutter variables
correctly replaced. Currently, however, the cookiecutter variables aren’t stored in the project.
Moreover, cookiecutter doesn’t support filters to use part of a template. so we must copy that
part into a seperate cookiecutter template. Hence, this is the plan:
- we remove the
docs
directory from themicc-package
template and from themicc-package-simple
template. - we create a
micc-package-simple-docs
template with the.rst
files frommicc-package-simple
- and a
micc-package-general-docs
template with the.rst
files that must be added or overwritten frommicc-package
Moreover, we split of the common parts of micc-package
and micc-package-simple
into
micc-package
, common partsmicc-package-general
, general package specific partsmicc-package-simple
, simple package specific parts
we also change all names of cookiecutter template from micc-whatever>
to template-<whatever>
Thus, when a general package is created, we must use the templates (in this order)
template-package
template-package-general
template-package-simple-docs
template-package-general-docs
and when a simple package is created, we must use the templates (in this order)
template-package
template-package-simple
template-package-simple-docs
Then converting a simple python project to a general python project is simple:
- add directory
<package_name>
to the project, - copy
<packaga_name>.py
to<packaga_name>/__init__.py
. - add the template
template-package-general-docs
v0.5.10 (2019-09-09)¶
Fixed issue #4:
- added command
micc build [-s|--soft-link]
, builds all binary projects (C++ and f2py) - builders removed from Makefile
Fixing issue #4 raised a problem: the environment of subprocess is not the same as
that in which micc script is run. In particular which python
returns /usr/bin/python
.
As a consequence pybind11 is not found. after using subproces.run( ... , env=my_env, ...)
things are working (with the caveat that subprocess picks up the environment in which it is
running. If run in eclipse, e.g. it will pick up the environment from which eclipse was started.
As a consequence it may pick up the system python and not find pybind11.)
v0.5.9 (2019-09-06)¶
fixed issue #1
v0.5.8 (2019-09-06)¶
fixed issue #3 for cpp modules
v0.5.7 (2019-09-06)¶
fixed issue #3 for f2py modules
v0.5.6 (2019-07-19)¶
in v0.5.5, we could do:
> micc module bar
> micc module bar --f2py
> micc module bar --cpp
without error, in arbitrary order. However, the f2py module and the C++ module
will generate the same bar.<something>.so
file in the package folder, which
is obviously wrong. In addition the .so files will be in conflict with bar.py
.
This version fixes this problem by verifying that the module name is not already in use. If it is, an exception is raised.
v0.5.5 (2019-07-19)¶
- add cookiecutter template for C++ modules.
We need:
- a C++ compiler (icpc or g++). On my mac i’d like to use GCC installed
with
brew install gcc
, or the g++ that comes with xcode. On the cluster I prefer Intel’sicpc
. - a Boost.Python library for the python in the environment. On the cluster
there are modules with Boost.Python such as the Intel Python distribution.
Building Boost with python isn’t particulary easy, so, let’s see what Anaconda
cloud provides.
- boost-cpp v1.70, which has 1.76M downloads, so it is certainly worth trying. bad luck: does not provide Boost.Python. A pity, I always like to have the latest Boost version.
- libboost: boost v1.67, not the most recent but certainly ok, unfortunately also no Boost.Python.
- py-boost <https://anaconda.org/anaconda/py-boost>: boost v1.67 and
``libboost_python.dylib` and
libboost_python37.dylib
. We used this before.
- The include files
numpy_boost.hpp
andnumpy_boost_python.hpp
are copied to the<package_name>/numpy_boost
directory. At some time they can be moved to a fixed place in the micc distribution. The user will probably never touch these files anyway.
For compiling the module we have several options:
- a build script, as for the f2pymodules,
- a separate makefile, or
- suitable targets in the project makefile.
For building we use the make system from et/make. For the time being, we
copy all makefiles into the <package_name>/make
directory. In the end
they could be
distributed with micc, together with numpy_boost.hpp` and
numpy_boost_python.hpp
. Note that the locations of the make
and
numpy_boost
directories are hardcoded as ./make
and ./numpy_boost
so that make
must be run in the package directory, for the time being.
That went fine, however, the issue with py-boost MACOS is still there. See
et/numpy_boost/test_numpy_boost/macosx_anaconda_py-boost.md
.
The problem could be fixed by reverting to python 3.6 using the conda
environment created as:
> conda create -n py-boost python=3.7 py-boost
> conda install py-boost python python-3.6.4 -c conda-forge
as suggested here (https://github.com/pybind/pybind11/issues/1579). Make sure to rerun the last line after installing a package This fix doesn’t seem to work anymore (2019.07.12). Python-3.6.4 is no longer available… Hence we look further:
- https://anaconda.org/anaconda/boost has libboost-python, but not also segfaults.
[At this point i checked that the test code works on Leibniz using the Intel Python distribution: it does! So the problem is the same as before, but the solution does no longer work.]
I gave up… Started compiling Boost.Python myself. Download the boost version you like, and uncompress
> cd ~/software/boost_1_70_0
> ./bootstrap.sh --with-python=`which python` --with-libraries="python"
> ./b2 toolset=darwin stage
Got errors like:
...failed darwin.compile.c++ bin.v2/libs/python/build/darwin-4.2.1/release/python-3.6/threading-multi/visibility-hidden/tuple.o...
darwin.compile.c++ bin.v2/libs/python/build/darwin-4.2.1/release/python-3.6/threading-multi/visibility-hidden/str.o
In file included from libs/python/src/str.cpp:4:
In file included from ./boost/python/str.hpp:8:
In file included from ./boost/python/detail/prefix.hpp:13:
./boost/python/detail/wrap_python.hpp:57:11: fatal error: 'pyconfig.h' file not found
# include <pyconfig.h>
^~~~~~~~~~~~
1 error generated.
"g++" -fvisibility-inlines-hidden -fPIC -m64 -O3 -Wall -fvisibility=hidden -dynamic -gdwarf-2 -fexceptions -Wno-inline
-DBOOST_ALL_NO_LIB=1 -DBOOST_PYTHON_SOURCE -DNDEBUG -I"." -I"/Users/etijskens/miniconda3/envs/pp2017/include/python3.6m"
-c -o "bin.v2/libs/python/build/darwin-4.2.1/release/python-3.6/threading-multi/visibility-hidden/str.o"
"libs/python/src/str.cpp"
The -I option specifies the wrong Python location! had to remove the Python
configuration in ~/user-config.jam
.
Still got “pyconfig.h not found errors”:
...failed darwin.compile.c++ bin.v2/libs/python/build/darwin-4.2.1/release/python-3.7/threading-multi/visibility-hidden/list.o...
darwin.compile.c++ bin.v2/libs/python/build/darwin-4.2.1/release/python-3.7/threading-multi/visibility-hidden/long.o
In file included from libs/python/src/long.cpp:5:
In file included from ./boost/python/long.hpp:8:
In file included from ./boost/python/detail/prefix.hpp:13:
./boost/python/detail/wrap_python.hpp:57:11: fatal error: 'pyconfig.h' file not found
# include <pyconfig.h>
^~~~~~~~~~~~
1 error generated.
"g++" -fvisibility-inlines-hidden -fPIC -m64 -O3 -Wall -fvisibility=hidden -dynamic -gdwarf-2
-fexceptions -Wno-inline -DBOOST_ALL_NO_LIB=1 -DBOOST_PYTHON_SOURCE -DNDEBUG -I"."
-I"/Users/etijskens/miniconda3/envs/ws2/include/python3.7" -c -o
"bin.v2/libs/python/build/darwin-4.2.1/release/python-3.7/threading-multi/visibility-hidden/long.o"
"libs/python/src/long.cpp"
The -I option specifies /Users/etijskens/miniconda3/envs/ws2/include/python3.7
whereas on my mac it is called /Users/etijskens/miniconda3/envs/ws2/include/python3.7m
.
a soft link python3.7
which links to python3.7m
solves the problem.
Alternatively, edit the project-config.jam
file and replace the using python :
line
with (on a single line, I guess):
using python : 3.7 : /Users/etijskens/miniconda3/envs/ws2/bin/python \
: /Users/etijskens/miniconda3/envs/ws2/include/python3.7m \
: /Users/etijskens/miniconda3/envs/ws2/lib ;
Now Boost.Python builds fine. The libraries are in stage/lib
:
> ll stage/lib
total 13952
-rw-r--r-- 1 etijskens staff 935152 Jul 12 12:23 libboost_numpy37.a
-rwxr-xr-x 1 etijskens staff 73852 Jul 12 12:23 libboost_numpy37.dylib*
-rw-r--r-- 1 etijskens staff 5750432 Jul 12 12:23 libboost_python37.a
-rwxr-xr-x 1 etijskens staff 374980 Jul 12 12:23 libboost_python37.dylib*
In /Users/etijskens/miniconda3/envs/ws2/lib
create soft links to both
libraries:
> cd /Users/etijskens/miniconda3/envs/ws2/lib
> ln -s path/to/boost_1_70_0/stage/lib/libboost_python37.dylib
> ln -s path/to/boost_1_70_0/stage/lib/libboost_numpy37.dylib
In /Users/etijskens/miniconda3/envs/ws2/include
, if there is no boost
subdirectory, create soft links to
libraries:
> cd /Users/etijskens/miniconda3/envs/ws2/include
> ln -s path/to/boost_1_70_0/boost
If there is already a boost
subdirectory:
> cd /Users/etijskens/miniconda3/envs/ws2/include/boost
> ln -s path/to/boost_1_70_0/boost/python.hpp
> ln -s path/to/boost_1_70_0/boost/python
… on Linux this works fine. On my Mac however I keep running into problems.
While googling for a solution, I came across pybind11. These sections in the readme makes me particularly curious:
- pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa, mainly to create Python bindings of existing C++ code.
- Think of this library as a tiny self-contained version of Boost.Python with everything stripped away that isn’t relevant for binding generation. Without comments, the core header files only require ~4K lines of code and depend on Python (2.7 or 3.x, or PyPy2.7 >= 5.7) and the C++ standard library. This compact implementation was possible thanks to some of the new C++11 language features (specifically: tuples, lambda functions and variadic templates). Since its creation, this library has grown beyond Boost.Python in many ways, leading to dramatically simpler binding code in many common situations.
- Find some examples of its use in Using pybind11
Works as a charm. Comes with a cross-platform CMake build system that works out
of the box. Must put a soft link to the pybind11 repository in the project
directory for the add_subdirectory(pybind11)
statement to work. (in section
“6.3.3 find_package vs. add_subdirectory” of the pybind11 manual, it is stated
that this can be overcome with find_package(pybind11 REQUIRED)
(this finds
the pybind11 installed in the the current python environment out of the box -
which is a good reason to conda|pip install pybind11
, rather than check it
out from github).
That releaves me from maintaining the make
stuff i wrote. (With a little bit
of CMake code for the f2py modules everything becomes much better streamlined and
the et/make
system is no longer needed.
Had some trouble making it work on Leibniz. CMake’s FindPythonInterp
relies on
python-config
to pick up the location current python executable. Unfortunately,
the Python environment I was using only defines python3-config
, and not
python-config
. That soft link still had to be added.
Must now figure out how to deal with numpy arrays… There is some interesting information in section “11.6 Eigen” of the pybind11 manual, allowing numpy arrays to be passed by reference and work with Eigen matrices on the C++ side. Using Eigen instead of Boost.MultiArray has the additional advantage that there is some dense and sparse linear algebra routines are also available.
Set Boost_INCLUDE_DIR
and Eigen3_INCLUDE_DIR
in pybind11/CMakeLists.txt
.
Now Eigen tests are run as well and no failures observed.
The fact that cmake does so well, make me wonder if i shouldn’t use cmake for the f2py modules as well instead of a shell script that will fail on windows. Scikit-build contains a FindF2py.cmake The tool, however, lets f2py find out the Fortran and C compiler it should use, and does not add any optimisation flags.
There is a slight catch in using CMake: the filename CMakeLists.txt
is fixed.
As “fixed”” as in “impossible to change”
(see this).
That requires us to have a separate directory for each cpp module we want to add.
We did not have that restriction with f2py modules. For reasons of consistency,
we might change that. Module directory names should then, probably, be set as
f2py_module
and cpp_module
rather than the other way around, so modules
of the same type appear consecutivel in a directory listing. Sofar, Python modules
added through micc module my_module
appear as my_module.py in the package directory.
CMake version is working. F2py autoselects the compilers (Fortran and C) We further accomplished
- get rid of the soft link to pybind11 by referring to its installation
directory in CMakeLists.txt. Fixed! Finds the pybind11 installed in the current
Python environment out of the box! (see pybind11 documentation
Build systems/Building with CMake/find_package vs add_subdirectory)
). done. - get rid of the directory cmake_f2py_tools by referring to its location inside
micc in CMakeLists.txt (we do not want endless copies of these, nor soft links)
How can i install the micc CMake files so i can find them in the same way
find_package(micc CONFIG REQUIRED)
. Playing the trick of pybind11 to install files into/path/to/my_conda_environment/share/pybind11
seems to be non-trivial (https://github.com/pybind/pybind11/issues/1628. Let’s see if we can work around this. We can indeed easily work around. Themicc.utils
module now has a functionpath_to_cmake_tools()
that returns the path to the cmake_tools directory (using__file__
). This path is added to the template parameters (before they are exported to cookiecutter.json). Then Cookiecutter knows the path and can insert it in theCMakeLists.txt
. Simple, and no loose ends. done.
The approach to expose the micc/cmake_tools
directory to the CMakeLists.txt
of an fpy module through utils.path_to_cmake_tools()
is a bit static as it
hardcodes the path of the micc version that was use to create the module. If it
changes, because that micc version is moved, or because development continues in
another python environment, the build will fail. Instead we rely on the CMake
variable ${PYTHON_SITE_PACKAGES}
and append /micc/cmake_tools
to it.
Now add the cookiecutter templates for C++ modules
Tested f2py and cpp modules and their test scripts.
v0.5.4 (2019-07-10)¶
- add cookiecutter template for fortran modules with f2py. We need:
- f2py, comes with Numpy
- a fortran compiler
- a C compiler
- what can be provided out of the box by conda?
- support for this on on the clusters
I followed this advice:
f2py-running-fortran-code-in-python
and installed gcc from homebrew brew install gcc
. Inside the Conda
environment I created soft links to gcc, g++ and gfortran.
There is an issue with Fortran arguments of type real*16, which become
long_double
instead of long double
in the <modulename>module.c
wrapper file. The issue is circumvented by editing that file and running
f2py_local.sh a second time. The issue occurred in gcc 7.4.0, 8.3.0 and
9.1.0. Switching to gcc provided by XCode does not help either. However,
adding -Dlong_double="long double"
to the f2py command line options
solves the problem nicely. :)
I, typically, had different bash scripts for running f2py, one for building
locally and one for each cluster. It would be nice if a single script would
do and pickup the right compiler from the environment where it is run, as
well as set the correct compiler options. There may be different f2py modules,
so there will be a different script for every f2py module: f2py_<module_name>.sh
.
Preferentially <module_name>
ends with f90
. The module name appears
also inside the script. The script looks for a ifort, and if absent for
gfortran in the environment. It uses gcc for compiling the C-wrappers and
for f2py. If one of the components is missing, the script exits with a non-
zero error code and an error message. The makefile can call:
for s in f2py_*.sh; do ./${s}; done
Do we want a fortran module or not? the fortran module complicates stuff, as it appears as a namespace inside the python module:
# a) with a fortran module:
# import the python module (built from compute_f90_a.f90) which lives
# in the proj_f2py package:
import proj_f2py.compute_f90_a as python_module_a
# create an alias for the fortran module inside that python module, which
# is called 'f90_module'. The fortran module behaves as any other member
# in the python module.
f90 = python_module.f90_module
# b) without a fortran module:
# import the python module (built from compute_f90_b.f90)
# this doesn not have a fortran module inside.
import proj_f2py.compute_f90_b as python_module_b
Documenting fortran modules with sphinx is problematic. There exists a sphinx
extension sphinx-fortran,
but this works presumably only with sphinx
versions older than 1.8, and it is not avtively maintained/developed, which is a
pity imho. As an alternative we include a file <project_name>/<package_name>/<module_f2py>.rst
which has a suitable template for adding the documentation. As we actually want to
document a python module (built from Fortran code with f2py), we expect the user to
enter documentation for the wrapper functions, not for the pure Fortran functions.
That goes in the <project_name>/<package_name>/<module_f2py>.f90
file but is
not exposed in the project documentation.
v0.5.3 (2019-07-09)¶
- check for overwriting files (we must specify
overwrite_if_exists
for cookiecutter because it will already report an error if just the directories exist. Adding files to existing directories is not supported out of the box.) The more components one can add, the higher the chance that there is going to be a name clash and files are going to be overwritten. We do not want this to happen. We propose thatmicc
should fail when files are overwritten, and that the command be run again with a--force
option.- Maybe, we can monkey patch this problem in cookiecutter. No success.
- Create a tree of directories and files to be created and check against the pre-existing tree. Seems complicated.
- Create the tree to be added in a temporary dir which does not yet exist, and than check for collisions. That seems feasible.
v0.5.2 (2019-07-09)¶
- add option
--f2py
tomicc module ...
v0.5.1 (2019-07-09)¶
micc create ...
must write a .gitignore file and other configuration files. Addition of modules, apps do not change these.- Cookiecutter template micc-module-f2py added, no code to use it yet
v0.5.0 (2019-07-04)¶
- Fixed poetry issue #1182
v0.4.0 (2019-06-11)¶
- First functional working version with
micc create
micc app
micc module
micc version
micc tag
v0.2.5 (2019-06-11)¶
- git support
git init` in ``micc create
micc tag
v0.2.4 (2019-06-11)¶
- Makefile improvements:
- documentation
- tests
- install/uninstall
- install-dev/uninstall-dev
v0.2.3 (2019-06-11)¶
- Using pyproject.toml, instead of the flawed setup.py
- Proper local install and uninstall. By Local we mean: not installing from PyPI. we had that in et/backbone using pip. But pip uses setup.py which we want to avoid. There is not pyproject.toml file sofar…
Moving away from setup.py and going down the pyproject.toml road, we can choose
between poetry
and flit
.
Although, I am having some trouble with reusing some poetry code, i have the impression that it is better developed, and has a more active community (more watchters, downloads, commits, …)
A pyproject.toml was added (used poetry init
to generate pyproject.toml).
First issue is how to automatically transfer the version number to our python
project. This is a good post
about that.
- using pkg_resources implies a dependence on setuptools = no go
- using tomlkit for reading the pyproject.toml file implies that the pyproject.toml file must be included in the distribution of the package. Since pyproject.toml is complete unnnecessary for the functioning of the module, we’d rather not do that. So, we agree with copying the version string from pyproject.toms to the python package (=duplicating). This is basically the same strategy as used by bumpversion.
- the command poetry version … allows to modify the version string in
pyproject.toml. In principle we can recycle that code. However, we could not
get it to work properly (see issue https://github.com/sdispater/poetry/issues/1182).
This could probably be circumvented by creating my own fork of poetry.
- it is simple to write a hack around this (read the file into a string, replace the version line, and write it back. this preserves the formatting but in the unlikely case that there is another version string in some toml table it will be incorrect.
- the toml package is much simpler than tomlkit, does not cause these problems, but it does not preserve the formatting of the file.
- poetry itself uses a separate __version__.py file in the package, containing
nothin but
__version__ = "M.m.p"
. This is imported in __init__.py asfrom .__version__ import __version__
. This makes transferring the version from pyproject.toml to __version__.py easy.
Let’s first check if we can achieve a proper local install with poetry … Install a package:
> poetry build
> pip install dist/<package>-<version>-py3-none-any.whl
Uninstall:
> pip uninstall <package>
This seems to do the trick:
> pip install -e <project_dir>
Install a dev package use cmd:
> pip install --editable <project_dir>
Uninstall:
> rm -r $(find . -name '*.egg-info')
But take care, uninstalling like this:
> pip uninstall <package>
removed the source files. See this post.
v0.1.21 (2019-06-11)¶
first working version
v0.0.0 (2019-06-06)¶
Start of development.
Open Issues¶
#25
#24 [feature] add indentation to logging¶
> micc -p foo create
[INFO] Creating project (foo):
Python package (foo): structure = (foo/foo/__init__.py)
[INFO] Creating git repository
[INFO] ... done.
[INFO] ... done.
should look like:
> micc -p foo create
[INFO] Creating project (foo):
Python package (foo): structure = (foo/foo/__init__.py)
...
[INFO] Creating git repository
...
[INFO] done.
[INFO] done.
#22 [issue] building cpp module fails when pybind11 is pip-installed¶
(at least on a conda environment) it works however when it is conda-installed.
DEBUG micc-build-cpp_mod4.log:utils.py:295 (stderr)
CMake Error at CMakeLists.txt:14 (find_package):
Could not find a package configuration file provided by "pybind11" with any
of the following names:
pybind11Config.cmake
pybind11-config.cmake
Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
"pybind11_DIR" to a directory containing one of the above files. If
"pybind11" provides a separate development package or SDK, be sure it has
been installed.
also building an f2py module fails for the same reason.
conda installs the missing files in ~/miniconda3/envs/micc/share/cmake/pybind11
#5 [feature] packaging and deployment, and the use of poetry in general¶
The Python Packaging User Guide has a section Packaging binary extensions, with two interesting subsections Publishing binary extensions and Cross-platform wheel generation with scikit-build. Thus, it seems we are on the right track as scikit-build is based on CMake, which we are using too for building our f2py and C++ binary modules. Unfortunately, the section Publishing binary extensions is broken, as it mentions “For interim guidance on this topic, see the discussion in this issue.” There, two interesting links are found:
- Building and testing a hybrid Python/C++ package: this is describe a setup.py approach, also based on pybind11 and CMake.
- cmake-pip: is a simple wrapper around CMake in order to be able to have CMake extensions as python modules.
Note that the setup.py approach can become rather convoluted: https://github.com/zeromq/pyzmq/blob/master/setup.py.
The scikit-build approach is mentioned .
These approaches request the user that installs your package to also build the binary Python extensions (f2py, C++), which may be challenging on Windows.
The Python Packaging User Guide suggests Appveyor , a CI solution, for providing windows support: https://packaging.python.org/guides/supporting-windows-using-appveyor/.
how does poetry deal with binary extensions?
issue #8 cookiecutter.json files¶
These files are written in the template directories of the micc installation. If micc happens to be installed in a location where the user has no write access, micc will not work.
Closed Issues¶
#1 [bug] FileExistsError in micc module¶
in v0.5.6:
The commands:
> micc module --f2py <module_name>
> micc module --cpp <module_name>
generate:
FileExistsError: [Errno 17] File exists: '<package_name>/tests'
if the flag --overwrite
is not specified. This behavior is incorrect.
Only if existing files are overwritten an exception must be raised, not
when a new file is added to an existing directory.
#3 [feature] add useful example code to templates¶
Put more useful example code in
cpp_{{cookiecutter.module_name}}/{{cookiecutter.module_name}}.cpp
-> added in v0.5.7.f2py_{{cookiecutter.module_name}}/{{cookiecutter.module_name}}.f90
as well as in the corresponding test files.
v0.5.10
#4 [bug] build commands for f2py and cpp modules¶
<package_name>/Makefile
contains wrong builder for f2py modules and no builder for
cpp modules.
Running CMake build from the ccd ..ommand line:
> cd <package_name>/cpp_<module_name>
> mkdir build_
> cd build_
> cmake CMAKE_BUILD_TYPE=RELEASE ..
> make
Then, either copy the .so
file to <package_name>
, or make a softlink.
A simple package (feature #2) should have simple documentation, and complete documentation when
converted to a full blown package.
feature #11 add log files to micc build
¶
controlling the output with verbose is not sufficient. If one of the build commands fails we want to print all output for building that module. that’s hard to control with verbose.
issue #9 prohibit creation of a micc project under another project¶
This implies asserting that none of the parent directories of the output directory
is a project directory (in micc_create_simple
and micc_create_general
issue #7 cookiecutter.json files are temporary¶
While workin on issue #2 I realized that these are in fact temporary files, which do neither belong in the template directories (although cookiecutter requires them). It is better to remove these files when cookiecutter is done.
v0.5.11
#2 [feature] simple python project¶
add --simple
flag to micc create
to create a simple (=unnested) python module <package_name>.py
instead of the nested <package_name/>__init__.py
a simple package should be convertible to a normal package
v0.5.13
issue #12 common items in micc.json files¶
While workin on issue #2 I realized that there are now several micc.json` files with common
items which are in fact copies. we need either a single ``micc.json
or a way of isolating
the common parts in a single file.
Fixed by itself. If there are multiple templates, every new template adds parameters to the original.
#13 [feature] os.path -> pathlib¶
more obvious manipulation of file paths
#14 [feature] add flag for nesting a project inside another project¶
mainly for running tests.
#6 [feature] decomposition¶
maybe it is usefull to limit the number of files in the cookiecutter templates. For now even the
simples project contains 11 .rst
files. For a beginner that may be too much to grasp. Maybe it is ]
usefull to start with a README.rst
only and have a micc doc [options]
command that adds documentation
topics one at a time:
> micc doc --authors
> micc doc --changelog|-c # or
> micc doc --history|-h
> micc doc --api|-a
> micc doc --installation|-i
this is perhaps useful, but rather more complicated. E.g if we first create a package with several modules (python, f2py, cpp) and then start to add documentation. This is a more complicated situation and one in which errors will be easily made, and more difficult to maintain.
#16 [issue] poetry 1.0.0b1 uses different cleo than 0.12.17¶
this break our code for retrieving the current version number.
#15 [issue] using poetry bumping the version in pyproject.toml¶
currently we do this by using poetry’s source code (import). As poetry recommends a singly installation of poetry system-wide, this adds an extra dependency (i.e. poetry itself) on top of the single installation. There is no way of guaranteeing that both versions are the same. Ideally, we would rely on only the system version of poetry.
#19 [issue] avoiding poetry¶
Poetry is currently used for bumping versions only. As poetry does not play well with conda, (see issue #18) relying on poetry (and therefore adding it as a dependency of micc) is a ticking time bomb because when poetry is there users will use it, even if it is documented that it does not work. So i propose to use bump(2)version for bumping versions by default. This will always work well with Anaconda Python versions.
interesting links:
#21 [issue] micc create¶
raises exception (cannot create project inside another project) when run from a project directory with a relative path. E.g.:
~/path/to/micc/ > micc create
fails when given the path ../foo although this would be created in ~/path/to/soep
.
#20 [issue] install dependencies in current conda environment¶
As poetry build works, and pip install dist/<wheel> too, even in a conda python environment, this is no longer an issue,.
#18 [issue] two tools for reading/writing toml files¶
Currently, we are using both tomlkit and toml for reading and writing toml files. Better stick to one.
#17 [issue] poetry 1.0.0b1 does not play well with conda¶
Poetry made me a virtual environment for micc, based on miniconda3
’s active python version
(which was 3.7.3). However, it did not pickup the correct python standard library (used 3.6.whatever
instead), obviously a nightmare. Thus, if we want to use poetry, we must use a non-conda Python, or
if we want to use conda python versions, we must refrain from poetry.
solved: only poetry install
does not work well with a conda python environment,
poetry build
does fine
#23 [issue] remove interactivity from micc app and micc module¶
behavior is now steered with flags. default behavior is abort if pre-existing files would be overwritten.
TODO¶
check for a the presence of a global CMake or add it as a dev dependency in pyproject.toml.
allow Fortran binary extensions to be build and installed with CMake_, just as for C++ binary extensions. This provides a more uniform interface.
pybind11 v2.6.1 depends on a more recent version of CMake (v3.4).
update tutorial 7
move
LINKS.rst
to a separate git repo (it changes too often)add a remove command for removing a component of a micc project see https://github.com/etijskens/et-micc/issues/32
add a rename command for renaming a component of a micc project or the project itself. see https://github.com/etijskens/et-micc/issues/29
Put makefile targets into micc commands and remove micc/makefile? This makes a more uniform interface. Use subprocess or lrcmd for this?
Fortran compiler options (while using f2py): is
-O3
enough?allow for multiple Fortran/C++ source files?
how to add external projects for f2py and C++ modules (include files, libraries)?
- this was fixed for C++ by using the CMake framework
check cppimport
undo
micc app ...
?undo
micc module ...
?check if project_name exists already on readthedocs or pypi. If not abort and print a message that suggests to use a different name, or to create the project anyway by using
--force
.remove dependency on toml in favor of tomlkit which comes with poetry. (now that we are fixed poetry issue #1182)
regression tests
Reflect about “do we really need poetry? (see below)
Poetry considerations¶
- What are we using poetry for?
- Do we really need it?
- Maybe we should wait a bit for poetry to mature, before we start building our micc project around it.
- Maybe we should decouple micc and poetry?
- Maybe we should still use
setup.py
rather than poetry because it is well established?
There is a poetry issue on poetry+anaconda python Any plans to make it work with anaconda python?. Locally, we are completely relying on Anaconda Python. Consequentially, I am not completely feeling comfortable with poetry`_ - but it is very actively developed.
Anaconda Python used to be very convenient, but maybe the standard python+pip+virtualenv is good enough today? One advantage anaconda python still has is that its numpy well aligned numpy arrays which is in favor of vectorization.
So far we use poetry for:
building wheels (which are used for installing and publishing):
poetry build
, typically inside theMakefile
. However, I haven’t figured out how to go with e.g. f2py modules and C++ modules.poetry.console.commands.VersionCommand
for updating version strings- we are not using
poetry install
to create a virtual environmentpoetry run ...
to run code in that virtual environment
We could use
poetry install
to create a virtual environment and point to it in eclipse/pydev so that we will always run our code in that environmenttests should probably be run as:
> poetry run pytest tests/test*
Development plan¶
What do we actually need?
a standardized development environment
- click : for command line interfaces
- sphinx : for documentation
- pytest : for running tests
- flake8 : for assuring PEP 8 compatibility
- cookiecutter : if we want sth based on existing templates
- tox ?
- poetry?
a standardized way of creating projects for packages and apps.
automation of project management tasks, e.g. CI, publishing, …
This package was inspired by Cookiecutter.
Inspiration for the project templates came from:
Interesting posts:
- Here is a particularly readable and concise text about packaging Current State of Python Packaging - 2019 (Pycoder’s weekly #372 june 11, by Stefano Borini). The bottom line is: use poetry. After reading (just part) of the documentation I concluded that poetry solves a lot project management issues in an elegant way. I am likely to become addicted :).
- version numbers: adhere to Semantic Versioning
Think big, start small…¶
Maybe it is a good idea to get everything going locally + github, and add features such as:
- readthedocs,
- publishing to pypi,
- travis,
- pyup,
- …,
incrementally.