I'm trying to create a package (that includes several subpackages) for reuse and distribution. The plan is to provide a CLI entry point to allow easy launch. After building the package and installing it in a virtualenv, I get a ModuleNotFoundError for imports from the subpackages included in the main package.
I think this has something to do with setting the right paths in __init__.py, but having read multiple examples on the web, I'm still rather confused as to why anything should go in __init__py and what that something is.
The package is built so that the package name (and thus folder created in site-packages) is the same as the root in the directory structure below.
The directory structure is (simplified and with names changed):
mypackage
|- __init__.py
|- entrypoint.py
|- subpackage1
|-- __init__.py
|-- module1.py
|- subpackage2
|-- __init__.py
|-- module2.py
Note that all the __init__.py are empty
And entrypoint.py is:
from subpackage1.module1 import foo
from subpackage2.module2 import baz
if __name__ == "__main__":
pass
In my pyproject.toml, I define:
\[project.scripts\]
mypackage-cli = "maypackage:entrypoint"
After installing with pip, I run (in a virtualenv where I pip installed the package):
(myvenv) me#mymachine ~ % mypackage-cli
But I get:
ModuleNotFoundError: No module named subpackage1
Two things to note:
When running source locally, I have no issues
If I edit the files in site-packages to have from mypackage.subpackage1.module1 import foo I don't get the error anymore when running the installed package, but then when trying to run the same modified imports (i.e. changing to import mypackage.subpackage1.module1) locally in my dev env, I get a ModuleNotFoundError
What's the correct way to get the imports to work when packaged and when running locally in my dev env?
Thanks!
Your "top-level importable package" seems to be mypackage so all your import statements should start from there. For example from mypackage.subpackage1.module1 import foo.
In order to avoid confusion between "local" and "installed" (in site-packages), I recommend you to use the so-called "src-layout" for this project's directory structure. In combination with the "editable" installation, it is very convenient to work with.
Related
With Bazel I'm building an external library using http_archive together with
some patches which bring additional features:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name="some-lib",
build_file="#my-project//some-lib:BUILD.some-lib",
url="https://example/download/some-lib.tar.gz",
sha256="8d9405baf113a9f25e4fb961d56f9f231da02e3ada0f41dbb0fa4654534f717b",
patches=[
"//some-lib/patches:01-add-additional-features.dif",
],
patch_args=["-p1"],
patch_tool="patch",
)
The file structure looks like this:
some-lib
├── BUILD
├── BUILD.some-lib
├── include/additional_features.h
├── some-lib.bzl
└── patches
├── 01-add-additional-features.dif
└── BUILD
This basically works but I still struggle with adding include/additional_features.h
into the extracted source folder.
I first tried to just list the file in the filegroup I use to later run
configure_make like this:
filegroup(
name="all_srcs",
srcs=glob(["**"]) + ["#my-project//some-lib:include/additional_features.h"],
)
then I'm getting
no such target '//some-lib:includes/additional_features.h': target 'includes/additional_features.h' not declared in package 'some-lib'; however, a source file of this name exists.
My next idea was to use the tools http_archive provides to make the file part of the source folder.
While you can use patches to modify the extracted folder you'd need a dedicated
dif file just to create the extra header file, which then you would have to
create in advance (i.E. create a Bazel rule) and declare it a dependency etc..
which I'd like to avoid to keep things simple.
There is also patch_cmds which next to patches can be used to modify the
extracted source folder by running arbitrary bash commands so I tried something like this:
patch_cmds=[
"cp #my-project//some-lib:include/additional_features.h include/",
],
but this does not work for me, I'm getting
Error in fail: Error applying patch command cp #my-project//some-lib:include/additional_features.h include/:
cp: cannot stat '#my-project//some-lib:include/additional_features.h': No such file or directory
So it looks like the syntax for specifying a path like I do with build_file does
not work with patch_cmds or that file can't be accessed at that specific stage.
Does one of the approaches I tried actual work and I just didn't use the right
syntax?
What's the Bazel-way to add (a bunch of) readily available files (i.e. in the same
repository as the Bazel-rules I provide) to a http_archive based source directory?
Try putting exports_files(["include/additional_features.h"], visibility=["//visibility:public"]) in some-lib/BUILD, so that Bazel will let you reference source files from the //some-lib package in the external repository (and elsewhere).
I even thought the "no such target" error message suggested exports_files?
My Folder Structure looks like the following
person-package
|- __init__.py
|- person.py
|- person_manager.py
main.py
person_manager.py imports person.py
import person as x
The main.py imports person_manager.py
import person_package.person_manager as x
When running main.py I get:
ModuleNotFoundError: No module named 'person'
I know, I could solve that by changing the import of person_manager.py to the following
from . import person as x
However, when running now person_manager.py directly, I get:
ImportError: attempted relative import with no known parent package
So I can't test person_manager.py on its own.
What is the most elegant way to solve that?
1. I recommend to always use absolute imports (unless strictly impossible).
2. person-package is not a valid Python name since it contains a dash -, if I were you I would rename to person_package with an underscore _.
3. Since person_manager.py is part of a Python importable package (i.e. it is in a directory containing a __init__.py file), then it should not be run as python person_package/person_manager.py, but as python -m person_package.person_manager.
I have a Travis CI build that is failing; py.test seems to be trying to import the wrong module, though I cannot reproduce this locally. I expect it to import tools.lint.tests.test_lint, not lint.tests.test_lint, as you can see in the traceback, given that build has --full-trace! This leads to the error beneath it when it tries to do a relative import from the tools package.
The short trackback is:
___________________ ERROR collecting lint/tests/test_lint.py ___________________
.tox/py27/lib/python2.7/site-packages/py/_path/local.py:650: in pyimport
__import__(modname)
lint/__init__.py:1: in <module>
from . import lint
lint/lint.py:15: in <module>
from .. import localpaths
E ValueError: Attempted relative import beyond toplevel package
Given the name of the top level package is just the directory that the repo is in, I wouldn't be surprised to see that differ—but I'd still expect to see it there!
Take a look at the path Travis has that file at: /home/travis/build/w3c/wpt-tools/lint/tests/test_lint.py. The directory called tools on your computer is called wpt-tools on Travis, following the name of the repo on GitHub.
Vitally, wpt-tools isn't a valid Python package name, as Python packages cannot contain hyphens in the name. (They have to be an identifier). This leads py.test to conclude it isn't a package, despite the __init__.py contained within, and hence it doesn't include it in the import path, leading to the error when code tries a relative import from what is meant to be the top-level package.
There's a couple of solutions here:
The possibly simple one is renaming the repository so that it doesn't contain any hyphens, though obviously if you're an established repository this is likely undesirable.
Get Travis CI to run the code from some directory, by copying/moving the repository to a directory whose name doesn't contain a hyphen, at the start of before_install, using something like:
before_install:
- mv `pwd` /tmp/tools
- cd /tmp/tools
This will then run all the install and later steps from /tmp/tools, which will allow everything to run as expected.
(Note you cannot use a symbolic link here as os.getcwd() in Python will eliminate the link from the path, returning the real path, leading that seeming workaround not to work at all.)
I have this setup:
├── bin
│ ├── all.dart
│ ├── details
│ │ ├── script1.dart
│ │ └── script2.dart
| | .....
all.dart simply imports script1.dart and script2.dart and calls their main. The goal is to have a bunch of scripts under details that can be run individually. Additionally I want a separate all.dart script that can run them all at once. This will make debugging individual scripts simpler, yet still allowing all to run.
all.dart
import 'details/script1.dart' as script1;
import 'details/script2.dart' as script2;
main() {
script1.main();
script2.main();
}
script1.dart
main() => print('script1 run');
script2.dart
main() => print('script2 run');
So, this is working and I see the print statements expected when running all.dart but I have two issues.
First, I have to softlink packages under details. Apparently pub does not propagate packages softlinks down to subfolders. Is this expected or is there a workaround?
Second, there are errors flagged in all.dart at the point of the second import statement. The analyzer error is:
The imported libraries 'script1.dart' and 'script2.dart' should not have the same name ''
So my guess is since I'm importing other scripts as if they are libraries and since they do not have the library script[12]; statement at the top they both have the same name - the empty name?
Note: Originally I had all of these under lib and I could run them as scripts specifying a suitable --package-root on the command line even though they were libraries with main. But then to debug I need to run in Dart Editor, which is why I'm moving them to bin. Perhaps the editor should allow libraries under lib with a main to be run as a script since they run outside the editor just fine? The actual differences between script/library seems a bit unnecessary (as other scripting languages allow files to be both).
How do I clean this up?
I'm not sure what the actual question is.
If a library has not library statement then the empty string is used as a name.
Just add a library statement with an unique name to fix this.
Adding symlinks to subdirectories solves the problem with the imports for scripts in subdirectories.
I do this regularily.
It was mentioned several times in dartbug.com that symlinks should go away entirely but I have no idea how long this will take.
I have never tried to put script files with a main in lib but it is just against the package layout conventions and I guess this is why DartEditor doesn't support it.
The dart page http://pub.dartlang.org/doc/#adding-a-dependency describes how you can have a dart file (parser_test.dart) import files from its own package using the 'import "package:..." style of import. It seems to imply this is a good thing - better than using relative paths. This example shown is for a file in test which appears to be special. But, then why does it not make sense for importing same package lib files from a lib in the package. Maybe it does make sense, but if so the pub update does not make it convenient.
foo/
/lib/
foo_lib_1.dart
foo_lib_2.dart
src/
foo_lib_1/
foo_lib_1_impl.dart
foo_lib_2/
foo_lib_2_impl.dart
Assume foo_lib_2 uses foo_lib_1. There are two options for foo_lib_2.dart:
import "../foo_lib_1.dart";
import "packages:foo/foo_lib_1.dart";
My guess is the suggested approach is the first for any such import that resides under lib. The reason I think this is pub update seems to automagically provide a soft link in the packages folder of any of bin, test, or example to foo, like foo -> ../lib. Yet, it does not do the same for the packages folder in top level foo. This means to get the second type of import (i.e. the packages import) to work you need to add:
foo:
path: lib
to the dependencies of foo in the pubspec.yaml. Is there any advantage or disadvantage for a library to use the package style import to import another library (not in test, bin, or example) from its own package? Is there a reason for the apparent inconsistency?
After accepting the answer below, I still am not seeing it. Here is what I'm seeing in a shell session and I would like to reconcile this behavior with the answer. Any explanations appreciated. I am using emacs instead of DartEditor, thus the old-school command line approach here.
### Show all files, one dart library file and one yaml, plus empty
### lib and test folders
user#user-thinkpad:/tmp/uml_codegen_sample$ ls -R
.:
lib pubspec.yaml test
./lib:
plusauri.dart
./test:
### Show contents of pubspec
user#user-thinkpad:/tmp/uml_codegen_sample$ cat pubspec.yaml
name: domain_model
version: 0.0.1
description: >
Auto-generated support from /home/user/plusauri/modeling/plusauri.xmi.json
dependencies:
ebisu:
path: /home/user/open_source/codegen/dart/ebisu
### Run pub install and show the changes. Note there is a soft
### link to packages from test, but not lib.
user#user-thinkpad:/tmp/uml_codegen_sample$ pub install
Resolving dependencies...
Dependencies installed!
Some packages that were installed are not compatible with your SDK version 0.4.7+5.r21658 and may not work:
- 'pathos' requires >=0.5.0+1
You may be able to resolve this by upgrading to the latest Dart SDK
or adding a version constraint to use an older version of a package.
user#user-thinkpad:/tmp/uml_codegen_sample$ ls -R
.:
lib packages pubspec.lock pubspec.yaml test
./lib:
plusauri.dart
./packages:
domain_model ebisu pathos
./test:
packages
### Note here the program does not work, and suspiciously pub
### install put no packages link under lib like it did test
user#user-thinkpad:/tmp/uml_codegen_sample$ dart lib/plusauri.dart
Unable to open file: /tmp/uml_codegen_sample/lib/packages/ebisu/ebisu_utils.dart'file:///tmp/uml_codegen_sample/lib/plusauri.dart': Error: line 5 pos 1: library handler failed
import "package:ebisu/ebisu_utils.dart" as EBISU_UTILS;
^
### Copy the same dart file to test to show that it can run there
### just fine
user#user-thinkpad:/tmp/uml_codegen_sample$ cp lib/plusauri.dart test/
user#user-thinkpad:/tmp/uml_codegen_sample$ dart test/plusauri.dart
Main for library plusauri
user#user-thinkpad:/tmp/uml_codegen_sample$
### Finally, manually create the soft link in lib, to show it will
### then run
user#user-thinkpad:/tmp/uml_codegen_sample$ ln -s ../packages lib/packages
user#user-thinkpad:/tmp/uml_codegen_sample$ dart lib/plusauri.dart
Main for library plusauri
Actually, you can definitely import using the package:foo/foo_lib_1.dart syntax without needing to change your pubspec.yaml or even creating a pubspec.yaml in the first place!
You can see that this is true from a language level, in this test: https://github.com/dart-lang/bleeding_edge/blob/master/dart/tests/standalone/package/packages/package1.dart
and an example of this in the wild is: https://github.com/kevmoo/hop.dart/blob/master/lib/hop_tasks.dart#L17
I do not think there is any benefit from writing one way or another except that writing relative paths is slightly shorter.
From a project structure point of view, I would use relative path imports when I am drilling into subdirectories that are not going to be exposed to the user. src is generally seen as implementation specific details that won't be visible to external users, so use relative paths to your heart's content.
However, if you're working within multiple directories, then you should use package: imports to reinforce the idea that the parts are stand-alone and interchangeable. Within the lib directory itself, you want to say that these two libraries, although they might rely on each other, can live separately and are not bound by their physical location.
I would recommend not ever using ../ in your imports, as that is fragile and may break in strange ways if/when you modify directory structure or deploy.