So I'm pretty deep into making a custom job/process custom manager module, and when i got to integrating it I came across an interaction I hadn't seen before with modules and namespaces.
Code speaks better than words:
So there are two slightly different scripts:
test1:
import jobManager
jobManager.jobMap = {'test1':'test123'}
AND test2:
import jobManager
jobManager.jobMap = {'test2':'test222'}
Top Level Script:
import test1
import test2
print(test1.jobManager.jobMap)
print(test2.jobManager.jobMap)
So when I run the top level script it prints:
{'test2':'test222'}
{'test2':'test222'}
But my expected output is:
{'test1':'test123'}
{'test2':'test222'}
Is this just a case where test1.jobManager and test2.jobManager are actually the same namespace? Is there a way to keep them separate?
Is this just a case where test1.jobManager and test2.jobManager are actually the same namespace?
Yes.
Is there a way to keep them separate?
Not without creating another module.
Related
I have a question about save_main_session and best practices, and please let me know if there is a doc somewhere that covers this question. So with save_main_session set to False, if my DoFn in the process method uses for example standard lib copy module, Beam's FileSystems API or my custom module, if I import those at the module level (top of the file) in which the DoFn is defined, this would fail in Dataflow service with an error that says that copy (etc) module was not found from the process method (which all makes sense), and I could fix this by either:
importing copy inside the process method
"saving" copy reference/object as a field/provider/etc in the DoFn instance
setting save_main_session to True
I don't want to set save_main_session to True because afaiu it captures whole main session and I have bunch of objects that are not serializable in there, and overall find save_main_session to be smelly and hacky. 1st option is kinda smelly as well and doesn't always work - tho imports are cached so performance wise should be okish - but it would not work for my custom modules afaiu (unless I explicitly install/send them over to the workers). And lastly 2nd is kinda hacky - working around the Beam framework.
I'm leaning mostly towards the 2nd option, but it just doesn't feel right to no be able to just use the global imports and workaround it be adding and using instance field(s).
What is the best practice for this problem? I know the examples are suggesting to set save_main_session to True, but that again has consequences and just smells. Are there better options?
According to the documentation, if you have objects in your global namespace that cannot be pickled, you will get a pickling error. If the error is regarding a module that should be available in the Python distribution, you can solve this by importing the module locally, where it is used.
The DoFn class comes with a setup method that is called once per DoFn instance. You can override this method, and perform your imports there.
As a note, this method is available in Beam's Python release for 2.13.0. If you're using an earlier version, you can override start_bundle in your DoFn to perform the import there.
I am writing a shared library for Jenkins and am running across a bit of an organizational issue.
I have a number of pipeline scripts in var, however I'm finding there are a number of repeating functions and the code is not very dry.
One solution for this has been to create helper functions inside var like var/log.groovy, var/formatter.groovy. This has worked fine and I've been calling these functions from within my pipeline scripts like var/myPipeline.groovy.
I would just like to organize my var folder a bit better and keep my helper functions inside var/utils/log.groovy for example.
The problem is I'm not sure how to access them from my pipeline scripts inside var when I put them inside a sub-directory.
How can I access them? Or is there a better way to organize my global functions?
You can put them in src in a package structure that makes sense organizationally. Them import the right things in your var scripts.
in /src/com/yourco/Formatter.groovy
package com.yourco
class Formatter {
def static String formatThis(String something) {
"this is ${something}"
}
}
In your var
import com.yourco.Formatter
..
..
..
echo Formatter.formatThis('test')
Yes, I read dart import and part of directives in same file
I have this structure:
lib/
src/
one/
SomeClass.dart
one.dart
mylib.dart
main.dart
I'm trying to achieve this behavior:
All public and hidden variables are fully accessible inside library.
All public variables from library are accessible to main.dart.
There is a problem. For some weird reason I can't use any directive with 'part of'. So I can't use this in the one.dart:
part of mylib;
import 'SomeClass.dart';
//somecode
So I either need to move class definition from SomeClass.dart to one.dart (and that will make code less readable and mixed up) or I need to move 'import' in the mylib.dart.
library mylib;
import 'SomeClass.dart';
part ..
I don't like either of the options. In the second case I will need to parse all modules then and move import/exports. Which will definitely break something.
It may sound weird but the project will build from various modules automatically. And one/ is one of them.
This app design is bad, I know. But either I need to find a better way or just make all variables public and don't bother.
Default to defining one type per file, not using part, and importing only the files you need. This covers the majority of use cases.
Now, let's say you have two types that are commonly used together - for example, a Thing and a ThingException that gets thrown when Thing does bad things. Importing both of these files everywhere is tedious, so you have three options with their own tradeoffs:
Declare both types in the same file.
Declare each type in its own file, and have the 'primary' file export the other. So, thing.dart exports thing_exception.dart. Importing thing.dart gives the importing file access to both.
Declare each type in its own file, and have the other file be a 'part of' the primary file. So, thing_exception.dart declares that it is 'part of' thing.dart. Importing thing.dart gives the importing file access to both files.
For this simple type and its exception, your best bet is to use option 1. When the amount of code grows or the two types diverge in visibility, this option becomes less attractive. This puts options 2 and 3 are on the table.
When you have separate files, option 2 is often a better approach than options 3 because you maintain some flexibility - you could only import thing_exception.dart and not thing.dart. If you use option 3, you can't do this - you either import all of the parts or none of them. This is the error you are seeing when trying to do a part and import in the same file.
Option 3 becomes valuable when you the code is in the two files is highly dependent on one another and they need the ability to access private members of each other. This is less common.
When you have a bunch of files like this together, it becomes a 'library' in the more traditional sense. You declare a main library file (your my lib.dart file) that exports files:
export 'public.dart';
export 'other_public.dart';
The bin script imports the library as a whole, but it can't see anything that isn't explicitly exported from my_lib.dart.
import 'package:mylib/mylib.dart';
Here's an example of a smallish package that uses all three of these options together for a good reference.
I think you will have better luck using import, and export with show. (Use of part of is now discouraged.)
Answers to this question may help you: When to use part/part of versus import/export in Dart?
Also the Creating library packages documentation: https://www.dartlang.org/guides/libraries/create-library-packages
Many of the samples I have seen for angular2 have the following import statement:
import {bind} from 'angular2/di';
I am working in VS Code (with TypeScript) and it complains about not being able to find the angular2/di module.
However I do see a bind function defined in angular2/angular2.d.ts. If I change the import statement to the following, then the error goes away.
import {bind} from 'angular2/angular2';
Is the question in the title off-base and I am making some erroneous assumption?
If not, why do many samples reference one module to import the bind function from, yet I seem to be able to get it from a different module?
Most likely because you looked at versions from older alphas. Look at the angular2.ts file. Everything is exported from it. Also note that the d.ts is going to contain everything to resolve types in your IDE and at compilation time. What import does is actually importing the .js files.
I managed to add a relative path dependency in my app
name: myapp
description: A sample app
dependencies:
mylib:
path: ../mylib
and then import it in my source code
import 'package:mylib/mylib.dart';
However, the name 'mylib' is taken from the library package pubspec and if I want to change it (for example 'mynewlib'), I have to change the name everywhere (pubspec AND dart source code)
It also prevent having 2 packages with the same name (yes I know, weird, but I don't control what people put in pub.dartlang.org). What I'd like to do is something like
name: myapp
description: A sample app
dependencies:
mylib:
path: ../mylib
name: mynewlib
and have in source code
import 'package:mynewlib/mylib.dart';
However I cannot find the proper syntax and whether that's possible or not. (Sample code ready for testing is here: https://github.com/alextekartik/dart-test/tree/master/lib_test). To note that here I'm not talking about library name but package name (and naming the package mylib can be confusing)
There is no way to define another name for a package itself (as far as I know - if there is, I'd be interested too).
However, as a workaround, you could rewrap it. For example, let's assume you have two "mylib" packages.
Create a new library application "mylib1". There, you import the first "mylib" and reexport it using export. Create another library application "mylib2" for the other "mylib". Then you have different package names to use in the same application.
Yes, it is kinda awkward, but as long as there is no better way...
You can resolve name conflicts at import with:
import 'package:mylib/mylib.dart' as Foo;
This will create a top level name to access the library API:
Foo.bar();