How to find path to the package directory when the script is running with `pub run` command - dart

I am writing a package that loads additional data from the lib directory and would like to provide an easy way to load this data with something like this:
const dataPath = 'mypackage/data/data.json';
initializeMyLibrary(dataPath).then((_) {
// library is ready
});
I've made two separate libraries browser.dart and standalone.dart, similar to how it is done in the Intl package.
It is quite easy to load this data from the "browser" environment, but when it comes to the "standalone" environment, it is not so easy, because of the pub run command.
When the script is running with simple $ dart myscript.dart, I can find a package path using dart:io.Platform Platform.script and Platform.packageRoot properties.
But when the script is running with $ pub run tool/mytool, the correct way to load data should be:
detect that the script is running from the pub run command
find the pub server host
load data from this server, because there could be pub transformers and we can't load data directly from the file system.
And even if I want to load data directly from the file system, when the script is running with pub run, Platform.script returns /mytool path.
So, the question is there any way to find that the script is running from pub run and how to find server host for the pub server?

I am not sure that this is the right way, but when I am running script with pub run, Package.script actually returns http://localhost:<port>/myscript.dart. So, when the scheme is http, I can download using http client, and when it is a file, load from the file system.
Something like this:
import 'dart:async';
import 'dart:io';
import 'package:path/path.dart' as ospath;
Future<List<int>> loadAsBytes(String path) {
final script = Platform.script;
final scheme = Platform.script.scheme;
if (scheme.startsWith('http')) {
return new HttpClient().getUrl(
new Uri(
scheme: script.scheme,
host: script.host,
port: script.port,
path: 'packages/' + path)).then((req) {
return req.close();
}).then((response) {
return response.fold(
new BytesBuilder(),
(b, d) => b..add(d)).then((builder) {
return builder.takeBytes();
});
});
} else if (scheme == 'file') {
return new File(
ospath.join(ospath.dirname(script.path), 'packages', path)).readAsBytes();
}
throw new Exception('...');
}

Related

Executing Request against Go function in Docker image

I am trying to create an AWS Lambda function written in Go. To do this, I've followed the steps provided [here]. I can successfully build my Docker image. However, when I run the image, I receive the following warning:
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
I am using an M1 MacBook. But, since it was a warning, I thought I would still try to submit a request. My .go file looks like this:
package main
import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/lambda"
)
type MyEvent struct {
Name string `json:"name"`
}
func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
fmt.Println("Hello, world")
return fmt.Sprintf("Hello %s!", name.Name), nil
}
func main() {
lambda.Start(HandleRequest)
}
The cURL request looks like this:
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"name":"steve"}'
When I submit the request, the Terminal window just sits there. I do not see anything written in the Docker console window. What am I doing wrong? I'm not seeing any errors. At the same time, I'm not seeing any activity.

Install Custom Dependency for KFP Op

I'm trying to setup a simple KubeFlow pipeline, and I'm having trouble packaging up dependencies in a way that works for KubeFlow.
The code simply downloads a config file and parses it, then passes back the parsed configuration.
However, in order to parse the config file, it needs to have access to another internal python package.
I have a .tar.gz archive of the package hosted on a bucket in the same project, and added the URL of the package as a dependency, but I get an error message saying tarfile.ReadError: not a gzip file.
I know the file is good, so it's some intermediate issue with hosting on a bucket or the way kubeflow installs dependencies.
Here is a minimal example:
from kfp import compiler
from kfp import dsl
from kfp.components import func_to_container_op
from google.protobuf import text_format
from google.cloud import storage
import training_reader
def get_training_config(working_bucket: str,
working_directoy: str,
config_file: str) -> training_reader.TrainEvalPipelineConfig:
download_file(working_bucket, os.path.join(working_directoy, config_file), "ssd.config")
pipeline_config = training_reader.TrainEvalPipelineConfig()
with open("ssd.config", 'r') as f:
text_format.Merge(f.read(), pipeline_config)
return pipeline_config
config_op_packages = ["https://storage.cloud.google.com/my_bucket/packages/training-reader-0.1.tar.gz",
"google-cloud-storage",
"protobuf"
]
training_config_op = func_to_container_op(get_training_config,
base_image="tensorflow/tensorflow:1.15.2-py3",
packages_to_install=config_op_packages)
def output_config(config: training_reader.TrainEvalPipelineConfig) -> None:
print(config)
output_config_op = func_to_container_op(output_config)
#dsl.pipeline(
name='Post Training Processing',
description='Building the post-processing pipeline'
)
def ssd_postprocessing_pipeline(
working_bucket: str,
working_directory: str,
config_file:str):
config = training_config_op(working_bucket, working_directory, config_file)
output_config_op(config.output)
pipeline_name = ssd_postprocessing_pipeline.__name__ + '.zip'
compiler.Compiler().compile(ssd_postprocessing_pipeline, pipeline_name)
The https://storage.cloud.google.com/my_bucket/packages/training-reader-0.1.tar.gz IRL requires authentication. Try to download it in Incognito mode and you'll see the login page instead of file.
Changing the URL to https://storage.googleapis.com/my_bucket/packages/training-reader-0.1.tar.gz works for public objects, but your object is not public.
The only thing you can do (if you cannot make the package public) is to use google.cloud.storage library or gsutil program to download the file from the bucket and then manually install it suing subprocess.run([sys.executable, '-m', 'pip', 'install', ...])
Where are you downloading the data from?
What's the purpose of
pipeline_config = training_reader.TrainEvalPipelineConfig()
with open("ssd.config", 'r') as f:
text_format.Merge(f.read(), pipeline_config)
return pipeline_config
Why not just do the following:
def get_training_config(
working_bucket: str,
working_directory: str,
config_file: str,
output_config_path: OutputFile('TrainEvalPipelineConfig'),
):
download_file(working_bucket, os.path.join(working_directoy, config_file), output_config_path)
the way kubeflow installs dependencies.
Export your component to loadable component.yaml and you'll see how KFP Lighweight components install dependencies:
training_config_op = func_to_container_op(
get_training_config,
base_image="tensorflow/tensorflow:1.15.2-py3",
packages_to_install=config_op_packages,
output_component_file='component.yaml',
)
P.S. Some small pieces of info:
#dsl.pipeline(
Not required unless you want to use the dsl-compile command-line program
pipeline_name = ssd_postprocessing_pipeline.name + '.zip'
compiler.Compiler().compile(ssd_postprocessing_pipeline, pipeline_name)
Did you know that you can just kfp.Client(host=...).create_run_from_pipeline_func(ssd_postprocessing_pipeline, arguments={}) to run the pipeline right away?

Using electron-boilerplate to create an .exe for windows. It needs to run a .bat file. Once it's packaged, it doesn't run

Using the electron-boilerplate to create an .exe for windows, it needs to run a .bat file. However, using npm start it works but when it gets packaged with npm run release, it doesn't run the .bat
This is my code for the function
const spawn = require('child_process').spawn;
const bat = spawn('cmd.exe', ['/c', 'Install.bat']);
bat.stdout.on('data', (data) => {
var str = String.fromCharCode.apply(null, data);
addLog(data);
console.info(str);
});
bat.stderr.on('data', (data) => {
var str = String.fromCharCode.apply(null, data);
addLog(data,"error");
console.error(str);
});
bat.on('exit', (code) => {
console.log(`Exit ${code}`);
});
Already checked for child-process
When you run electron via npm start it will typically set the current working directory to the folder for the app (containing your package.json). So it will look for cmd.exe in that folder.
After you build the app and run it, the current working directory might be somewhere else, for example C:\\ (on Windows). You can find the current working directory with process.cwd().
To find the app folder regardless of how the app is running, Electron provides electron.app.getAppPath().
So you can use it like this:
const path = require('path');
const cmdPath = path.join(electron.app.getAppPath(),'cmd.exe');
const bat = spawn(cmdPath, ['/c', 'Install.bat']);

Dart: How to Implement and install a simple HTTP server

I have been able to run the dart-by-example http-server Hello web server.
The websocket uses port 9223 and the http server uses 8080.
After I do a build I do not find the server side code in build/bin.
What do I do next to install everything so that I could try run ws_server.dart and connect from my browser?
My Editor organization:
Server
packages
bin
packages
fireimager_server.dart (server side websocket handling code)
build
bin
web
lib
communication.dart (common code)
web
index.html
main.css
main.dart
WebsocketClient.dart (client websocket code)
If you follow the package layout convention, you server.dart should be in $PROJECT/bin and your web stuff in $PROJECT/web.
By running pub build you should get a new directory $PROJECT/build/web. Now you can use the following server.dart code to expose this build directory :
library simple_http_server;
import 'dart:io';
import 'package:http_server/http_server.dart' show VirtualDirectory;
void main() {
final MY_HTTP_ROOT_PATH = Platform.script.resolve('../build/web').toFilePath();
final virDir = new VirtualDirectory(MY_HTTP_ROOT_PATH)
..allowDirectoryListing = true;
HttpServer.bind(InternetAddress.LOOPBACK_IP_V4, 8080).then((server) {
server.listen((request) {
virDir.serveRequest(request);
});
});
}

how to deploy your dart app (using Web ui) without using Pub Deploy

What is the best strategy to deploy a Dart Web-ui app manually ?
pub deploy doesn't work for me and I have raised bug report. So am thinking what is the best way to manually deploy.
This is how I started:
1) From project root I compile the webui components (dwc.dart)
2) change directory to web/out then run dart2js
3) copy all .js files into that scripts/js public folder on server
4) copy appname.html to server changing css and script paths to option 3
5) Make sure dart.js is also in the same directory as item 3
this is as far as I got. So what else do I need to do ?
A few questions:
1) Do I manually change the file paths in the generated .js files to point to public folders on server for the files they are referencing and make sure those files are on server also ?
2) Do I need to copy all packages to server also ?
3) Any preferred file structure on server?
Any tips on this really appreciated.
Thanks.
I wrote a Grunt script for it (since I had no time to look up how to properly write code for Grunt, I did not share the code since it's a mess) but I basically do this:
compiling a list of files with dwc to a given out dir
compile it to javascript
clean up all non-deployable files
change some paths inside the HTML to match the server paths (for some reasons, this gets changed by the compilation process)
remove all packages except the ones I really need (JS interopt and browser)
Since I'm only using the JS version, I remove all dart packages. Since the paths inside the HTML files are up to you, you can already use a structure that suits you/your server.
I can provide you with a Grunt script to understand the order of tasks. Practically the order I use is this one:
Create the build directory. I usually use /build/web. I usually create these files (index.html, main.dart, /css and so on into the /web dir). I create the rest of components into /lib directory.
Compile the .dart file that contains the main() function ("main.dart" in my case for simpler projects) file to Javascript and put it into /build/web directory
Copy the other needed files and folders to the /build/web directory. Also, during this process you'll be copying the packages that your project needs. You'll see in the example provided below.
Remove all empty folders from the project
You can create a Grunt task to open the /index.html file in the browser once the building process has ended (I will not provide this example)
The structure of the dart test project:
testApp
- gruntfile.js
- package.js
/lib
/packages
/angular
/web
- index.html
- main.dart
/css
/img
So, the Grunt example script to cover steps from 1 - 4 looks like this (copy it to gruntfile.js):
module.exports = function (grunt) {
grunt.initConfig({
// 1.
// create build web directory
mkdir: {
build: {
options: {
create: ['build/web']
}
}
},
// 2.
// compile dart files
dart2js: {
options: {
// use this to fix a problem into dart2js node module. The module calls dart2js not dart2js.bat.
// this is needed for Windows. So use the path to your dart2js.bat file
"dart2js_bin": "C:/dart/dart-sdk/bin/dart2js.bat"
},
compile: {
files: {'build/web/main.dart.js': 'web/main.dart'}
}
},
// 3.
// copy all needed files, including all needed packages
// except the .dart files.
copy: {
build: {
files: [
{
expand: true,
src: [
'web/!(*.dart)',
'web/css/*.css',
'web/res/*.svg',
'web/packages/angular/**/!(*.dart)',
'web/packages/browser/**/!(*.dart)'
],
dest: 'build'
}
]
}
},
// 4.
// remove empty directories copied using the previous task
cleanempty: {
build: {
options: {
files: false
},
src: ['build/web/packages/**/*']
}
},
});
require('matchdep').filterDev('grunt-*').forEach(grunt.loadNpmTasks);
grunt.registerTask('default', [
'mkdir:build',
'dart2js',
'copy:build',
'cleanempty:build'
]);
};
So this is the Grunt script example.
Create a /gruntfile.js file into your project's root directory and copy/paste the script to it.
Create a /package.json file into your project's root directory and copy/paste the following script:
{
"name": "testApp",
"version": "0.0.1",
"description": "SomeDescriptionForTheTestApp",
"main": "",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "YourName",
"peerDependencies": {
"grunt-cli": "^0.1.13"
},
"devDependencies": {
"grunt": "^0.4.5",
"grunt-cleanempty": "^1.0.3",
"grunt-contrib-copy": "^0.7.0",
"grunt-dart2js": "0.0.5",
"grunt-mkdir": "^0.1.2",
"matchdep": "^0.3.0"
}
}
Open Command Prompt in Windows, Terminal in Linux, navigate to your project's root directory and use this command:
npm install
Wait untill all Grunt modules needed will be downloaded to your local project. Once this is finished, issue this command in Command Prompt or Terminal:
node -e "require('grunt').cli()"
You can use this to initiate Grunt default task without having Grunt installed globally on your system.
Now, to know the exact build structure for your project (including the packages that the project needs), make a build using Pub Build. Then you will be able to instruct Grunt to create the same dir structure.
You can add other tasks (like minification) if you want.
Hope this will help you all to understand the process and get you started with a test app first. Add your comments to make this even better and simplify it even more.

Resources