Here is the Makefile: https://github.com/somersbmatthews/vault/blob/master/Makefile
Here is what happens when I run it:
somersbmatthews#pop-os:~/go/src/vault$ make static-dist dev-ui
--> Installing JavaScript assets
yarn install v1.19.1
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
Done in 0.75s.
> node-sass#4.14.1 install /home/somersbmatthews/go/src/vault/ui/node_modules/node-sass
> node scripts/install.js
node-sass build Binary found at /home/somersbmatthews/go/src/vault/ui/node_modules/node-sass/vendor/linux-x64-64/binding.node
> node-sass#4.14.1 postinstall /home/somersbmatthews/go/src/vault/ui/node_modules/node-sass
> node scripts/build.js
Binary found at /home/somersbmatthews/go/src/vault/ui/node_modules/node-sass/vendor/linux-x64-64/binding.node
Testing binary
Binary is fine
node-sass#4.14.1 /home/somersbmatthews/go/src/vault/ui/node_modules/node-sass
--> Building Ember application
yarn run v1.19.1
$ ember build -prod
INFORMATION (ember-cli-pretender)
ember-auto-import seems to be in your package dependencies.
As a result, you don't need pretender to be wrapped anymore.
You can install pretender and remove ember-cli-pretender.
⠋ BuildingWARNING: Option "nodeWorker" is deprecated since workerpool#5.0.0. Please use "workerType" instead.
WARNING: Option "nodeWorker" is deprecated since workerpool#5.0.0. Please use "workerType" instead.
WARNING: Option "nodeWorker" is deprecated since workerpool#5.0.0. Please use "workerType" instead.
Environment: production
⠏ BuildingThe 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten
⠦ Building'#ember/string' is imported by ../../../../../../tmp/broccoli-607060b8WPADOlU6j8/cache-260-rollup/build/-private/system/normalize-model-name.js, but could not be resolved – treating it as an external dependency
'#ember/string' is imported by ../../../../../../tmp/broccoli-607060b8WPADOlU6j8/cache-260-rollup/build/-private/adapters/build-url-mixin.js, but could not be resolved – treating it as an external dependency
'#ember/string' is imported by ../../../../../../tmp/broccoli-607060b8WPADOlU6j8/cache-260-rollup/build/-private/system/debug/debug-adapter.js, but could not be resolved – treating it as an external dependency
⠏ Building[BABEL] Note: The code generator has deoptimised the styling of /home/somersbmatthews/go/src/vault/ui/node_modules/swagger-ui-dist/swagger-ui-bundle.js as it exceeds the max of 500KB.
Generating files needed by Storybook
Parsing /tmp/broccoli-607060b8WPADOlU6j8/out-630-broccoli_merge_trees/index.html
Generating preview-head.html
Generating files needed by Storybook
Generating .env
cleaning up...
Built project successfully. Stored in "../pkg/web_ui".
File sizes:
- ../pkg/web_ui/assets/chunk.3.e73ac42f48b4e5ab3d48.js: 1.08 MB (316.69 KB gzipped)
- ../pkg/web_ui/assets/node-asset-manifest.js: 1.02 KB (445 B gzipped)
- ../pkg/web_ui/assets/vault-895816690cab246cbd3b9423defc2f53.css: 482.96 KB (56.99 KB gzipped)
- ../pkg/web_ui/assets/vault-b8afdc29f93ad91f89268835698b0711.js: 1.2 MB (185.17 KB gzipped)
- ../pkg/web_ui/assets/vendor-8381b7eebdb7ea85cb88b80f3029e0e8.css: 14.21 KB (3.66 KB gzipped)
- ../pkg/web_ui/assets/vendor-ded9c2047ac30c216b8015683667178a.js: 1.82 MB (457.27 KB gzipped)
- ../pkg/web_ui/ember-fetch/fetch-fastboot-38cfd9007f94f81f5a2bc13690efc343.js: 1020 B (562 B gzipped)
- ../pkg/web_ui/engines-dist/kmip/assets/engine-ce86d837f49968e27331ecc744f8288d.js: 68.55 KB (9.29 KB gzipped)
- ../pkg/web_ui/engines-dist/kmip/assets/engine-vendor-d41d8cd98f00b204e9800998ecf8427e.css: 0 B
- ../pkg/web_ui/engines-dist/kmip/assets/engine-vendor-d41d8cd98f00b204e9800998ecf8427e.js: 0 B
- ../pkg/web_ui/engines-dist/kmip/config/environment-0123205ae026fc9ed3e41f1d552270f8.js: 86 B (100 B gzipped)
- ../pkg/web_ui/engines-dist/open-api-explorer/assets/engine-83cdd1e87b4c1568b63b394b62e6e0c5.js: 27.16 KB (5.14 KB gzipped)
- ../pkg/web_ui/engines-dist/open-api-explorer/assets/engine-9dcfdf942f31c3caa1d6dfd57c3cc072.css: 3.38 KB (829 B gzipped)
- ../pkg/web_ui/engines-dist/open-api-explorer/assets/engine-vendor-6faadde6d1de73cd00d4f818f4f60c75.css: 149.46 KB (22.77 KB gzipped)
- ../pkg/web_ui/engines-dist/open-api-explorer/assets/engine-vendor-d41d8cd98f00b204e9800998ecf8427e.js: 0 B
- ../pkg/web_ui/engines-dist/open-api-explorer/config/environment-6da0fcce17b2031e2559754701e92d69.js: 194 B (170 B gzipped)
- ../pkg/web_ui/engines-dist/replication/assets/engine-52dc634acbe2629436188771450e81ba.js: 97.81 KB (15.78 KB gzipped)
- ../pkg/web_ui/engines-dist/replication/assets/engine-vendor-d41d8cd98f00b204e9800998ecf8427e.css: 0 B
- ../pkg/web_ui/engines-dist/replication/assets/engine-vendor-d41d8cd98f00b204e9800998ecf8427e.js: 0 B
- ../pkg/web_ui/engines-dist/replication/config/environment-fcc3a0f22bdfd265a50708864776440a.js: 100 B (104 B gzipped)
- ../pkg/web_ui/sw-registration-65dd6e15d4d40ce435383a9edaccfc03.js: 1.14 KB (616 B gzipped)
- ../pkg/web_ui/sw.js: 1.26 KB (675 B gzipped)
Done in 70.33s.
--> Generating static assets
make[1]: Entering directory '/home/somersbmatthews/go/src/vault'
goimports -w $(find . -name '*.go' | grep -v pb.go | grep -v vendor)
make[1]: Leaving directory '/home/somersbmatthews/go/src/vault'
==> Checking compiled UI assets...
==> Checking that build is using go version >= 1.14.7...
==> Using go version 1.15.2...
==> Removing old directory...
==> Building...
flag provided but not defined: -gcflags
Usage: gox [options] [packages]
Gox cross-compiles Go applications in parallel.
If no specific operating systes or architectures are specified, Gox
will build for all pairs supported by your version of Go.
Options:
-arch="" Space-separated list of architectures to build for
-build-toolchain Build cross-compilation toolchain
-ldflags="" Additional '-ldflags' value to pass to go build
-os="" Space-separated list of operating systems to build for
-osarch="" Space-separated list of os/arch pairs to build for
-output="foo" Output path template. See below for more info
-parallel=-1 Amount of parallelism, defaults to number of CPUs
-verbose Verbose mode
Output path template:
The output path for the compiled binaries is specified with the
"-output" flag. The value is a string that is a Go text template.
The default value is "{{.Dir}}_{{.OS}}_{{.Arch}}". The variables and
their values should be self-explanatory.
Platforms (OS/Arch):
The operating systems and architectures to cross-compile for may be
specified with the "-arch" and "-os" flags. These are space separated lists
of valid GOOS/GOARCH values to build for, respectively. You may prefix an
OS or Arch with "!" to negate and not build for that platform. If the list
is made up of only negations, then the negations will come from the default
list.
Additionally, the "-osarch" flag may be used to specify complete os/arch
pairs that should be built or ignored. The syntax for this is what you would
expect: "darwin/amd64" would be a valid osarch value. Multiple can be space
separated. An os/arch pair can begin with "!" to not build for that platform.
The "-osarch" flag has the highest precedent when determing whether to
build for a platform. If it is included in the "-osarch" list, it will be
built even if the specific os and arch is negated in "-os" and "-arch",
respectively.
make: *** [Makefile:39: dev-ui] Error 2
Here is the full repo: https://github.com/somersbmatthews/vault
Lines 38 and 39 in the Makefile are:
dev-ui: assetcheck prep
#CGO_ENABLED=$(CGO_ENABLED) BUILD_TAGS='$(BUILD_TAGS) ui' VAULT_DEV_BUILD=1 sh -c "'$(CURDIR)/scripts/build.sh'"
How do I get more information on this error? "Error 2" appears twice in the code in two files as errors for a MongoDB dependency:
https://github.com/somersbmatthews/vault/blob/master/vendor/go.mongodb.org/mongo-driver/x/mongo/driver/auth/internal/gssapi/sspi_wrapper.h
https://github.com/somersbmatthews/vault/blob/master/vendor/go.mongodb.org/mongo-driver/x/mongo/driver/auth/internal/gssapi/gss_wrapper.h
Thanks for any help :)
Problem solved as per https://groups.google.com/g/vault-tool/c/xyV7-FMHrEE?pli=1
use command make bootstrap to update gox and other go tools.
NOTE: previous versions of gox will make an empty file called bindata.go Just delete this.
Related
I am Arch GNU/Linux user who usually manages almost every package with pacman; I manage TeX and LaTeX-related things with tlmgr. I installed tlmgr from source.
I am writing paper. I would like to use bibliography.
When I tried latexmk -pdflua main.ltx:
Rc files read:
latexmkrc
Latexmk: This is Latexmk, John Collins, 20 November 2021, version: 4.76.
Latexmk: applying rule 'biber main'...
Rule 'biber main': The following rules & subrules became out-of-date:
'biber main'
------------
Run number 1 of rule 'biber main'
------------
------------
Running 'biber "main.bcf"'
------------
biber: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
biber main: Could not open biber log file for 'main'
Latexmk: Use the -f option to force complete processing,
unless error was exceeding maximum runs, or warnings treated as errors.
libcrypt.so* on my environment
$ pacman -Qo /usr/lib/libcrypt*
/usr/lib/libcrypto.so is owned by openssl 1.1.1.m-1
/usr/lib/libcrypto.so.1.1 is owned by openssl 1.1.1.m-1
/usr/lib/libcryptsetup.so is owned by cryptsetup 2.4.3-2
/usr/lib/libcryptsetup.so.12 is owned by cryptsetup 2.4.3-2
/usr/lib/libcryptsetup.so.12.7.0 is owned by cryptsetup 2.4.3-2
/usr/lib/libcrypt.so is owned by libxcrypt 4.4.28-1
/usr/lib/libcrypt.so.2 is owned by libxcrypt 4.4.28-1
/usr/lib/libcrypt.so.2.0.0 is owned by libxcrypt 4.4.28-1
What I tried else
I uninstalled and re-installed biber on tlmgr but did not work.
# ln -s /usr/lib/libcrypt.so /usr/lib/libcrypt.so.1
$ latexmkrc -pdflua main.ltx
Rc files read:
latexmkrc
Latexmk: This is Latexmk, John Collins, 20 November 2021, version: 4.76.
Latexmk: applying rule 'biber main'...
Rule 'biber main': The following rules & subrules became out-of-date:
'biber main'
------------
Run number 1 of rule 'biber main'
------------
------------
Running 'biber "main.bcf"'
------------
/tmp/par-716861/cache-0e6aa298f0c2e7a775de99938825b2d56bd2027f/biber: /usr/lib/libcrypt.so.1: version `GLIBC_2.2.5' not found (required by /tmp/par-716861/cache-0e6aa298f0c2e7a775de99938825b2d56bd2027f/biber)
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
biber main: Could not open biber log file for 'main'
Latexmk: Use the -f option to force complete processing,
unless error was exceeding maximum runs, or warnings treated as errors.
File source
latexmkrc:
$latex='lualatex %O -synctex=1 -interaction=nonstopmode %S';
#$bibtex='upbibtex %O %B';
$bibtex='biber %O %B';
$makeindex='upmendex %O -o %D %S';
$pdf_mode=3;
Install libxcrypt-compat from the Core (1) repository, as suggest in this answer.
This made my biber from TeX Live 2020 work again. The interesting question is if newer TeX distributions will require this package, too.
(1) Thanks to #samueldy for the hint that the package was moved from the AUR to Core.
Adding on to #Christoph90's answer, as of 25 May 2022 the libxcrypt-compat package is still required in TeX Live 2022 in order for biber to function on Manjaro 21.2.6. However, the package has moved to core/libxcrypt-compat, so install by doing
sudo pacman -S libxcrypt-compat
I'm trying to add some patches to the llvm Opam package, but I'm having issues testing it because it seems like running opam install . from the package root ignores the url section and doesn't download & decompress the source archive, thus failing when applying patches.
This is the opam file for reference:
opam-version: "2.0"
maintainer: "Kate <kit.ty.kate#disroot.org>"
authors: [
"whitequark <whitequark#whitequark.org>"
"The LLVM team"
]
license: "MIT"
doc: "http://llvm.moe/ocaml"
bug-reports: "http://llvm.org/bugs/"
dev-repo: "git+http://llvm.org/git/llvm.git"
homepage: "http://llvm.moe"
install: [
["bash" "-ex" "install.sh" "%{conf-llvm:config}%" lib "%{conf-cmake:cmd}%" make]
]
depends: [
"ocaml" {>= "4.00.0"}
"ctypes" {>= "0.4"}
"ounit" {with-test}
"ocamlfind" {build}
"conf-llvm" {build & = version}
"conf-python-2-7" {build}
"conf-cmake" {build}
]
patches: [
"fix-shared.patch"
]
synopsis: "The OCaml bindings distributed with LLVM"
description: "Note: LLVM should be installed first."
extra-files: [
["link-META.patch" "md5=ef4ebb8706be2ed402f31fc351d7dc75"]
["install.sh" "md5=683ec0478ee422a57dcd3716277b3ef3"]
["fix-shared.patch" "md5=dce86b1db352332968ceb6d042b408a8"]
["META.patch" "md5=1d0af08bab7a0f831f68849b6556e414"]
["add-buildfence-llvm.ml.patch" "md5=a3bc667bd2fc937ee51c3b9d33b8ad63"]
["add-buildfence-llvm.mli.patch" "md5=99c739d74deeb1b990fe63cf914fc479"]
["add-buildfence-llvm_ocaml.c.patch" "md5=a29282f2e1e435abff57cecfd269ccb9"]
]
url {
src: "https://github.com/llvm/llvm-project/releases/download/llvmorg-11.1.0/llvm-11.1.0.src.tar.xz"
checksum: "sha256=ce8508e318a01a63d4e8b3090ab2ded3c598a50258cc49e2625b9120d4c03ea5"
}
and this is the result of running opam install . -vvv on the package root:
Processing 1/1: [llvm.11.0.0: rsync]
+ /usr/bin/rsync "-rLptgoDrvc" "--exclude" ".git" "--exclude" "_darcs" "--exclude" ".hg" "--exclude" ".#*" "--exclude" "_opam*" "--delete" "--delete-excluded" "/home/frabert/opam-repository/packages/llvm/llvm.11.0.0/" "/home/frabert/.opam/4.11.1/.opam-switch/sources/llvm"
- sending incremental file list
- ./
- out
-
- sent 828 bytes received 39 bytes 1,734.00 bytes/sec
- total size is 19,120 speedup is 22.05
[llvm.11.0.0] synchronised from file:///home/frabert/opam-repository/packages/llvm/llvm.11.0.0
+ /usr/bin/lsb_release "-s" "-r"
- 18.04
+ /usr/bin/ocamlc "-vnum"
- 4.05.0
The following actions will be performed:
∗ install llvm 11.0.0*
+ /usr/bin/rsync "-rLptgoDrvc" "--exclude" ".git" "--exclude" "_darcs" "--exclude" ".hg" "--exclude" ".#*" "--exclude" "_opam*" "--delete" "--delete-excluded" "/home/frabert/opam-repository/packages/llvm/llvm.11.0.0/" "/home/frabert/.opam/4.11.1/.opam-switch/sources/llvm"
- sending incremental file list
- ./
- opam
- out
- files/
- files/META.patch
- files/add-buildfence-llvm.ml.patch
- files/add-buildfence-llvm.mli.patch
- files/add-buildfence-llvm_ocaml.c.patch
- files/fix-shared.patch
- files/install.sh
- files/link-META.patch
-
- sent 20,648 bytes received 202 bytes 41,700.00 bytes/sec
- total size is 19,775 speedup is 0.95
[llvm.11.0.0] synchronised from file:///home/frabert/opam-repository/packages/llvm/llvm.11.0.0
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
+ /bin/cp "-PRp" "/home/frabert/.opam/4.11.1/.opam-switch/sources/llvm" "/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0"
+ /bin/cp "/home/frabert/.opam/4.11.1/.opam-switch/overlay/llvm/files/link-META.patch" "/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0/link-META.patch"
+ /bin/cp "/home/frabert/.opam/4.11.1/.opam-switch/overlay/llvm/files/install.sh" "/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0/install.sh"
+ /bin/cp "/home/frabert/.opam/4.11.1/.opam-switch/overlay/llvm/files/fix-shared.patch" "/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0/fix-shared.patch"
+ /bin/cp "/home/frabert/.opam/4.11.1/.opam-switch/overlay/llvm/files/add-buildfence-llvm_ocaml.c.patch" "/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0/add-buildfence-llvm_ocaml.c.patch"
+ /bin/cp "/home/frabert/.opam/4.11.1/.opam-switch/overlay/llvm/files/add-buildfence-llvm.mli.patch" "/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0/add-buildfence-llvm.mli.patch"
+ /bin/cp "/home/frabert/.opam/4.11.1/.opam-switch/overlay/llvm/files/add-buildfence-llvm.ml.patch" "/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0/add-buildfence-llvm.ml.patch"
+ /bin/cp "/home/frabert/.opam/4.11.1/.opam-switch/overlay/llvm/files/META.patch" "/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0/META.patch"
+ /bin/cp "/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0/fix-shared.patch" "/home/frabert/.opam/log/processed-patch-13793-c743ac"
Processing 1/2: [llvm: patch]
+ /usr/bin/patch "-p1" "-i" "/home/frabert/.opam/log/processed-patch-13793-c743ac" (CWD=/home/frabert/.opam/4.11.1/.opam-switch/build/llvm.11.0.0)
- can't find file to patch at input line 5
- Perhaps you used the wrong -p or --strip option?
- The text leading up to this was:
- --------------------------
- |diff --git a/cmake/modules/AddOCaml.cmake b/cmake/modules/AddOCaml.cmake
- |index 554046b20..b27cbd36c 100644
- |--- a/cmake/modules/AddOCaml.cmake
- |+++ b/cmake/modules/AddOCaml.cmake
- --------------------------
- File to patch:
- Skip this patch? [y]
- Skipping patch.
- 1 out of 1 hunk ignored
<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build llvm 11.0.0
└─
╶─ No changes have been performed
Is this a known problem?
EDIT: Clarification regarding the workflow
I have a local git clone of the opam-repository, of which I have edited and committed the llvm.11.0.0 package definition.
To test the edits, I run opam install . from inside the llvm.11.0.0 directory which contains the opam file.
The correct1 workflow for changing a package definition in the ocaml/opam-repository is the following.
clone the opam-repository
git clone git#github.com:ocaml/opam-repository.git
add the local repository to the opam repo list
opam repo add local ./opam-repository
make a copy of the package that you would like to change (using llvm as the working example), we will use -<patch> since we're just adding a patch, not releasing a new version,
cd opam-repository/packages/llvm/
cp -r llvm.11.0.0/ llvm.11.0.0-1
work with the patched version ...
commit the work
git add llvm.11.0.0-1
git commit -m 'wip'
test it
opam update
opam install llvm
1) Well, at least it is the workflow that I am using everyday :)
I was using the wrong workflow, as hinted correctly by #ivg
The correct one, though, appears to be the one described here: https://github.com/ocaml/opam/issues/4654
Basically, I needed to add a local repository, and then install the llvm package as usual.
opam repo add local ~/opam-repository
opam install llvm
I have used spiffsimg to create a single file containing multiple lua files:
# ./spiffsimg -f lua.img -c 262144 -r lua.script
f 4227 init.lua
f 413 cfg.lua
f 2233 setupWifi.lua
f 7498 configServer.lua
f 558 cfgForm.htm
f 4255 setupConfig.lua
f 14192 main.lua
#
I then use esptool.py to flash the NodeMCU firmware and the file containing the lua files to the esp8266 (NodeMCU dev kit):
c:\esptool-master>c:\Python27\python esptool.py -p COM7 write_flash -fs 32m -fm dio 0x00000 nodemcu-dev-9-modules-2016-07-18-12-06-36-integer.bin 0x78000 lua.img
esptool.py v1.0.2-dev
Connecting...
Running Cesanta flasher stub...
Flash params set to 0x0240
Writing 446464 # 0x0... 446464 (100 %)
Wrote 446464 bytes at 0x0 in 38.9 seconds (91.9 kbit/s)...
Writing 262144 # 0x78000... 262144 (100 %)
Wrote 262144 bytes at 0x78000 in 22.8 seconds (91.9 kbit/s)...
Leaving...
I then run ESPLorer to check the status and get:
PORT OPEN 115200
Communication with MCU..Got answer! AutoDetect firmware...
Can't autodetect firmware, because proper answer not received.
NodeMCU custom build by frightanic.com
branch: dev
commit: b21b3e08aad633ccfd5fd29066400a06bb699ae2
SSL: true
modules: file,gpio,http,net,node,rtctime,tmr,uart,wifi
build built on: 2016-07-18 12:05
powered by Lua 5.1.4 on SDK 1.5.4(baaeaebb)
lua: cannot open init.lua
>
----------------------------
No files found.
----------------------------
>
Total : 3455015 bytes
Used : 0 bytes
Remain: 3455015 bytes
The NodeMCU firmware flashed correctly, but the lua files can't be located.
I have tried flashing to other locations (0x84000, 0x7c000), but I am just guessing at these locations based on reading threads on github.
I used the NodeMCU file.fscfg() routine to get the flash address and size. If I only flash the NodeMCU firmware I get the following:
print (file.fscfg())
524288 3653632
534288 is 0x80000, so I tried flashing only the spiffsimg file (lua.img) to 0x8000, then ran the same print statement and got:
print (file.fscfg())
786432 3391488
The flash address incremented by the exact number of bytes in the lua.img - which I don't understand, why would the flash address change? Is the first number returned by file.fscfg not the starting flash address, but the ending flash address?
What is the correct address for flashing an image file, contain lua files, that was created by spiffsimg?
The version of spiffsimg found here will provide the correct address for flashing an image file that contains lua files.
Do not use this version of spiffsimg as it is out of date.
To install the spiffsimg utility, you need to download and install the entire nodemcu-firmware package (into a linux environment, use make to install - note: make on my debian linux box generated an error, but i was able to go to the ../tools/spiffsimg subdirectory and run make on the Makefile found in that directory to create the utility).
The spiffsimg instructions found here are quite clear, with one exception: the file name you specify, with the -f parameter, needs to include the characters %x. The %x will be replaced with the address that the image file should be flashed to.
For example, the command
spiffsimage -f %x-luaFiles.img -S 4MB -U 465783 -r lua.script
will create a file, in the local directory, with a name like: 80000-luaFiles.img. Which means you should install that image file at address 0x80000 on the ESP8266.
I've never done that myself but I'm reasonably confident the correct answer can be extracted from the docs.
-f specifies the filename for the disk image. '%x' will be replaced
by the calculated offset of the file system.
And a bit further down
The disk image file is placed into the bin directory and it is named
0x<offset>-<size>.bin where the offset is the location where it
should be flashed, and the size is the size of the flash part.
However, there's a slight mismatch between the two statements. We may have a bug in the docs. If "'%x' will be replaced..." then I'd expected the final name won't contain 0x anymore.
Furthermore, it is possible to define a fixed SPIFFS position when you build the firmware.
#define SPIFFS_FIXED_LOCATION 0x100000
This specifies that the SPIFFS filesystem starts at 1Mb from the start of the flash. Unless
otherwise specified, it will run to the end of the flash (excluding
the 16k of space reserved by the SDK).
How we can generate FortiFy report using command ??? on linux.
In command, how we can include only some folders or files for analyzing and how we can give the location to store the report. etc.
Please help....
Thanks,
Karthik
1. Step#1 (clean cache)
you need to plan scan structure before starting:
scanid = 9999 (can be anything you like)
ProjectRoot = /local/proj/9999/
WorkingDirectory = /local/proj/9999/working
(this dir is huge, you need to "rm -rf ./working && mkdir ./working" before every scan, or byte code piles underneath this dir and consume your harddisk fast)
log = /local/proj/9999/working/sca.log
source='/local/proj/9999/source/src/**.*'
classpath='local/proj/9999/source/WEB-INF/lib/*.jar; /local/proj/9999/source/jars/**.*; /local/proj/9999/source/classes/**.*'
./sourceanalyzer -b 9999 -Dcom.fortify.sca.ProjectRoot=/local/proj/9999/ -Dcom.fortify.WorkingDirectory=/local/proj/9999/working -logfile /local/proj/working/9999/working/sca.log -clean
It is important to specify ProjectRoot, if not overwrite this system default, it will put under your /home/user.fortify
sca.log location is very important, if fortify does not find this file, it cannot find byte code to scan.
You can alter the ProjectRoot and Working Directory once for all if your are the only user: FORTIFY_HOME/Core/config/fortify_sca.properties).
In such case, your command line would be ./sourceanalyzer -b 9999 -clean
2. Step#2 (translate source code to byte code)
nohup ./sourceanalyzer -b 9999 -verbose -64 -Xmx8000M -Xss24M -XX:MaxPermSize=128M -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+UseParallelGC -Dcom.fortify.sca.ProjectRoot=/local/proj/9999/ -Dcom.fortify.WorkingDirectory=/local/proj/9999/working -logfile /local/proj/9999/sca.log -source 1.5 -classpath '/local/proj/9999/source/WEB-INF/lib/*.jar:/local/proj/9999/source/jars/**/*.jar:/local/proj/9999/source/classes/**/*.class' -extdirs '/local/proj/9999/source/wars/*.war' '/local/proj/9999/source/src/**/*' &
always unix background job (&) in case your session to server is timeout, it will keep working.
cp : put all your known classpath here for fortify to resolve the functiodfn calls. If function not found, fortify will skip the source code translation, so this part will not be scanned later. You will get a poor scan quality but FPR looks good (low issue reported). It is important to have all dependency jars in place.
-extdir: put all directories/files you don't want to be scanned here.
the last section, files between ' ' are your source.
-64 is to use 64-bit java, if not specified, 32-bit will be used and the max heap should be <1.3 GB (-Xmx1200M is safe).
-XX: are the same meaning as in launch application server. only use these to control the class heap and garbage collection. This is to tweak performance.
-source is java version (1.5 to 1.8)
3. Step#3 (scan with rulepack, custom rules, filters, etc)
nohup ./sourceanalyzer -b 9999 -64 -Xmx8000M -Dcom.fortify.sca.ProjectRoot=/local/proj/9999 -Dcom.fortify.WorkingDirectory=/local/proj/9999/working -logfile /local/ssap/proj/9999/working/sca.log **-scan** -filter '/local/other/filter.txt' -rules '/local/other/custom/*.xml -f '/local/proj/9999.fpr' &
-filter: file name must be filter.txt, any ruleguid in this file will not be reported.
rules: this is the custom rule you wrote. the HP rulepack is in FORTIFY_HOME/Core/config/rules directory
-scan : keyword to tell fortify engine to scan existing scanid. You can skip step#2 and only do step#3 if you did notchange code, just want to play with different filter/custom rules
4. Step#4 Generate PDF from the FPR file (if required)
./ReportGenerator -format pdf -f '/local/proj/9999.pdf' -source '/local/proj/9999.fpr'
[See edit, it seems the extra size comes from debugging symbols added at linking time, but the reason why this happens is still unclear!]
I am cross compiling OpenCV 2.4.11 Ubuntu x86 64bit -> armeabi.
I am using the toolchain available here https://developer.android.com/tools/sdk/ndk/index.html, choosing the 4.9 compiler.
When I compile the dynamic libraries, they get considerably bigger than the static library. Examples:
3793082 Mar 12 17:21 libopencv_core.a
6131716 Mar 12 17:29 libopencv_core.so
446060 Mar 12 17:22 libopencv_highgui.a
5510352 Mar 12 17:30 libopencv_highgui.so
3477794 Mar 12 17:21 libopencv_imgproc.a
5325504 Mar 12 17:29 libopencv_imgproc.so
38004 Mar 12 17:19 libopencv_info.so
844990 Mar 12 17:21 libopencv_ml.a
3827136 Mar 12 17:29 libopencv_ml.so
747744 Mar 12 17:22 libopencv_objdetect.a
2370188 Mar 12 17:30 libopencv_objdetect.so
405920 Mar 12 17:22 libopencv_video.a
2196268 Mar 12 17:30 libopencv_video.so
For the static library the size corresponds more or less to the total size of the object files. Example for core and highgui.
du -chs `find -iname \*.o|grep opencv_core.dir`
[...]
3,5M total
du -chs `find -iname \*.o|grep opencv_highgui.dir`
[...]
352K total
The same happens if I build with make or ninja.
There is just a small difference in the compiler flags at build time, but if I check the object files generated for the static and the dynamic build, they have exactly the same size. That's the command I use to generate such a list:
ls -s `find -iname \*.o`|grep core
So, I thought, it must something in the linking phase. I took a look at the build.ninja file differences, and these are lines present only for the shared version:
LINK_FLAGS = -llog -Wl,--fix-cortex-a8 -Wl,--no-undefined -Wl,--gc-sections -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now
LINK_LIBRARIES = lib/libopencv_features2d.so -ldl -lm -llog -ltbb lib/libopencv_flann.so lib/libopencv_highgui.so lib/libopencv_imgproc.so lib/libopencv_core.so
I do not think the additional libraries linked (dl, m, log, tbb) influence the final size, as they are all much smaller than the difference I found. Furthermore, I started to verify and for log there's only the .so available, and for tbb (100Kb) I have both shared and static version. BTW, I tried to build also without tbb.
To be 100% sure, I took the actual command line that was linking the object file, removed the -no-undefined option, and then removed all other options and linked libraries. The file size did not change, apart from when removing -Wl,--gc-sections which caused the file size to increase (it's some garbage collection option).
So, the only option that is left is a... linker bug?!? Does anybody have any idea what is happening?
Some additional information:
Compiler details:
./arm-linux-androideabi-gcc -v
Using built-in specs.
COLLECT_GCC=./arm-linux-androideabi-gcc
COLLECT_LTO_WRAPPER=/opt/toolchain-arm17/bin/../libexec/gcc/arm-linux-androideabi/4.9/lto-wrapper
Target: arm-linux-androideabi
Configured with: /s/ndk-toolchain/src/build/../gcc/gcc-4.9/configure --prefix=/tmp/ndk-andrewhsieh/build/toolchain/prefix --target=arm-linux-androideabi --host=x86_64-linux-gnu --build=x86_64-linux-gnu --with-gnu-as --with-gnu-ld --enable-languages=c,c++ --with-gmp=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-mpfr=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-mpc=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-cloog=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-isl=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-ppl=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --disable-ppl-version-check --disable-cloog-version-check --disable-isl-version-check --enable-cloog-backend=isl --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --disable-libssp --enable-threads --disable-nls --disable-libmudflap --disable-libgomp --disable-libstdc__-v3 --disable-sjlj-exceptions --disable-shared --disable-tls --disable-libitm --with-float=soft --with-fpu=vfp --with-arch=armv5te --enable-target-optspace --enable-initfini-array --disable-nls --prefix=/tmp/ndk-andrewhsieh/build/toolchain/prefix --with-sysroot=/tmp/ndk-andrewhsieh/build/toolchain/prefix/sysroot --with-binutils-version=2.24 --with-mpfr-version=3.1.1 --with-mpc-version=1.0.1 --with-gmp-version=5.0.5 --with-gcc-version=4.9 --with-gdb-version=7.6 --with-python=/usr/local/google/home/andrewhsieh/mydroid/ndk/prebuilt/linux-x86_64/bin/python-config.sh --with-gxx-include-dir=/tmp/ndk-andrewhsieh/build/toolchain/prefix/include/c++/4.9 --with-bugurl=http://source.android.com/source/report-bugs.html --enable-languages=c,c++ --disable-bootstrap --enable-plugins --enable-libgomp --disable-libsanitizer --enable-gold --enable-graphite=yes --with-cloog-version=0.18.0 --with-isl-version=0.11.1 --enable-eh-frame-hdr-for-static --with-arch=armv5te --program-transform-name='s&^&arm-linux-androideabi-&' --enable-gold=default
Thread model: posix
gcc version 4.9 20140827 (prerelease) (GCC)
I also tried to see what would happen trying another version of the linker, but no changes in size
arm-linux-androideabi-g++.exe -v
Using built-in specs.
COLLECT_GCC=arm-linux-androideabi-g++.exe
COLLECT_LTO_WRAPPER=lto-wrapper.exe
Target: arm-linux-androideabi
Configured with: /s/ndk-toolchain/src/build/../gcc/gcc-4.8/configure --prefix=/tmp/ndk-andrewhsieh/build/toolchain/prefix --target=arm-linux-androideabi --host=x86_64-pc-mingw32msvc --build=x86_64-lin
ux-gnu --with-gnu-as --with-gnu-ld --enable-languages=c,c++ --with-gmp=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-mpfr=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-mpc=/tmp/n
dk-andrewhsieh/build/toolchain/temp-install --with-cloog=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-isl=/tmp/ndk-andrewhsieh/build/toolchain/temp-install --with-ppl=/tmp/ndk-andrewhsieh/
build/toolchain/temp-install --disable-ppl-version-check --disable-cloog-version-check --disable-isl-version-check --enable-cloog-backend=isl --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc+
+,-Bdynamic -lm' --disable-libssp --enable-threads --disable-nls --disable-libmudflap --disable-libgomp --disable-libstdc__-v3 --disable-sjlj-exceptions --disable-shared --disable-tls --disable-libitm
--with-float=soft --with-fpu=vfp --with-arch=armv5te --enable-target-optspace --enable-initfini-array --disable-nls --prefix=/tmp/ndk-andrewhsieh/build/toolchain/prefix --with-sysroot=/tmp/ndk-andrew
hsieh/build/toolchain/prefix/sysroot --with-binutils-version=2.24 --with-mpfr-version=3.1.1 --with-mpc-version=1.0.1 --with-gmp-version=5.0.5 --with-gcc-version=4.8 --with-gdb-version=7.6 --with-pytho
n=/usr/local/google/home/andrewhsieh/mydroid/ndk/prebuilt/windows-x86_64/bin/python-config.sh --with-gxx-include-dir=/tmp/ndk-andrewhsieh/build/toolchain/prefix/include/c++/4.8 --with-bugurl=http://so
urce.android.com/source/report-bugs.html --enable-languages=c,c++ --disable-bootstrap --enable-plugins --enable-libgomp --disable-libsanitizer --enable-gold --enable-graphite=yes --with-cloog-version=
0.18.0 --with-isl-version=0.11.1 --enable-eh-frame-hdr-for-static --with-arch=armv5te --program-transform-name='s&^&arm-linux-androideabi-&' --enable-gold=default
Thread model: posix
gcc version 4.8 (GCC)
EDIT:
As suggested by MarcB, I tried to strip the library. The result is suprising (to me :) )
$ arm-linux-androideabi/bin/strip -g libopencv_core.so -o libopencv_core_stripped.so
$ ls -la *core*
6293308 Mar 13 10:40 libopencv_core.so
3224840 Mar 13 12:18 libopencv_core_stripped.so
Where did all those debug symbols came out if the object files where compiled without -g (or even with -g0)?
Note: the library stripped like that seem to be fully functional. The nm -D output of the stripped/unstripped library are the same, and nm output is just a few lines smaller (like 50 lines less out of 12000).
Just to be sure, I tried also to strip objects before linking, but their file does not change (it increases just a bit), and linking the "stripped" object files produces a library of the same big size of before.
Those are not debug symbols. They are regular linker symbols.
A shared library may have two sets of symbols: one is for linking, the other one is for dynamic loading. strip removes the first kind of symbols. You cannot link with a stripped shared library, but you can load it at run time normally (e.g if you use dlopen, or link with the library then strip it).
See nm yourlib.so and nm -D yourlib.so both before and after running strip.
CORRECTION it is possible to link with a stripped library. A good explanation about the two kinds of symbol tables is here.