I'm trying to build and run Go on a self-hosted jailbroken iOS device (Corellium virtual devices). Go bootstraps and can build and run regular binaries. However, Go archives fail to build because ar doesn't work:
$ dpkg -S /usr/bin/clang /usr/bin/ar /usr/bin/libtool
org.coolstar.llvm-clang64: /usr/bin/clang
org.coolstar.cctools: /usr/bin/ar
org.coolstar.cctools: /usr/bin/libtool
$ apt policy org.coolstar.cctools org.coolstar.llvm-clang64
org.coolstar.cctools:
Installed: 895
Candidate: 895
Version table:
*** 895 500
500 http://apt.thebigboss.org/repofiles/cydia stable/main iphoneos-arm Packages
100 /Library/dpkg/status
org.coolstar.llvm-clang64:
Installed: 5.0.1-2
Candidate: 5.0.1-2
Version table:
*** 5.0.1-2 500
500 http://apt.thebigboss.org/repofiles/cydia stable/main iphoneos-arm Packages
100 /Library/dpkg/status
$ ../bin/go build -buildmode=c-archive -ldflags="-v" -work ../misc/ios/detect.go
WORK=/tmp/go-build734936238
# command-line-arguments
HEADER = -H1 -T0x2000 -R0x1000
0.00 deadcode
0.05 symsize = 0
0.08 pclntab=740717 bytes, funcdata total 185492 bytes
0.08 dodata
0.08 symsize = 0
0.08 symsize = 0
0.09 dynreloc
0.09 dwarf
0.12 asmb
0.13 datblk
0.14 reloc
0.20 sym
0.22 header
archive: ar -q -c -s $WORK/b001/exe/a.out.a /tmp/go-link-569975294/go.o /tmp/go-link-569975294/000000.o /tmp/go-link-569975294/000001.o /tmp/go-link-569975294/000002.o /tmp/go-link-569975294/000003.o /tmp/go-link-569975294/000004.o /tmp/go-link-569975294/000005.o /tmp/go-link-569975294/000006.o /tmp/go-link-569975294/000007.o /tmp/go-link-569975294/000008.o /tmp/go-link-569975294/000009.o /tmp/go-link-569975294/000010.o /tmp/go-link-569975294/000011.o /tmp/go-link-569975294/000012.o /tmp/go-link-569975294/000013.o /tmp/go-link-569975294/000014.o
/var/mobile/go-tip/pkg/tool/darwin_arm64/link: running ar failed: exit status 1
fatal error: ar: can't find or exec: /usr/bin/arm64-apple-darwin14-ranlib (No such file or directory)
ar: internal ranlib command failed
Optimistic as I am, I tried symlinking ranlib:
# ln -s /usr/bin/ranlib /usr/bin/arm64-apple-darwin14-ranlib
$ ../bin/go build -buildmode=c-archive -ldflags="-v" -work ../misc/ios/detect.go
WORK=/tmp/go-build239621581
# command-line-arguments
HEADER = -H1 -T0x2000 -R0x1000
0.00 deadcode
0.06 symsize = 0
0.08 pclntab=740717 bytes, funcdata total 185492 bytes
0.08 dodata
0.09 symsize = 0
0.09 symsize = 0
0.09 dynreloc
0.10 dwarf
0.12 asmb
0.13 datblk
0.15 reloc
0.20 sym
0.22 header
archive: ar -q -c -s $WORK/b001/exe/a.out.a /tmp/go-link-618780113/go.o /tmp/go-link-618780113/000000.o /tmp/go-link-618780113/000001.o /tmp/go-link-618780113/000002.o /tmp/go-link-618780113/000003.o /tmp/go-link-618780113/000004.o /tmp/go-link-618780113/000005.o /tmp/go-link-618780113/000006.o /tmp/go-link-618780113/000007.o /tmp/go-link-618780113/000008.o /tmp/go-link-618780113/000009.o /tmp/go-link-618780113/000010.o /tmp/go-link-618780113/000011.o /tmp/go-link-618780113/000012.o /tmp/go-link-618780113/000013.o /tmp/go-link-618780113/000014.o
/var/mobile/go-tip/pkg/tool/darwin_arm64/link: running ar failed: exit status 1
/usr/bin/arm64-apple-darwin14-ranlib: archive member: $WORK/b001/exe/a.out.a(go.o) offset in archive not a multiple of 8 (must be since member is an 64-bit object file)
fatal error: ar: fatal error in /usr/bin/arm64-apple-darwin14-ranlib
I even switched Go to use libtool:
$ ../bin/go build -buildmode=c-archive -ldflags="-v" -work ../misc/ios/detect.go
WORK=/tmp/go-build502332895
# command-line-arguments
HEADER = -H1 -T0x2000 -R0x1000
0.00 deadcode
0.04 symsize = 0
0.08 pclntab=740717 bytes, funcdata total 185492 bytes
0.08 dodata
0.08 symsize = 0
0.09 symsize = 0
0.09 dynreloc
0.10 dwarf
0.12 asmb
0.14 datblk
0.15 reloc
0.20 sym
0.22 header
archive: libtool -static -o $WORK/b001/exe/a.out.a /tmp/go-link-320669019/go.o /tmp/go-link-320669019/000000.o /tmp/go-link-320669019/000001.o /tmp/go-link-320669019/000002.o /tmp/go-link-320669019/000003.o /tmp/go-link-320669019/000004.o /tmp/go-link-320669019/000005.o /tmp/go-link-320669019/000006.o /tmp/go-link-320669019/000007.o /tmp/go-link-320669019/000008.o /tmp/go-link-320669019/000009.o /tmp/go-link-320669019/000010.o /tmp/go-link-320669019/000011.o /tmp/go-link-320669019/000012.o /tmp/go-link-320669019/000013.o /tmp/go-link-320669019/000014.o
/var/mobile/go-tip/pkg/tool/darwin_arm64/link: running libtool failed: signal: segmentation fault
Now, ar (with symlinked ranlib) fails on even simple object files:
$ cat blah.c
int blah() {
return 0;
}
$ clang -c blah.c
$ ar -q -c -s blah.a blah.o
/usr/bin/arm64-apple-darwin14-ranlib: archive member: blah.a(blah.o) offset in archive not a multiple of 8 (must be since member is an 64-bit object file)
Libtool works:
$ libtool -static -o blah.a blah.o
But segfaults on more complex object files as seen above.
Go can already successfully create archives using the Xcode ar tool. It's only the iOS native "cctools" version that fails.
I worked around the problem by using llvm-ar. The command line flags are slightly different, but at least I now have a working archive.
Related
I found a related issue on github but it is under the msphpsql repo so I'm not sure if it applies to the driver in general for alpine. It says MS does not support yet.
Official MS installation page has installation instructions only for Debian, RedHat, SUSE, Ubuntu.
If there are any workarounds, please suggest.
If this is not possible, please elaborate your answer as technically as possible. I could not find a suitable answer for this question anywhere.
This is the Dockerfile I've tested.
FROM alpine
# Install dependencies
RUN apk --no-cache add curl gnupg
# Download the desired package(s)
RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.6.1.1-1_amd64.apk
RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/mssql-tools_17.6.1.1-1_amd64.apk
# (Optional) Verify signature, if 'gpg' is missing install it using 'apk add gnupg':
RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.6.1.1-1_amd64.sig
RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/mssql-tools_17.6.1.1-1_amd64.sig
RUN curl https://packages.microsoft.com/keys/microsoft.asc | gpg --import -
RUN gpg --verify msodbcsql17_17.6.1.1-1_amd64.sig msodbcsql17_17.6.1.1-1_amd64.apk
RUN gpg --verify mssql-tools_17.6.1.1-1_amd64.sig mssql-tools_17.6.1.1-1_amd64.apk
# Install the package(s)
RUN apk add --allow-untrusted msodbcsql17_17.6.1.1-1_amd64.apk
RUN apk add --allow-untrusted mssql-tools_17.6.1.1-1_amd64.apk
and this is the log I got,
Sending build context to Docker daemon 2.174MB
Step 1/11 : FROM alpine
---> a24bb4013296
Step 2/11 : RUN apk --no-cache add curl gnupg
---> Running in d4f7b3b86157
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/32) Installing ca-certificates (20191127-r4)
(2/32) Installing nghttp2-libs (1.41.0-r0)
(3/32) Installing libcurl (7.69.1-r0)
(4/32) Installing curl (7.69.1-r0)
(5/32) Installing libgpg-error (1.37-r0)
(6/32) Installing libassuan (2.5.3-r0)
(7/32) Installing libcap (2.27-r0)
(8/32) Installing libffi (3.3-r2)
(9/32) Installing libintl (0.20.2-r0)
(10/32) Installing libblkid (2.35.2-r0)
(11/32) Installing libmount (2.35.2-r0)
(12/32) Installing pcre (8.44-r0)
(13/32) Installing glib (2.64.4-r0)
(14/32) Installing ncurses-terminfo-base (6.2_p20200523-r0)
(15/32) Installing ncurses-libs (6.2_p20200523-r0)
(16/32) Installing libgcrypt (1.8.5-r0)
(17/32) Installing libsecret (0.20.3-r0)
(18/32) Installing pinentry (1.1.0-r2)
Executing pinentry-1.1.0-r2.post-install
(19/32) Installing libbz2 (1.0.8-r1)
(20/32) Installing gmp (6.2.0-r0)
(21/32) Installing nettle (3.5.1-r1)
(22/32) Installing p11-kit (0.23.20-r5)
(23/32) Installing libtasn1 (4.16.0-r1)
(24/32) Installing libunistring (0.9.10-r0)
(25/32) Installing gnutls (3.6.14-r0)
(26/32) Installing libksba (1.4.0-r0)
(27/32) Installing db (5.3.28-r1)
(28/32) Installing libsasl (2.1.27-r6)
(29/32) Installing libldap (2.4.50-r0)
(30/32) Installing npth (1.6-r0)
(31/32) Installing sqlite-libs (3.32.1-r0)
(32/32) Installing gnupg (2.2.20-r0)
Executing busybox-1.31.1-r16.trigger
Executing ca-certificates-20191127-r4.trigger
OK: 27 MiB in 46 packages
Removing intermediate container d4f7b3b86157
---> f874be62c59e
Step 3/11 : RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.6.1.1-1_amd64.apk
---> Running in 264bf790ce99
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 923k 100 923k 0 0 21.9M 0 --:--:-- --:--:-- --:--:-- 21.9M
Removing intermediate container 264bf790ce99
---> cb999aaac06f
Step 4/11 : RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/mssql-tools_17.6.1.1-1_amd64.apk
---> Running in 7fe887056e7e
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 304k 100 304k 0 0 8236k 0 --:--:-- --:--:-- --:--:-- 8236k
Removing intermediate container 7fe887056e7e
---> 07885ffff557
Step 5/11 : RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.6.1.1-1_amd64.sig
---> Running in efb39bbe31a7
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 481 100 481 0 0 17178 0 --:--:-- --:--:-- --:--:-- 17178
Removing intermediate container efb39bbe31a7
---> 0b47b4bdc30d
Step 6/11 : RUN curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/mssql-tools_17.6.1.1-1_amd64.sig
---> Running in 08a264662da9
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 481 100 481 0 0 16033 0 --:--:-- --:--:-- --:--:-- 16033
Removing intermediate container 08a264662da9
---> dad9d122c7ae
Step 7/11 : RUN curl https://packages.microsoft.com/keys/microsoft.asc | gpg --import -
---> Running in ab82306e09cf
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 983 100 983 0 0 1386 0 --:--:-- --:--:-- --:--:-- 1386
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key EB3E94ADBE1229CF: public key "Microsoft (Release signing) <gpgsecurity#microsoft.com>" imported
gpg: Total number processed: 1
gpg: imported: 1
Removing intermediate container ab82306e09cf
---> 3a524358b99f
Step 8/11 : RUN gpg --verify msodbcsql17_17.6.1.1-1_amd64.sig msodbcsql17_17.6.1.1-1_amd64.apk
---> Running in 0621d8db0ef8
gpg: Signature made Tue Jul 7 21:23:47 2020 UTC
gpg: using RSA key EB3E94ADBE1229CF
gpg: Good signature from "Microsoft (Release signing) <gpgsecurity#microsoft.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: BC52 8686 B50D 79E3 39D3 721C EB3E 94AD BE12 29CF
Removing intermediate container 0621d8db0ef8
---> 5e4b528817f0
Step 9/11 : RUN gpg --verify mssql-tools_17.6.1.1-1_amd64.sig mssql-tools_17.6.1.1-1_amd64.apk
---> Running in 8a51c0d21e7e
gpg: Signature made Mon Jul 6 18:59:24 2020 UTC
gpg: using RSA key EB3E94ADBE1229CF
gpg: Good signature from "Microsoft (Release signing) <gpgsecurity#microsoft.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: BC52 8686 B50D 79E3 39D3 721C EB3E 94AD BE12 29CF
Removing intermediate container 8a51c0d21e7e
---> a06ecc7c2031
Step 10/11 : RUN apk add --allow-untrusted msodbcsql17_17.6.1.1-1_amd64.apk
---> Running in 8f5c10bfb5ea
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/11) Installing krb5-conf (1.0-r2)
(2/11) Installing libcom_err (1.45.6-r0)
(3/11) Installing keyutils-libs (1.6.1-r1)
(4/11) Installing libverto (0.3.1-r1)
(5/11) Installing krb5-libs (1.18.2-r0)
(6/11) Installing libgcc (9.3.0-r2)
(7/11) Installing libstdc++ (9.3.0-r2)
(8/11) Installing openssl (1.1.1g-r0)
(9/11) Installing readline (8.0.4-r0)
(10/11) Installing unixodbc (2.3.7-r2)
(11/11) Installing msodbcsql17 (17.6.1.1)
Executing msodbcsql17-17.6.1.1.pre-install
Using this product constitutes agreement with the
End User License, which can be downloaded from
https://aka.ms/odbc17eula and found in
/usr/share/doc/msodbcsql17/LICENSE.txt .
If you do not agree with the terms, please uninstall
the product by typing 'apk del msodbcsql17'
Executing msodbcsql17-17.6.1.1.post-install
Executing busybox-1.31.1-r16.trigger
OK: 186 MiB in 57 packages
Removing intermediate container 8f5c10bfb5ea
---> 3e449c097c43
Step 11/11 : RUN apk add --allow-untrusted mssql-tools_17.6.1.1-1_amd64.apk
---> Running in 889ede754d73
(1/1) Installing mssql-tools (17.6.1.1)
Executing mssql-tools-17.6.1.1.pre-install
Using this product constitutes agreement with the
End User License, which can be downloaded from
http://go.microsoft.com/fwlink/?LinkId=746949 and found in
/usr/share/doc/mssql-tools/LICENSE.txt .
If you do not agree with the terms, please uninstall
the product by typing 'apk del mssql-tools'
OK: 340 MiB in 58 packages
Removing intermediate container 889ede754d73
---> 0141e9b50f20
Successfully built 0141e9b50f20
Successfully tagged microsoft_odbc_17_alpine:latest
I hope this will solve you problem. You can find all the versions such as Alpine Linux, Debian, Red Hat Enterprise Server and Oracle Linux, SUSE Linux Enterprise Server and Ubuntu as well as you can also find Previous versions installation details from Install the Microsoft ODBC driver for SQL Server (Linux) article.
When my Docker build reaches this part of the Dockerfile:
ARG cabal=1.22.9.0
RUN curl -L http://hackage.haskell.org/package/cabal-install-${cabal}/cabal-install-${cabal}.tar.gz | tar -xzf- && \
cd cabal-install-${cabal} && \
EXTRA_CONFIGURE_OPTS= ./bootstrap.sh --global
It eventually ends with this error.
Downloading zlib-0.6.1.1...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 139k 0 139k 0 0 293k 0 --:--:-- --:--:-- --:--:-- 293k
[1 of 1] Compiling Main ( Setup.hs, Setup.o )
Linking Setup ...
Configuring zlib-0.6.1.1...
Setup: Missing dependency on a foreign library:
* Missing (or bad) header file: zlib.h
* Missing C library: z
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
Error during cabal-install bootstrap:
Configuring the zlib package failed.
The command '/bin/sh -c curl -L http://hackage.haskell.org/package/cabal-install-${cabal}/cabal-install-${cabal}.tar.gz | tar -xzf- && cd cabal-install-${cabal} && EXTRA_CONFIGURE_OPTS= ./bootstrap.sh --global' returned a non-zero code: 2
Is there something I can include in my Dockerfile that can prevent this?
So I've been trying to run the command
opencv_traincascade -data HandsData -vec hands.vec -bg HandsNeg.txt -numPos 3641 -numNeg 2578 -numStages 20 -w 27 -h 48 -mode ALL -minHitRate 0.999 -maxFalseAlarmRate 0.5 -precalcValBufSize 1024 -precalcIdxBufSize 1024
and I get the error
===== TRAINING 0-stage =====
<BEGIN
POS count : consumed 3641 : 3641
OpenCV Error: Assertion failed (_img.cols == winSize.width) in get, file /builddir/build/BUILD/OpenCV-2.0.0/apps/traincascade/imagestorage.cpp, line 86
terminate called after throwing an instance of 'cv::Exception'
Aborted
I've seen suggestions around to change the positive image number, where I use 1043 instead and still get the same error. Then I also see suggestions of editing the source code. The problem is that I built OpenCV with yum and would like to not have to rebuild from source.
sudo find / -name imagestorage.cpp
turns up nothing.
I'm at a complete loss of what to do.
Additional info: Steps I took to get to this point
I created everything from some videos using ffmpeg. These were from a phone and VLC lists the info as
Resolution: 1920x1090
Display resolution: 1920x1080
The ffmpeg command was (replacing input/output with respective videos and locations)
steven ~/computer_vision $ ffmpeg -i videos/Not\ hands\ stuff.mp4 -y -r 40 -s 27x48 -f image2 NotHandsorFists/Negs-%4d.png
Files in Hands.txt are of the form
steven ~/computer_vision $ cat Hands.txt | head -n 1
Hands/LeftHand-0001.png 1 0 0 27 48
I compiled the vec file with
steven ~/computer_vision $ opencv_createsamples -info Hands.txt -num 3641 -w 27 -h 48 -vec hands.vec
The negative file is in the form
steven ~/computer_vision $ cat HandsNeg.txt | head -n 1
Fists/LeftFist-0001.png
and the working directory is
steven ~/computer_vision $ ls
Fists fists.txt Hands HandsData HandsNeg.txt Hands.txt hands.vec NotHandsorFists NotHandsorFists.txt videos
Edit:
I've tried changing png to jpg and bmp to get rid of the channels. No help.
In spite of compiling my code with -pg flag set, I am unable to view the complete details of the gprof output from gmon.out
Specifically, I am not getting the details of calls, self Ts/call and total Ts/call.
This is the output of gprof deepflow-static gmon.out :
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls Ts/call Ts/call name
52.72 7.18 7.18 sor_coupled
34.07 11.82 4.64 compute_data_and_match
2.28 12.13 0.31 convolve_vert
2.28 12.44 0.31 compute_smoothness
2.06 12.72 0.28 convolve_horiz
1.47 12.92 0.20 color_image_resize_vert
1.03 13.06 0.14 sub_laplacian
0.88 13.18 0.12 image_resize_vert
0.88 13.30 0.12 image_warp
0.73 13.40 0.10 color_image_resize_horiz
0.59 13.48 0.08 image_resize_horiz
0.18 13.51 0.03 frexp
0.15 13.53 0.02 compute_one_level
0.15 13.55 0.02 fwrite
0.15 13.57 0.02 memcpy
0.11 13.58 0.02 __floorf_sse41
0.07 13.59 0.01 brk
0.07 13.60 0.01 color_image_png_load
0.07 13.61 0.01 get_derivatives
0.07 13.62 0.01 png_read_filter_row_paeth_multibyte_pixel
% the percentage of the total running time of the
time program used by this function.
cumulative a running sum of the number of seconds accounted
seconds for by this function and those listed above it.
self the number of seconds accounted for by this
seconds function alone. This is the major sort for this
listing.
calls the number of times this function was invoked, if
this function is profiled, else blank.
self the average number of milliseconds spent in this
ms/call function per call, if this function is profiled,
else blank.
total the average number of milliseconds spent in this
ms/call function and its descendents per call, if this
function is profiled, else blank.
name the name of the function. This is the minor sort
for this listing. The index shows the location of
the function in the gprof listing. If the index is
in parenthesis it shows where it would appear in
the gprof listing if it were to be printed.
Copyright (C) 2012 Free Software Foundation, Inc.
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.
This is my Makefile :
CC=gcc
CFLAGS=-Wall -g -O3
LDFLAGS=-g -Wall -O3
LIBFLAGS=-lm -ljpeg -lpng -pg
LIBAFLAGS=-static /usr/lib/x86_64-linux-gnu/libjpeg.a /usr/local/lib/libpng16.a /usr/lib/x86_64-linux-gnu/libz.a /usr/lib/x86_64-linux-gnu/libm.a
all: deepflow-static
deepflow: deepflow.o image.o io.o opticalflow_aux.o opticalflow.o solver.o
$(CC) $(LDFLAGS) $(LIBFLAGS) -o $# $^
deepflow-static: deepflow.o image.o io.o opticalflow_aux.o opticalflow.o solver.o
$(CC) $(LIBFLAGS) -o $# $^ $(LIBAFLAGS)
%.o: %.c
$(CC) -o $# $(CFLAGS) -c $+
clean:
rm -f *.o deepflow
Please help me in figuring out what is missing.
Thanks in advance
This has happened to me when the application being run with gprof was not terminating successfully.
The generation of the analysis file could not complete as the information was not fully written to gmon.out.
Try exiting the program after a few lines with:
exit(EXIT_SUCCESS);
And check whether or not the file is correctly generated.
You can also run the program through gdb.
I am not a Makefile expert, but you should be using the -pg flag on two steps:
GPROF=-pg
all: $(SOURCES) $(EXECUTABLE)
$(EXECUTABLE): $(OBJECTS)
$(CC) $(LDFLAGS) $(OBJECTS) -o $# $(LIBS) $(GPROF)
.cpp.o:
$(CC) $(CPPFLAGS) $(CFLAGS) $< -o $# $(GPROF)
I am trying to run the Lanczos Example of mahout.
I am having trouble finding the input file. and what should be the format of input file.
I have used the commands to convert the .txt file into sequence File format by running:
bin/mahout seqdirectory -i input.txt -o outseq -c UTF-8
bin/mahout seq2sparse -i outseq -o ttseq
bin/hadoop jar mahout-examples-0.9-SNAPSHOT-job.jar org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver --input /user/hduser/outputseq --output /out1 --numCols 2 --numRows 4 --cleansvd "true" --rank 5
14/03/20 13:36:12 INFO lanczos.LanczosSolver: Finding 5 singular vectors of matrix with 4 rows, via Lanczos
14/03/20 13:36:13 INFO mapred.FileInputFormat: Total input paths to process : 7
Exception in thread "main" java.lang.IllegalStateException: java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/ttseq/df-count/data
at org.apache.mahout.math.hadoop.DistributedRowMatrix.timesSquared(DistributedRowMatrix.java:245)
at org.apache.mahout.math.decomposer.lanczos.LanczosSolver.solve(LanczosSolver.java:104)
at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver.run(DistributedLanczosSolver.java:200)
at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver.run(DistributedLanczosSolver.java:152)
at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver.run(DistributedLanczosSolver.java:111)
at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver$DistributedLanczosSolverJob.run(DistributedLanczosSolver.java:283)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver.main(DistributedLanczosSolver.java:289)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/ttseq/df-count/data
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:51)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
at org.apache.mahout.math.hadoop.DistributedRowMatrix.timesSquared(DistributedRowMatrix.java:237)
... 13 more
Any idea please?
In your case you are doing input.txt -> outseq -> ttseq.
You are using outputseq (but not outseq) as input to generate out1.
And you are getting error with ttseq. That is strange? Perhaps you are missing some step in your post.
For me:
This PASSES: text-files -> output-seqdir -> output-seq2sparse-normalized
This FAILS: text-files -> output-seqdir -> output-seq2sparse -> output-seq2sparse-normalized
More details.
I am seeing this error in a different situation:
Create sequence files
$ mahout seqdirectory -i /data/lda/text-files/ -o /data/lda/output-seqdir -c UTF-8
Running on hadoop, using ....hadoop-1.1.1/bin/hadoop and HADOOP_CONF_DIR=
MAHOUT-JOB: ....mahout-distribution-0.7/mahout-examples-0.7-job.jar
14/03/24 20:47:25 INFO common.AbstractJob: Command line arguments: {--charset=[UTF-8], --chunkSize=[64], --endPhase=[2147483647], --fileFilterClass=[org.apache.mahout.text.PrefixAdditionFilter], --input=[/data/lda/ohsumed_full_txt/ohsumed_full_txt/], --keyPrefix=[], --output=[/data/lda/output], --startPhase=[0], --tempDir=[temp]}
14/03/24 20:57:20 INFO driver.MahoutDriver: Program took 594764 ms (Minutes: 9.912733333333334)
Convert sequence files to sparse vectors. Use TFIDF by default.
$ mahout seq2sparse -i /data/lda/output-seqdir -o /data/lda/output-seq2sparse/ -ow
Running on hadoop, using ....hadoop-1.1.1/bin/hadoop and HADOOP_CONF_DIR=
MAHOUT-JOB: ....mahout-distribution-0.7/mahout-examples-0.7-job.jar
14/03/24 21:00:08 INFO vectorizer.SparseVectorsFromSequenceFiles: Maximum n-gram size is: 1
14/03/24 21:00:09 INFO vectorizer.SparseVectorsFromSequenceFiles: Minimum LLR value: 1.0
14/03/24 21:00:09 INFO vectorizer.SparseVectorsFromSequenceFiles: Number of reduce tasks: 1
14/03/24 21:00:10 INFO input.FileInputFormat: Total input paths to process : 1
14/03/24 21:00:11 INFO mapred.JobClient: Running job: job_201403241418_0001
.....
14/03/24 21:02:51 INFO driver.MahoutDriver: Program took 162906 ms (Minutes: 2.7151)
Following command fails ( using /data/lda/output-seq2sparse as input )
$ mahout seq2sparse -i /data/lda/output-seq2sparse -o /data/lda/output-seq2sparse-normalized -ow -a org.apache.lucene.analysis.WhitespaceAnalyzer -chunk 200 -wt tfidf -s 5 -md 3 -x 90 -ng 2 -ml 50 -seq -n 2 -nr 5
Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/data/lda/output-seq2sparse/df-count/data
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:528)
at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:63)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
....SKIPPED....
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
However this works just fine ( using /data/lda/output-seqdir as input )
$ mahout seq2sparse -i /data/lda/output-seqdir -o /data/lda/output-seq2sparse-normalized -ow -a org.apache.lucene.analysis.WhitespaceAnalyzer -chunk 200 -wt tfidf -s 5 -md 3 -x 90 -ng 2 -ml 50 -seq -n 2 -nr 5
Running on hadoop, using .../hadoop-1.1.1/bin/hadoop and HADOOP_CONF_DIR=
MAHOUT-JOB: ..../mahout-distribution-0.7/mahout-examples-0.7-job.jar
14/03/24 21:35:55 INFO vectorizer.SparseVectorsFromSequenceFiles: Maximum n-gram size is: 2
14/03/24 21:35:56 INFO vectorizer.SparseVectorsFromSequenceFiles: Minimum LLR value: 50.0
14/03/24 21:35:56 INFO vectorizer.SparseVectorsFromSequenceFiles: Number of reduce tasks: 5
14/03/24 21:35:57 INFO input.FileInputFormat: Total input paths to process : 1
...SKIPPED...
14/03/24 21:45:11 INFO common.HadoopUtil: Deleting /data/lda/output-seq2sparse-normalized/partial-vectors-0
14/03/24 21:45:11 INFO driver.MahoutDriver: Program took 556420 ms (Minutes: 9.273666666666667)