We are currently building an embedded Linux OS using Yocto inside an Docker container. All persistent directories are mounted as volumes.
This is accomplished by generating an conf/site.conf setting those directories:
DL_DIR="/artifacts/downloads"
TMPDIR="/artifacts/tmp"
SSTATE_DIR="/artifacts/sstate_cache"
PERSISTENT_DIR="/artifacts/persistent"
DEPLOY_DIR_IMAGE="/images"
DEPLOY_DIR_IPK="/ipk"
And therefore running the image with
docker run --rm \
-v /opt/yocto/projectname:/artifacts \
-v /opt/deploy/projectname/ipk:/ipk \
-v /opt/deploy/projectname/images:/images \
-it <container>
All of this is working fine, the output is deployed as expected and everything works great.
However, upon rebuilding various recipes due to updates, we see yoctos pseudo build-environment abort()ing frequently. Most of the time its an rm or an tar command being killed.
Almost all errors are ino path mismatches like
path mismatch [1 link]: ino 23204894 db '/ipk/aarch64/glibc-charmap-jis-c6229-1984-a_2.35-r0_aarch64.ipk' req '/ipk/aarch64/locale-base-is-is_2.35-r0.1_aarch64.ipk'.
dir err : 107508467 ['/artifacts/tmp/work/aarch64-agl-linux/glibc-locale/2.35-r0/packages-split/glibc-binary-localedata-nb-no.iso-8859-1/CONTROL'] (db '/artifacts/tmp/work/aarch64-agl-linux/glibc-locale/2.35-r0/packages-split/glibc-binary-localedata-sgs-lt/CONTROL/control') db mode 0100644, header mode 040755 (unlinking db)
Child process exit status 4: lock_held
Couldn't obtain lock: Resource temporarily unavailable.
lock already held by existing pid 3365057.
(I appended the error logs of the this example at the end of this post)
or just plain
path mismatch [1 link]: ino 23200106 db '/ipk/aarch64/libcap-ng-doc_0.8.2-r0_aarch64.ipk' req '/ipk/aarch64/libcap-ng-doc_0.8.2-r0.1_aarch64.ipk'.
Setup complete, sending SIGUSR1 to pid 2167709.
When we were building the same project natively without docker, we never have seen errors like this. So we assume there are some compatibility issues with docker and pseudo. We already tried dockers devicemapper and overlay2 storage drivers.
The current workaround is to delete those affected files manually. But this mostly leads to other problems down the line.
We are out of ideas where to look in solving the problem. No yocto resources regarding pseudo-errors were of any help.
Is there any hint to debug those errors in a meaningful way or do we have to refactor the docker builds somehow to prevent those pseudo-errors?
Logs
Bitbake output
DEBUG: Hardlink test failed with [Errno 18] Invalid cross-device link: '/artifacts/tmp/work/aarch64-agl-linux/glibc-locale/2.35-r0/deploy-ipks/aarch64/glibc-localedata-tk-tm_2.35-r0.1_aarch64.ipk' -> '/ipk/testfile'
ERROR: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:sstate_task_postfunc(d)
0003:
File: '/yoctoagl/external/poky/meta/classes/sstate.bbclass', lineno: 822, function: sstate_task_postfunc
0818:
0819: sstateinst = d.getVar("SSTATE_INSTDIR")
0820: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir'])
0821:
*** 0822: sstate_installpkgdir(shared_state, d)
0823:
0824: bb.utils.remove(d.getVar("SSTATE_BUILDDIR"), recurse=True)
0825:}
0826:sstate_task_postfunc[dirs] = "${WORKDIR}"
File: '/yoctoagl/external/poky/meta/classes/sstate.bbclass', lineno: 418, function: sstate_installpkgdir
0414:
0415: for state in ss['dirs']:
0416: prepdir(state[1])
0417: bb.utils.rename(sstateinst + state[0], state[1])
*** 0418: sstate_install(ss, d)
0419:
0420: for plain in ss['plaindirs']:
0421: workdir = d.getVar('WORKDIR')
0422: sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
File: '/yoctoagl/external/poky/meta/classes/sstate.bbclass', lineno: 343, function: sstate_install
0339:
0340: # Run the actual file install
0341: for state in ss['dirs']:
0342: if os.path.exists(state[1]):
*** 0343: oe.path.copyhardlinktree(state[1], state[2])
0344:
0345: for postinst in (d.getVar('SSTATEPOSTINSTFUNCS') or '').split():
0346: # All hooks should run in the SSTATE_INSTDIR
0347: bb.build.exec_func(postinst, d, (sstateinst,))
File: '/yoctoagl/external/poky/meta/lib/oe/path.py', lineno: 134, function: copyhardlinktree
0130: s_dir = os.getcwd()
0131: cmd = 'cp -afl --preserve=xattr %s %s' % (source, os.path.realpath(dst))
0132: subprocess.check_output(cmd, shell=True, cwd=s_dir, stderr=subprocess.STDOUT)
0133: else:
*** 0134: copytree(src, dst)
0135:
0136:def copyhardlink(src, dst):
0137: """Make a hard link when possible, otherwise copy."""
0138:
File: '/yoctoagl/external/poky/meta/lib/oe/path.py', lineno: 94, function: copytree
0090: # This way we also preserve hardlinks between files in the tree.
0091:
0092: bb.utils.mkdirhier(dst)
0093: cmd = "tar --xattrs --xattrs-include='*' -cf - -S -C %s -p . | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
*** 0094: subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
0095:
0096:def copyhardlinktree(src, dst):
0097: """Make a tree of hard links when possible, otherwise copy."""
0098: bb.utils.mkdirhier(dst)
File: '/usr/lib/python3.9/subprocess.py', lineno: 424, function: check_output
0420: else:
0421: empty = b''
0422: kwargs['input'] = empty
0423:
*** 0424: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
0425: **kwargs).stdout
0426:
0427:
0428:class CompletedProcess(object):
File: '/usr/lib/python3.9/subprocess.py', lineno: 528, function: run
0524: # We don't call process.wait() as .__exit__ does that for us.
0525: raise
0526: retcode = process.poll()
0527: if check and retcode:
*** 0528: raise CalledProcessError(retcode, process.args,
0529: output=stdout, stderr=stderr)
0530: return CompletedProcess(process.args, retcode, stdout, stderr)
0531:
0532:
Exception: subprocess.CalledProcessError: Command 'tar --xattrs --xattrs-include='*' -cf - -S -C /artifacts/tmp/work/aarch64-agl-linux/glibc-locale/2.35-r0/deploy-ipks -p . | tar --xattrs --xattrs-include='*' -xf - -C /ipk' returned non-zero exit status 134.
Subprocess output:
abort()ing pseudo client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.
Check logfile: /artifacts/tmp/work/aarch64-agl-linux/glibc-locale/2.35-r0/pseudo//pseudo.log
Aborted (core dumped)
DEBUG: Python function sstate_task_postfunc finished
pseudo.log
debug_logfile: fd 2
pid 1234878 [parent 1234771], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 1234771.
db cleanup for server shutdown, 16:26:33.548
memory-to-file backup complete, 16:26:33.548.
db cleanup finished, 16:26:33.548
debug_logfile: fd 2
pid 997357 [parent 997320], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 997320.
db cleanup for server shutdown, 16:52:05.287
memory-to-file backup complete, 16:52:05.288.
db cleanup finished, 16:52:05.288
debug_logfile: fd 2
pid 30407 [parent 30405], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 30405.
dir err : 20362030 ['/artifacts/tmp/work/aarch64-agl-linux/glibc-locale/2.35-r0/packages-split/locale-base-kab-dz/CONTROL'] (db '/artifacts/tmp/work/aarch64-agl-linux/glibc-locale/2.35-r0/packages-split/locale-base-fr-ca.iso-8859-1/CONTROL/control') db mode 0100644, header mode 040755 (unlinking db)
debug_logfile: fd 2
pid 1901634 [parent 1901625], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 1901625.
db cleanup for server shutdown, 10:40:12.401
memory-to-file backup complete, 10:40:12.402.
db cleanup finished, 10:40:12.402
debug_logfile: fd 2
pid 3365057 [parent 3364988], doing new pid setup and server start
Setup complete, sending SIGUSR1 to pid 3364988.
debug_logfile: fd 2
pid 3365111 [parent 3365055], doing new pid setup and server start
lock already held by existing pid 3365057.
Couldn't obtain lock: Resource temporarily unavailable.
Child process exit status 4: lock_held
dir err : 107508467 ['/artifacts/tmp/work/aarch64-agl-linux/glibc-locale/2.35-r0/packages-split/glibc-binary-localedata-nb-no.iso-8859-1/CONTROL'] (db '/artifacts/tmp/work/aarch64-agl-linux/glibc-locale/2.35-r0/packages-split/glibc-binary-localedata-sgs-lt/CONTROL/control') db mode 0100644, header mode 040755 (unlinking db)
path mismatch [1 link]: ino 23204894 db '/ipk/aarch64/glibc-charmap-jis-c6229-1984-a_2.35-r0_aarch64.ipk' req '/ipk/aarch64/locale-base-is-is_2.35-r0.1_aarch64.ipk'.
db cleanup for server shutdown, 10:41:32.164
memory-to-file backup complete, 10:41:32.164.
db cleanup finished, 10:41:32.164
I'm a developer who's recently moved from macOS to WSL2 on Windows 10. I've finally managed to get everything working, but when I call a lambda locally through Docker, the os.Getenv() function returns my whole env file, instead of just the one key.
Things I've tried:
Set "files.eol" = "\n" in vscode settings.json
Set core.autocrlf = input in git config --global
Set eol = lf in git config --global
I've been super stumped for ages, and haven't been able to find any solutions online. Any help would be super appreciated!
Edit: Apologies, I should have had the foresight to post the code and env. necessary to replicate the problem
////////////
// Code: //
////////////
func init() {
if err := config.Load(); err != nil {
api.ReportError(err)
}
dbo = db.Instance{
DSN: os.Getenv("DBReadDataSourceName"),
}
log.Println("DSN: ", dbo.DSN)
}
// Load ...
func Load() error {
stage := os.Getenv("Stage")
log.Println("stage: ", stage)
if len(data) <= 0 && stage != "local" {
log.Println("stage != local")
log.Println("do production config and ssm stuff")
}
return nil
}
//////////
// Env: //
//////////
Stage=local
ServerPort=:1234
DBDriverName=mysql
DBReadDataSourceName=MySQLReadDataSourceCredentials
DBWriteDataSourceName=MySQLWriteDataSourceCredentials
RiakAddress=RiakAddress
RedisAddress=RedisAddress
ElasticSearchUrl=ElasticSearchUrl
ElasticSearchPrefix=ElasticSearchPrefix
ThidPartyBaseURL=https://api-sandbox.ThirdParty.com
ThidPartyCountryCode=SG
ThidPartyClientID=asdf12345qwertyuiadsf
ThidPartyClientSecret=asdf12345qwertyuiadsf
/////////////
// Output: //
/////////////
2020/07/13 17:04:03 stage: local
ServerPort=:1234
DBDriverName=mysql
DBReadDataSourceName=MySQLReadDataSourceCredentials
DBWriteDataSourceName=MySQLWriteDataSourceCredentials
RiakAddress=RiakAddress
RedisAddress=RedisAddress
ElasticSearchUrl=ElasticSearchUrl
ElasticSearchPrefix=ElasticSearchPrefix
ThidPartyBaseURL=https://api-sandbox.ThirdParty.com
ThidPartyCountryCode=SG
ThidPartyClientID=asdf12345qwertyuiadsf
ThidPartyClientSecret=asdf12345qwertyuiadsf
2020/07/13 17:04:03 stage != local
2020/07/13 17:04:03 do production config and ssm stuff
2020/07/13 17:04:03 DSN: DBReadDataSourceName
When I run this lambda locally on a mac, os.Getenv() functions as intended, returning local for Stage, and returning MySQLReadDataSourceCredentials for DBReadDataSourceName. However, running this lambda locally through WSL2 on a windows machine results in the above output. Stage returns the entire file, and DBReadDataSourceName returns DBReadDataSourceName.
I'm super stumped, and have tried everything I could think of, including manually writing \r\n on the end of each env. value. Any help would be extremely appreciated! Thank you very much for your time
Edit 2: From the comments
The command I used to load the env. file is env -S "`cat env.local`" sam local start-api --template template.yaml --profile company_local, with env.local being the file name.
The command env -S $'a=5\nb=6' sh -c 'echo "$a"' prints
5
b=6
The command env -S $'a=5\r\nb=6' sh -c 'echo "$a"' prints the exact same output as above,
5
b=6
The command env -S 'a=5;b=6' sh -c 'echo "$a"' prints
5;b=6
And env -S 'a=5:b=6' sh -c 'echo "$a"' prints
5:b=6
And the command xxd env.local | head -n 3 prints
00000000: 5374 6167 653d 6c6f 6361 6c0a 5365 7276 Stage=local.Serv
00000010: 6572 506f 7274 3d3a 3132 3334 0a44 4244 erPort=:1234.DBD
00000020: 7269 7665 724e 616d 653d 6d79 7371 6c0a riverName=mysql.
I am new to Fabric 2.0 and recently installed all samples and I was able to run test-network without an issue with 2 orgs. Then I followed the directory on addOrg3 to add 3rd organization and join the channel I created earlier.
Now the fun part came in when I wanted to add 4th organization. What I did was, I copied the addOrg3 folder and renamed almost everything in each file to represent 4th organization. I even assigned new PORT for this organization. However I am seeing the following error.
I've also added the following in Scripts/envVar.sh
export PEER0_ORG4_CA=${PWD}/organizations/peerOrganizations/org4.example.com/peers/peer0.org4.example.com/tls/ca.crt
And added the following in envVarCLI.sh
elif [ $ORG -eq 4 ]; then
CORE_PEER_LOCALMSPID="Org4MSP"
CORE_PEER_TLS_ROOTCERT_FILE=$PEER0_ORG4_CA
CORE_PEER_ADDRESS=peer0.org4.example.com:12051
CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/organizations/peerOrganizations/org4.example.com/users/Admin#.../msp
I have also added step1Org4.sh and step2Org4.sh basically following by following addOrg3's structure.
What steps do you follow to add additional organizations ? Please help.
"No such container: Org4cli"
Sorry for the formatting since I wasn't able to put in to coding style but here is the output from running the command "./addOrg4.sh up"
**Add Org4 to channel 'mychannel' with '10' seconds and CLI delay of '3' seconds and using database 'leveldb'
Desktop/blockchain/BSI/fabric-samples/test-network/addOrg4/../../bin/cryptogen
##########################################################
##### Generate certificates using cryptogen tool #########
##########################################################
##########################################################
############ Create Org4 Identities ######################
##########################################################
+ cryptogen generate --config=org4-crypto.yaml --output=../organizations
org4.example.com
+ res=0
+ set +x
Generate CCP files for Org4
Desktop/blockchain/BSI/fabric-samples/test-network/addOrg4/../../bin/configtxgen
##########################################################
####### Generating Org4 organization definition #########
##########################################################
+ configtxgen -printOrg Org4MSP
2020-05-29 13:33:04.609 EDT [common.tools.configtxgen] main -> INFO 001 Loading configuration
2020-05-29 13:33:04.617 EDT [common.tools.configtxgen.localconfig] LoadTopLevel -> INFO 002 Loaded configuration: /Desktop/blockchain/BSI/fabric-samples/test-network/addOrg4/configtx.yaml
+ res=0
+ set +x
###############################################################
####### Generate and submit config tx to add Org4 #############
###############################################################
Error: No such container: Org4cli
ERROR !!!! Unable to create config tx **
In your addOrg4.sh have condition check like this:
CONTAINER_IDS=$(docker ps -a | awk '($2 ~ /fabric-tools/) {print $1}')
if [ -z "$CONTAINER_IDS" -o "$CONTAINER_IDS" == " " ]; then
echo "Bringing up network"
Org4Up
fi
If you already run addOrg3.sh up, CONTAINER_IDS alway have value (Example: 51b4ad60d812). It is ContainerID of Org3cli. So function Org4Up will never call. Simple way is just comment code like this:
# CONTAINER_IDS=$(docker ps -a | awk '($2 ~ /fabric-tools/) {print $1}')
# if [ -z "$CONTAINER_IDS" -o "$CONTAINER_IDS" == " " ]; then
echo "Bringing up network"
Org4Up
# fi
It will bring up Org4cli you missing.
First check the container is up or not and if it is up then I think the CLI where the command is executed is not bootstrapped with the Org4 details.
I have added the 4th Organization from the three Org Hyperledger Fabric Network .Firstly, you have to create the Org4-artifacts (Crypto.yaml and Org4 docker file including the Org4Cli) and then try to follow the manual (step by step) process to add the new Organization from the official documentation.
https://hyperledger-fabric.readthedocs.io/en/release-2.0/channel_update_tutorial.html
Omit the process of editing scripts (step1 Org3.sh ...) because the workflow for adding the 4th or a new Org is slightly changed.So,you will spend a lot of time in just modifying the scripts.
I will write an article to add a new Org (4th) on medium,will paste the link here too.
I know the Enterprise (Cloudera for example) way, by using a CM (via browser) or by Cloudera REST API one can access monitoring and configuring facilities.
But how to schedule (run and rerun) flume agents livecycle, and monitor their running/failure status without CM? Are there such things in the Flume distribution?
Flume's JSON Reporting API can be used to monitor health and performance.
Link
I tried adding flume.monitoring.type/port to flume-ng on start. And it completely fits my needs.
Lets create a simple agent a1 for example. Which listens on localhost:44444 and logs to console as a sink:
# flume.conf
a1.sources = s1
a1.channels = c1
a1.sinks = d1
a1.sources.s1.channels = c1
a1.sources.s1.type = netcat
a1.sources.s1.bind = localhost
a1.sources.s1.port = 44444
a1.sinks.d1.channel = c1
a1.sinks.d1.type = logger
a1.channels.c1.type = memory
a1.channels.c1.capacity = 100
a1.channels.c1.transactionCapacity = 10
Run it with additional parameters flume.monitoring.type/port:
flume-ng agent -n a1 -c conf -f flume.conf -Dflume.root.logger=INFO,console -Dflume.monitoring.type=http -Dflume.monitoring.port=44123
And then monitor output in browser at localhost:44123/metrics
{"CHANNEL.c1":{"ChannelCapacity":"100","ChannelFillPercentage":"0.0","Type":"CHANNEL","EventTakeSuccessCount":"570448","ChannelSize":"0","EventTakeAttemptCount":"570573","StartTime":"1567002601836","EventPutAttemptCount":"570449","EventPutSuccessCount":"570448","StopTime":"0"}}
Just try some load:
dd if=/dev/urandom count=1024 bs=1024 | base64 | nc localhost 44444
enter image description here
enter image description here
my flume configuration file is as follows:
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /home/hadoop/flume-1.5.0-bin/log_exec_tail
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
And start my flume agent with the following stript:
bin/flume-ng agent -n a1 -c conf -f conf/flume_log.conf -Dflume.root.logger=INFO,console
question 1: the run result is as follows, however I don't konw if it run successful or not!
question 2: And there is the sentences as follows and I don't know what the mean is about "the queation of flume test":
NOTE: To test that the Flume agent is running properly, open a new terminal window and change directories to /home/horton/solutions/:
horton#ip:~$ cd /home/horton/solutions/
Run the following script, which writes log entries to nodemanager.log:
$ ./test_flume_log.sh
If successful, you should see new files in the /user/horton/flume_sink directory in HDFS
Stop the logagent Flume agent
As per your flume configuration, whenever the file /home/hadoop/flume-1.5.0-bin/log_exec_tail is changed, it will do a tail operation and append the results in the console.
So to test it working correctly,
1. run the command bin/flume-ng agent -n a1 -c conf -f conf/flume_log.conf -Dflume.root.logger=INFO,console
2. Open a terminal and add few lines in the file /home/hadoop/flume-1.5.0-bin/log_exec_tail
3. Save it
4. Now check the terminal where you triggered flume command
5. You can see newly added lines displayed