IBM Blockchain - Simple Application - swagger

Alright, I need to develop a simple application in an IBM Blockchain(starter plan). But I can't get it done, after almost 1 month of trial and error and thousands of tutorials.
So, my case is really simple(I guess):
I have a models.cto file:
namespace com.test.models
/**
* A company asset.
*/
asset Company identified by company_id {
o String company_id
o String document_key
o String document_value
o String name
o String telephone
o String email
}
/**
* A person_in_charge asset.
*/
asset PersonInCharge identified by person_in_charge_id {
o String person_in_charge_id
o String company_id
o String name
o String document_key
o String document_value
o String email
o String language
o String created_date_time
}
I tried to make it as participant instead. I tried to make transaction and then do something with .js file but there are only use cases and tutorials about complex examples using assets, participants, etc. I just want to insert "PersonInCharge" and "Company" separately and query (each Company or by id whatever) also separately.
I was able to made a example using Swagger api generator, but it didn't worked as expected, even thought it created all the methods I wanted, like this:
For summarize, I wanted to deploy my chaincode in a peer, inside of a channel at IBM Blockchain. I did deployed a marble example and it did worked perfectly.
Really self describing error, not with a general message:
I am following a "how to" guide for develop locally and deploy to IBM. Therefore, I am creating a bna file composer archive create -t dir -n . and then creating a card with the downloaded connection settings(from IBM) and else using this tutorial Deploying a business network on Starter Plan
and I am failing at this step: composer network start -c adminCard -n bna-blockchain -V 0.0.1 -A admin -C ./credentials/admin-pub.pem -f delete_me.card. perhaps beucause I'm not being able to instantiate my chaincode on IBM Blockchain.
Sorry if it looks confunsing, just ask me if you have any questions about my issue.
Thanks in advance and any help would be awesome!

Before starting the network, you need to install it:
composer network install -c adminCard -a vehicle-manufacture-network.bna

I took a break in the development and came back after a month. In that time, I had help of a person inside IBM who could help me to get it done.
And I'm here so point out, some failures too.
The IBM guy(I won't say his name, of course) itself said that the service to upload a chaincode(channels/chaincode/Install chaincode) is broken and does not work well.
Also, I asked an easy to answer question in the IBM Blockchain paid forum but had no help at all(they did not even answer the question), even after almost 3 months. And you can't find a tutorial(besides marbles samples) that works from start to end.
Sorry if it looks like I'm raging for nothing in here, but I had a really hard time to put it all together to work 100%.
I will not show how to create a model file or transactional scripts(because there are a plenty of it on examples) but the commands you need to execute in order to run the chaincode on IBM and generate a loopback API. With that said...
Let's go to what's important: the code
First things first, Environment settings:
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
IMPORTANT: IBM Blockchain(until now 2019-03-23) won't run under virtualized machines like virtual box or hyperv because of a couple npm dependencies. But a hardware virtual machine(HVM) like amazon ones will do just fine.
STEPS
Install node and npm(follow the versions strictly as listed)
node -v
v8.15.1
npm -v
6.4.1
curl -O https://hyperledger.github.io/composer/unstable/prereqs-ubuntu.sh
chmod u+x prereqs-ubuntu.sh
./prereqs-ubuntu.sh
npm install -g --unsafe-perm composer-cli#0.20.5
npm install -g --unsafe-perm composer-rest-server#0.20.5
composer-rest-server -v
v0.20.5
composer -v
v0.20.5
Create a project using hyperledger composer-playground. Export it and navigate through the root directory of the project you just downloaded(the project can be developed locally too).
Download the connection settings file at IBM Blockchain(Overview/Connection Profile/Download) and name it as connection-profile.json. Find the node registrar, you'll use it to create the cards.
Run npm install. If anything goes wrong while installing the dependencies, check if you're using exactly the same versions that I am using. If you're not, uninstall it all and start again from the beginning.
tricky zone
composer card create -f ca.card -p connection-profile.json -u admin -s *your-password*
composer card import -f ca.card -c ca
composer identity request --card ca --path ./credentials -u admin -s *your-password*
You'll see that the credentials folder were generated. Inside this folder, find
the file admin-pub.pem (credentials/admin-pub.pem) copy the whole content of it, and then upload it to IBM Blockchain(Members/Certificates/Add Certificate). You'll be prompted to restart peers, click "yes"
After the peers come back online, in the same Certificates tab, find the certificate you just add, in the ACTION column click at the menu, and then choose the option "Sync Certificate".
composer archive create -t dir -n .
composer card create -f adminCard.card -p connection-profile.json -u admin -c ./credentials/admin-pub.pem -k ./credentials/admin-priv.pem --role PeerAdmin --role ChannelAdmin
composer card import -f ./adminCard.card -c adminCard
composer network install -c adminCard -a bna-name#version.bna
composer network start -c adminCard -n *bna-name* -V *bna-version* -A admin -C ./credentials/admin-pub.pem -f delete_me.card (Output: Successfully created business network card: Filename: delete_me.card)
composer card create -n *bna-name* -p connection-profile.json -u admin -c ./credentials/admin-pub.pem -k ./credentials/admin-priv.pem (Output file: admin#*bna-name*.card)
composer card import -f ./admin#*bna-name*.card (Card file: ./admin#*bna-name*.card Card name: admin#*bna-name*)
Now you'll generate the rest api using (automatically generated using swagger)
composer-rest-server -c admin#*bna-name* -n never -w true -p 8080
Enjoy!

Related

Error: failed to request identity. Error trying to enroll user and return certificates

I am biginner in block chain development. I succesfully run ./startFabric.sh and ./createPeerAdminCard.sh .Then i succesfully run two command as follow below.
composer runtime install -c PeerAdmin#byfn-network-org1-only -n trade-network
composer runtime install -c PeerAdmin#byfn-network-org2-only -n trade-network
Then i tried to run
composer identity request -c PeerAdmin#byfn-network-org1-only -u admin -s adminpw -d alice
But i got error like the following
Error: failed to request identity. Error trying to enroll user and return certificates. Error: Calling enrollment endpoint failed with error [Error: connect ECONNREFUSED 127.0.0.1:7054]
I tried many time to kill all process in docker and remove .composer file from home directory. But always showing the same error message.
Please help me as possible
Os: Ubuntu 16.04
Fabric vertion:1.0.4
Composer vertion:0.16.6
Thanks..
This error "Failed to import the business network card Connection profile has no x-type property defined." means that a v0.20 client can't/won't import a card from an earlier version (probably v0.16).
I think maybe you are running the wrong version of createPeerAdminCard.sh script and it is trying to create and import an 'old' card.
With Composer v0.20 you need to get 'new' copies of the fabric-development-servers scripts which work with Fabric 1.2.
So I would suggest running the following:
run teardownAllDocker.sh script
remove the fabric-dev-server folder (might be fabric-tools if old)
remove the ~/.composer folder
follow the install doc from Step 4 . - make sure to export FABRIC_VERSION=hlfv12 - this is what makes sure you get Fabric 1.2.
This is difficult to help ...
The first 2 scripts you have run (./startFabric.sh and ./createPeerAdminCard.sh) are for a basic development Fabric.
The next 2 commands look like they are from the Multi-Org tutorial which uses a different Fabric and has different cards and crypto material.
Unless there is a particular reason for using an old version of Composer, I would suggest using Composer v0.20, which requires Fabric 1.2.
If you are a beginner I would suggest working with the Developer Tutorial to start with.
if you are using a default file (e.g. connection.json, env.json, etc.) to hold your base connnection profile, simply change the "type" term in that default file to "x-type"
In converting from v0.16 to v0.20, you must first remove the v0.16 composer tools from your system, then install the v0.20 version.
To remove v0.16.6:
npm uninstall -g composer-cli
npm uninstall -g generator-hyperledger-composer
npm uninstall -g composer-rest-server
To install V0.20.2:
npm install -g --python=python2.7 composer-cli#0.20.2
npm install -g --python=python2.7 generator-hyperledger-composer#0.20.2
npm install -g --python=python2.7 composer-rest-server#0.20.2
npm install -g --python=python2.7 yo
If you're running nodejs, then you also have to update both package.json files (network/package/json as well as your {base folder}package.json) from 0.16.x to ^0.20.2
The reason for error is Missing of Certificate Authorities(CA). I Just up my CA server. Now Working perfectly.

`docker-credential-gcloud` not in system PATH

After the latest updates to gcloud and docker I'm unable to access images on my google container repository. Locally when I run: gcloud auth configure-docker as per the instructions after updating gcloud, I get the following message:
WARNING: `docker-credential-gcloud` not in system PATH.
gcloud's Docker credential helper can be configured but it will not work until this is corrected.
gcloud credential helpers already registered correctly.
Running which docker-credential-gcloud returns docker-credential-gcloud not found.
I have no other gcloud-related path issues and for the life of me can't figure out how to install/add docker-credential-gcloud to path. Here's what I have installed (shown via gcloud version):
Google Cloud SDK 197.0.0
beta 2017.09.15
bq 2.0.31
container-builder-local
core 2018.04.06
docker-credential-gcr
gsutil 4.30
I also have Docker CE Version 18.03.0-ce-mac60 (23751).
Here's my $PATH:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
I also ran source /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.zsh.inc on original gcloud install.
Notice: All docker-credential-gcr below can be replaced with docker-credential-gcloud. I think it is just different versions of gcloud, I might be wrong.
I used Homebrew Cask to install gcloud too. I installed docker-credential-gcr with
$ gcloud components install docker-credential-gcr
And then like you said, which docker-credential-gcr doesn't gave you anything.
So I ran which gcloud to find there is a symlink to gcloud in /usr/local/bin. This symlink is created by Homebrew when you installed gcloud at first place. Now docker-credential-gcr wasn't installed by Homebrew but by gcloud itself, so there isn't a symlink.
I called readlink /usr/local/bin/gcloud and found out gcloud is installed in /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin/.
Then:
$ ls /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin
There you should see docker-credential-gcr listed there.
I simply linked it to /usr/local/bin:
$ ln -s \
/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin/docker-credential-gcr \
/usr/local/bin/
Then run:
$ docker-credential-gcr configure-docker
It should succeed.
Just had the same issue on Windows, running Docker with Linux containers, Docker engine v19.03.8. Using docker compose. I do not use gcloud for my dockerfiles...
DT1001 dockerpycreds.errors.InitializationError:
docker-credential-gcloud not installed or not available in PATH
Option 1: Edit the docker configuration file and remove all gcloud entries from there.
Windows c:/Users/<your account>/.docker/config.json
Linux & MacOS ~/.docker/config.json
Option 2: Go to Troubleshoot -> Reset to factory defaults.
After this my docker compose was creating containers and running the images without any issues.
On MacOS
Step 1:
Install gcloud and docker-credential-gcr,
following this tutorial
Step 2:
$ ln -s /usr/local/google-cloud-sdk/bin/docker-credential-gcr /usr/local/bin/docker-credential-gcloud
Step 3:
$ rm -rf ~/.docker
Step 4:
$ docker-compose build --pull
Finished!
Never found a way to directly resolve the docker-credential-gcloud issue, but the following got me up and running again. WARNING: the following will delete all your existing docker images and install a bunch of gcloud utilities:
gcloud components install docker-credential-gcr,
Restart the terminal completely
docker-credential-gcr configure-docker.
screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
umount /var/lib/docker/overlay2
rm -rf /var/lib/docker
Restart the terminal completely.
The new version of google-cloud-sdk has only docker-credential-gcr but not docker-credential-gcloud anymore. On the other hand one of my python packages always requested docker-credential-gcloud.
The solution was to symlink docker-credential-gcloud to docker-credential-gcr:
ln -s /path/to/google-cloud-sdk/bin/docker-credential-gcr /usr/local/bin/docker-credential-gcloud
ls -l /usr/local/bin | grep docker should now print:
...
docker-credential-gcloud -> /path/to/google-cloud-sdk/bin/docker-credential-gcr
...
Usually, this error indicates that your $PATH variable has been clobbered by a package or program you have recently installed so that the Google Cloud SDK can't be found.
$PATH is altered by many programs when they install by altering ~/.profile, ~/.bash_profile or ~/.bashrc or their non-bash equivalents. With a bad $PATH, Google Cloud SDK is configured in docker but can't be seen as executables so we get this error. This assumes you have used the Google Cloud SDK in the past, but if gcloud is configured with your docker then you probably have. Don't reinstall gcloud or disable it, you already have it on your system and that is fine.
The solution then is to fix your $PATH, not to install anything.
echo $PATH
This should be a pretty long : delimited list of directories that your files are in. Do you see a google-cloud-sdk/bin in the string? Is the string way too short given all the trouble you've gotten into in your life on this computer? You use NVM but it is missing? Use Homebrew but it is missing? Try brew from the command line, does it work?
If the answer is "no" to any of the above, inspect the files above to see if there are any new entries at the bottom of each that might have broken things. Did you just install anything new?
Something is clobbering your $PATH and you need to figure out what that is. For me it is usually something to do with Anaconda Python via the conda init command. For you it might be nvm or something else. Figure out what it is and fix the problem. Don't start over with a new $PATH and install the same stuff over again or disable gcloud authentication.
It really seems to be something with the Homebrew Cask. I uninstalled the cask and then reinstalled the Google Cloud SDK by manually downloading the tar ball and running the packaged install script as described there.
Now docker-credential-gcloud is in my path:
$ which docker-credential-gcloud
/Users/moritz/google-cloud-sdk/bin/docker-credential-gcloud
I can't figure out what Google is trying to achieve here. On Linux there is docker-credential-gcloud and on Windows there is docker-credential-gcr.exe, and then there is docker-credential-gcloud.cmd which calls gcloud auth docker-helper. This is kind of a nightmare if you're trying to write portable build scripts or gradle rules because not everything seems capable of finding and calling docker-credential-gcloud.cmd when you exec docker-credential-gcloud... it might work from the dos prompt, but in general doesn't work.
After a ton of fooling around with .bat scripts, cygwin scripts, .cmd scripts and so forth, I found the best solution was to go into the gcloud installation and just copy docker-credential-gcr.exe docker-credential-gcloud.exe ... not a very satisfying solution, but is the only thing I found that would do the trick.
I got the issue when I tried to SSH from Google Cloud Build into an Engine VM Instance, so I had
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args: ['compute', 'ssh',
'--project', '$PROJECT_ID',
'--zone', 'asia-southeast1-b',
'--strict-host-key-checking=no',
'username#instance-1',
'--command' ,'sh start.sh'
My start.sh
#!/bin/sh
echo "Started: $(date --iso-8601=seconds)"
docker pull gcr.io/aaa/bbbc/cccc
echo "Finished: $(date --iso-8601=seconds)"
The issue was How to set PATH when running a ssh command?
https://unix.stackexchange.com/questions/332532/how-to-set-path-when-running-a-ssh-command
So I just faced the same problem where I am trying to pull an image from GCR to an GCP instance and want to share my solution.
I ran gcloud auth configure-docker and got the warning:
WARNING: `docker-credential-gcloud\` not in system PATH.
gcloud's Docker credential helper can be configured but it will not work until this is corrected.
I applied the accepted answer for this thread and ran gcloud components install docker-credential-gcr and got a long error:
ERROR: (gcloud.components.install) You cannot perform this action because this Cloud SDK installation is managed by an external package manager.
Please consider using a separate installation of the Cloud SDK created through the default mechanism described at: https://cloud.google.com/sdk/
When no solution was working, I uninstalled the Google provided google-cloud-sdk package that was installed via snap and instlled with distro specifice package manager, for me that is apt-get as instructed in the Installing Google Cloud SDK: Installation options page and re-ran the gcloud auth configure-docker and this time it solved my problem.
In my case the problem was due to how WSL 1 works with Docker on Windows. At first I only installed and initialized gcloud in WSL Ubuntu, not in Windows. However as Docker daemon is actually run by Windows, you need to install gcloud for Windows as well (and don't forget to run all of the inits and authorizations there).
On Windows 10/11, you need to ensure that C:\Users\USERNAME\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin\ is added to your system $PATH environment variable. It may not have been added if the Google Cloud SDK was not able to add it during GCloud installation. So add it manually like this:
Windows Task Bar ➔ Press the search icon 🔍 or the search bar
Type "environment" ➔ and click on "Edit the System Environment Variables" (ensure that you have Administrator access)
At the bottom of the dialog, click the Environment Variables... button
System Variables ➔ click Path ➔ Edit... ➔ New ➔ paste in C:\Users\USERNAME\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin\ (replace "USERNAME" with your username)
Close and restart any open Command Prompt windows.
Then verify on the Git Bash for Windows console:
Optional: Note that the AppData folder is hidden by default, so you may want to unhide AppData first, to see its contents.
Restart the Git Bash Terminal window
echo $PATH ➔ This should print a long string that contains: :/c/Users/USERNAME/AppData/Local/Google/Cloud SDK/google-cloud-sdk/bin
where docker-credential-gcloud ➔ This should print C:\Users\USERNAME\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin\docker-credential-gcloud.cmd

Start up Fabric Composer REST API server

I am trying to use the composer-rest-server package after completing the Fabric Composer quick start, but I failed.
Please give me a hint.
The composer-rest-server package was successfully installed.
npm install -g composer-rest-server
Composer-rest-server started.
composer-rest-server
I was asked to enter simple information about the business network.
After that, I failed to load the connector module 'composer-connector-hlf'.
? Enter your Fabric Connection Profile Name: defaultProfile 
? Enter your Business Network Identifier : digitalproperty-network
? Enter your Fabric username : WebAppAdmin
? Enter your secret: DJY27pEnl16d
? Specify if you want namespaces in the generated REST API: always use namespace
s
? Specify if you want the generated REST API to be secured: Yes
To restart the REST server using the same options, issue the following command:
composer-rest-server -p defaultProfile -n digitalproperty-network -i WebAppAdmin -s DJY27pEnl16d -N always -S true
Discovering types from business network definition ...
Connection fails: Error: Failed to load connector module "composer-connector-hlf" for connection profile "defaultProfile"
It will be retried for the next request.
Error: Failed to load connector module "composer-connector-hlf" for connection profile "defaultProfile"
at connectionProfileStore.load.then.e (/home/user1/.nvm/versions/node/v6.9.5/lib/node_modules/composer-rest-server/node_modules/composer-common/lib/connectionprofilemanager.js:148:27)
How can I solve the problem?
Your .composer-connection-profiles is not set to defaultprofile.
Check your root folder and there .composer-connection-profiles folder must be present.
Inside that folder the composer profiles will be there (if your defaultprofile is made, then there will be a folder named defaultprofile), check the connection.json inside the defaultprofile folder.
If there is any other folder with different name, use that as your profile. Or you can specifically define the folder defaultprofile and write connection.json with connection details.
Which version of npm are you using? I faced similar issue on ubuntu 16.4. When I retinstall evberything with lower version of npm it worked perfectly.
npm version when things worked:
npm --version
3.10.10
when I was getting problem as you have mentioned it was 5.x
Does your grpc module has grpc_node.node files?
It's in the composer-cli module.
In my environment
/usr/local/lib/node_modules/composer-cli/node_modules/grpc/src/node/extension_binary/grpc_node.node
if you use sudo, you may need to add --unsafe-perm option.
Like: sudo npm install -g --unsafe-perm composer-cli

How to run travis-ci locally

I'd rather not have to push every little change to .travis.yml and every little change I make to the source in order to run the build. With jenkins you can download jenkins and run locally. Does travis offer something like this?
Note: I've seen the travis-ci cli and downloaded it, but all it seems
to do is call their API, which then connects to my GitHub repo, so if
I don't push, it won't matter that I restart the last build.
This process allows you to completely reproduce any Travis build job on your computer. Also, you can interrupt the process at any time and debug. Below is an example where I perfectly reproduce the results of job #191.1 on php-school/cli-menu
.
Prerequisites
You have public repo on GitHub
You ran at least one build on Travis
You have Docker set up on your computer
Set up the build environment
Reference: https://docs.travis-ci.com/user/common-build-problems/
Make up your own temporary build ID
BUILDID="build-$RANDOM"
View the build log, open the show more button for WORKER INFORMATION and find the INSTANCE line, paste it in here and run (replace the tag after the colon with the newest available one):
INSTANCE="travisci/ci-garnet:packer-1512502276-986baf0"
Run the headless server
docker run --name $BUILDID -dit $INSTANCE /sbin/init
Run the attached client
docker exec -it $BUILDID bash -l
Run the job
Now you are now inside your Travis environment. Run su - travis to begin.
This step is well defined but it is more tedious and manual. You will find every command that Travis runs in the environment. To do this, look for for everything in the right column which has a tag like 0.03s.
On the left side you will see the actual commands. Run those commands, in order.
Result
Now is a good time to run the history command. You can restart the process and replay those commands to run the same test against an updated code base.
If your repo is private: ssh-keygen -t rsa -b 4096 -C "YOUR EMAIL REGISTERED IN GITHUB" then cat ~/.ssh/id_rsa.pub and click here to add a key
FYI: you can git pull from inside docker to load commits from your dev box before you push them to GitHub
If you want to change the commands Travis runs then it is YOUR responsibility to figure out how that translates back into a working .travis.yml.
I don't know how to clean up the Docker environment, it looks complicated, maybe this leaks memory
Travis-ci offers a new container-based infrastructure that uses docker. This can be very useful if you're trying to troubleshoot a travis-ci build by reproducing it locally. This is taken from Travis CI's documentation.
Troubleshooting Locally in a Docker Image
If you're having trouble tracking down the exact problem in a build it often helps to run the build locally. To do this you need to be using our container based infrastructure (ie, have sudo: false in your .travis.yml), and to know which Docker image you are using on Travis CI.
Running a Container Based Docker Image Locally
Download and install the Docker Engine.
Select an image from Docker Hub. If you're not using a language-specific image pick ci-ruby. Open a terminal and start an interactive Docker session using the image URL:
docker run -it travisci/ubuntu-ruby:18.04 /bin/bash
Switch to the travis user:
su - travis
Clone your git repository into the / folder of the image.
Manually install any dependencies.
Manually run your Travis CI build command.
UPDATE: I now have a complete turnkey, all-in-one answer, see https://stackoverflow.com/a/49019950/300224. Only took 3 years to figure out!
According to the Travis documentation: https://github.com/travis-ci/travis-ci there is a concoction of projects that collude to deliver the Travis CI web service we know and love. The following subset of projects appears to allow local make test functionality using the .travis.yml in your project:
travis-build
travis-build creates the build
script for each job. It takes the configuration from the .travis.yml file and
creates a bash script that is then run in the build environment by
travis-worker.
travis-cookbooks
travis-cookbooks holds the
Chef cookbooks that are used to provision the build environments.
travis-worker
travis-worker is responsible for
running the build scripts in a clean environment. It streams the log output to
travis-logs and pushes state updates (build starting/finishing)
to travis-hub.
(The other subprojects are responsible for communicating with GitHub, their web interface, email, and their API.)
Similar to Scott McLeod's but this also generates a bash script to run the steps from the .travis.yml.
Troubleshooting Locally in Docker with a generated Bash script
# choose the image according to the language chosen in .travis.yml
$ docker run -it -u travis quay.io/travisci/travis-jvm /bin/bash
# now that you are in the docker image, switch to the travis user
sudo - travis
# Install a recent ruby (default is 1.9.3)
rvm install 2.3.0
rvm use 2.3.0
# Install travis-build to generate a .sh out of .travis.yml
cd builds
git clone https://github.com/travis-ci/travis-build.git
cd travis-build
gem install travis
# to create ~/.travis
travis version
ln -s `pwd` ~/.travis/travis-build
bundle install
# Create project dir, assuming your project is `AUTHOR/PROJECT` on GitHub
cd ~/builds
mkdir AUTHOR
cd AUTHOR
git clone https://github.com/AUTHOR/PROJECT.git
cd PROJECT
# change to the branch or commit you want to investigate
travis compile > ci.sh
# You most likely will need to edit ci.sh as it ignores matrix and env
bash ci.sh
Use wwtd (what would travis do) ruby gem to run tests on your local machine roughly as they would run on travis.
It will recreate the build matrix and run each configuration, great to sanity check setup before pushing.
gem i wwtd
wwtd
tl;dr Use image specified at https://docs.travis-ci.com/user/common-build-problems/#troubleshooting-locally-in-a-docker-image in combination with https://github.com/travis-ci/travis-build#use-as-addon-for-travis-cli.
EDIT 2019-12-06
#troubleshooting-locally-in-a-docker-image section was replaced by #running-builds-in-debug-mode which also describes how to SSH to the job running in the debug mode.
EDIT 2019-07-26
#troubleshooting-locally-in-a-docker-image section is no longer part of the docs; here's why
https://github.com/travis-ci/docs-travis-ci-com/issues/2342
https://blog.travis-ci.com/2018-10-04-combining-linux-infrastructures
https://blog.travis-ci.com/2018-11-30-announcing-xenial-build-environment-for-enterprise
Though, it's still in git history: https://github.com/travis-ci/docs-travis-ci-com/pull/2193.
Look for (quite old, couldn't find newer) image versions at: https://travis-ci.org/travis-ci/docs-travis-ci-com/builds/230889063#L661.
I wanted to inspect why one of the tests in my build failed with an error I din't get locally.
Worked.
What actually worked was using the image specified at Troubleshooting Locally in a Docker Image docs page. In my case it was travisci/ci-garnet:packer-1512502276-986baf0.
I was able to add travise compile following steps described at https://github.com/travis-ci/travis-build#use-as-addon-for-travis-cli.
dm#z580:~$ docker run --name travis-debug -dit travisci/ci-garnet:packer-1512502276-986baf0 /sbin/init
dm#z580:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
travisci/ci-garnet packer-1512502276-986baf0 6cbda6a950d3 11 months ago 10.2GB
dm#z580:~$ docker exec -it travis-debug bash -l
root#912e43dbfea4:/# su - travis
travis#912e43dbfea4:~$ cd builds/
travis#912e43dbfea4:~/builds$ git clone https://github.com/travis-ci/travis-build
travis#912e43dbfea4:~/builds$ cd travis-build
travis#912e43dbfea4:~/builds/travis-build$ mkdir -p ~/.travis
travis#912e43dbfea4:~/builds/travis-build$ ln -s $PWD ~/.travis/travis-build
travis#912e43dbfea4:~/builds/travis-build$ gem install bundler
travis#912e43dbfea4:~/builds/travis-build$ bundle install --gemfile ~/.travis/travis-build/Gemfile
travis#912e43dbfea4:~/builds/travis-build$ bundler binstubs travis
travis#912e43dbfea4:~/builds/travis-build$ cd ..
travis#912e43dbfea4:~/builds$ git clone --depth=50 --branch=master https://github.com/DusanMadar/PySyncDroid.git DusanMadar/PySyncDroid
travis#912e43dbfea4:~/builds$ cd DusanMadar/PySyncDroid/
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$ ~/.travis/travis-build/bin/travis compile > ci.sh
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$ sed -i 's,--branch\\=\\\x27\\\x27,--branch\\=master,g' ci.sh
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$ bash ci.sh
Everything from .travis.yml was executed as expected (dependencies installed, tests ran, ...).
Note that before running bash ci.sh I had to change --branch\=\'\'\ to --branch\=master\ (see the second to last sed -i ... command) in ci.sh.
If that doesn't work the command bellow will help to identify the target line number and you can edit the line manually.
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$ cat ci.sh | grep -in branch
840: travis_cmd git\ clone\ --depth\=50\ --branch\=\'\'\ https://github.com/DusanMadar/PySyncDroid.git\ DusanMadar/PySyncDroid --echo --retry --timing
889:export TRAVIS_BRANCH=''
899:export TRAVIS_PULL_REQUEST_BRANCH=''
travis#912e43dbfea4:~/builds/DusanMadar/PySyncDroid$
Didn't work.
Followed the accepted answer for this question but didn't
find the image (travis-ci-garnet-trusty-1512502259-986baf0) mentioned by instance at https://hub.docker.com/u/travisci/.
Build worker version points to travis-ci/worker commit and its travis-worker-install references quay.io/travisci/ as image registry. So I tried it.
dm#z580:~$ docker run -it -u travis quay.io/travisci/travis-python /bin/bash
travis#370c23a773c9:/$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise
travis#370c23a773c9:/$
dm#z580:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/travisci/travis-python latest 753a216d776c 3 years ago 5.36GB
Definitely not Trusty (Ubuntu 14.04) and not small either.
You could try Trevor, which uses Docker to run your Travis build.
From its description:
I often need to run tests for multiple versions of Node.js. But I don't want to switch versions manually using n/nvm or push the code to Travis CI just to run the tests.
That's why I created Trevor. It reads .travis.yml and runs tests in all versions you requested, just like Travis CI. Now, you can test before push and keep your git history clean.
I'm not sure what was your original reason for running Travis locally, if you just wanted to play with it, then stop reading here as it's irrelevant for you.
If you already have experience with hosted Travis and you want to get the same experience in your own datacenter, read on.
Since Dec 2014 Travis CI offers an Enterprise on-premises version.
http://blog.travis-ci.com/2014-12-19-introducing-travis-ci-enterprise/
The pricing is part of the article as well:
The licensing is done per seats, where every license includes 20 users. Pricing starts at $6,000 per license, which includes 20 users and 5 concurrent builds. There's a premium option with unlimited builds for $8,500.
I wasn't able to use the answers here as-is. For starters, as noted, the Travis help document on running jobs locally has been taken down. All of the blog entries and articles I found are based on that. The new "debug" mode doesn't appeal to me because I want to avoid the queue times and the Travis infrastructure until I've got some confidence I have gotten somewhere with my changes.
In my case I'm updating a Puppet module and I'm not an expert in Puppet, nor particularly experienced in Ruby, Travis, or their ecosystems. But I managed to build a workable test image out of tips and ideas in this article and elsewhere, and by examining the Travis CI build logs pretty closely.
I was unable to find recent images matching the names in the CI logs (for example, I could find travisci/ci-sardonyx, but could not find anything with "xenial" or with the same build name). From the logs it appears images are now transferred via AMQP instead of a mechanism more familiar to me.
I was able to find an image travsci/ubuntu-ruby:16.04 which matches the OS I'm targeting for my particular case. It does not have all the components used in the Travis CI, so I built a new one based on this, with some components added to the image and others added in the container at runtime depending on the need.
So I can't offer a clear procedure, sorry. But what I did, essentially boiled down:
Find a recent Travis CI image in Docker Hub matching your target OS as closely as possible.
Clone the repository to a build directory, and launch the container with the build directory mounted as a volume, with the working directory set to the target volume
Now the hard work: go through the Travis build log and set up the environment. In my case, this meant setting up RVM, and then using bundle to install the project's dependencies. RVM appeared to be already present in the Travis environment but I had to install it; everything else came from reproducing the commands in the build log.
Run the tests.
If the results don't match what you saw in the Travis CI logs, go back to (3) and see where to go.
Optionally, create a reusable image.
Dev and test locally and then push and hopefully your Travis results will be as expected.
I know this is not concrete and may be obvious, and your mileage will definitely vary, but hopefully this is of some use to somebody. The Dockerfile and a README for my image are on GitHub for reference.
It is possible to SSH to Travis CI environment via a bounce host. The feature isn't built in Travis CI, but it can be achieved by the following steps.
On the bounce host, create travis user and ensure that you can SSH to it.
Put these lines in the script: section of your .travis.yml (e.g. at the end).
- echo travis:$sshpassword | sudo chpasswd
- sudo sed -i 's/ChallengeResponseAuthentication no/ChallengeResponseAuthentication yes/' /etc/ssh/sshd_config
- sudo service ssh restart
- sudo apt-get install sshpass
- sshpass -p $sshpassword ssh -R 9999:localhost:22 -o StrictHostKeyChecking=no travis#$bouncehostip
Where $bouncehostip is the IP/host of your bounce host, and $sshpassword is your defined SSH password. These variables can be added as encrypted variables.
Push the changes. You should be able to make an SSH connection to your bounce host.
Source: Shell into Travis CI Build Environment.
Here is the full example:
# use the new container infrastructure
sudo: required
dist: trusty
language: python
python: "2.7"
script:
- echo travis:$sshpassword | sudo chpasswd
- sudo sed -i 's/ChallengeResponseAuthentication no/ChallengeResponseAuthentication yes/' /etc/ssh/sshd_config
- sudo service ssh restart
- sudo apt-get install sshpass
- sshpass -p $sshpassword ssh -R 9999:localhost:22 -o StrictHostKeyChecking=no travisci#$bouncehostip
See: c-mart/travis-shell at GitHub.
See also: How to reproduce a travis-ci build environment for debugging

Jenkins Fail with: Host key verification failed

I downloaded and installed Jenkins for Mac OSX on my Macbook Pro (OS: Mountain Lion). I now want to set it up to pull down a project from bitbucket and do an automatic build.
I created the ssh key, added it to bitbucket and tried to setup a build job. However, I get the error:
Failed to connect to repository : Command "git ls-remote -h HEAD" returned status code 128:
stdout:
stderr: Host key verification failed.
fatal: The remote end hung up unexpectedly
I tried to remove the domain causing the problem from known_hosts but am still getting this error.
Please advise.
I think I've found a possible solution in this post: http://colonelpanic.net/2011/06/jenkins-on-mac-os-x-git-w-ssh-public-key/
Jenkins on Mac OS X I just finished setting up a build server on Mac
OS X using Jenkins (formerly Hudson). The company I’m working for
(GradeCam) uses git and gitolite for our source control and so I
expected no trouble using Jenkins to build our tools using the git
plugin.
However, I quickly ran into a snag: the source control server is on a
public address and so our source code is not available except via ssh,
and gitolite ssh access uses private key authentication. Well, I’m an
experience unix sysadmin, so that didn’t sound like a big issue —
after all, setting up public key authentication is childs play, right?
Default install
The default installation of Jenkins on Mac OS X (at the time of this
writing) installs a Launch Agent plist to
/Library/LaunchAgents/org.jenkins-ci.plist. This plist file causes
Jenkins to load as user “daemon”, which sounds fine — except that the
home directory for the “daemon” user is /var/root, same as for user
root. This means that the .ssh dir in there will never have the right
permissions for a private key to be used.
Creating a new hidden user
My solution was to create a new “hidden” user for Jenkins to run
under. Following instructions I found on a blog post, I created a user
“jenkins” with a home directory “/Users/Shared/Jenkins/Home”:
sudo dscl . create /Users/jenkins
sudo dscl . create /Users/jenkins PrimaryGroupID 1
sudo dscl . create /Users/jenkins UniqueID 300
sudo dscl . create /Users/jenkins UserShell /bin/bash
sudo dscl . passwd /Users/jenkins $PASSWORD
sudo dscl . create /Users/jenkins home /Users/Shared/Jenkins/Home/
I then stopped Jenkins: “sudo launchctl unload -w
/Library/LaunchAgents/org.jenkins-ci.plist” and edited the plist file
to set the username to jenkins instead of daemon.
“chown -R jenkins: /Users/Shared/Jenkins/Home”
sets the permissions how they need to be, and then “sudo launchctl
load -w /Library/LaunchAgents/org.jenkins-ci.plist” should get you up
and running!
To get git over ssh running, “sudo su – jenkins” to get a console as
the jenkins user and set up the ssh keys and such. Make sure you can
ssh to where you want to go (or even do a test git clone) because you
need to save the keys so it doesn’t ask for them when jenkins tries to
do the clone.
That should do you! Hope it helps someone.

Resources