I am using WriteToBigQuery in a beam python pipeline like this:
beam.io.gcp.bigquery.WriteToBigQuery(
table_id,
schema=table_schema,
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
write_disposition=beam.io.BigQueryDisposition.WRITE_EMPTY,
method='STREAMING_INSERTS')
But when I run the pipeline, I am seeing this erorr, which occurred after I fetched and rebased from the beam GitHub master branch today.
"ValueError: Write disposition WRITE_EMPTY is not supported for streaming inserts to BigQuery"
Looks like a recent change, intentionally disabled this due to a bug. So write_disposition=beam.io.BigQueryDisposition.WRITE_EMPTY is no longer allowed with method='STREAMING_INSERTS'
The fix is to now use write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND which should give the same behaviour.
beam.io.gcp.bigquery.WriteToBigQuery(
table_id,
schema=table_schema,
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
method='STREAMING_INSERTS')
I want to list all available plugins names and their short names using command line option, so that I could automate required plugin installation through jenkins command line.
Kindly advise. Thanks
So far I tried to find answer on same however I got answer only for how to list installed plugins, not for all available plugins.
I've found this link http://updates.jenkins-ci.org/download/plugins/ which lists all plugins but with their short names only
You were so close! The LAYOUT is detailed here. The information is nearby for you to parse, hopefully I got it right.
http://updates.jenkins-ci.org/download/plugins/ is indeed the location of the plugins, with the actual plugin versions sitting inside each folder.
https://updates.jenkins.io/ is the root level. You will find the list of all plugins and details at plugin-versions.json.
update-center.js, update-center.json, and update-center.json.html contain actual update center metadata in JSON, JSONP, and HTML format respectively. You can parse the json to pull everything you are looking for. There are also lists for the documentation url and the release history, as well as the updates.
This is where it's nuanced; there's stable (ie:LTS) or latest (ie:weekly) and major releases of each. Each one has its own sublist, depending on minimum version and compatibility.
Plugin Selection
Since none of this tells you what the plugins actually do, the best thing is to choose your plugins at https://plugins.jenkins.io/. Clicking on any plugin (eg: mailer) reveals a header block with details:
Mailer 1.23
Minimum Jenkins requirement: 1.642.3
ID: mailer
The ID is the short name you are looking for. Go through and find the plugins you want to use and that's your list. Don't worry about the dependencies.
About Plugin Management
Even on a standalone instance, I use a modified script of Docker install_plugins.sh to generate the full list of plugins to install .
Update 2021: As part of GSOC 2019 and refined in GSOC 2020, a new and quite functional Plugin Installation Manager CLI Tool has been introduced to replace all non-GUI plugin mgmt tools, including inatall_plugins.sh. Achieves similar results.
You can examine the outputs or use the groovy script that follows to simplify your "must have" list. Also, as dependency updates happen all the time, I also generate a list of actual installed updates if I need to reapply identically to a different instance rather than from my curated list. My curated list is ~45 plugins, with over 115 getting installed.
eg: workflow-api includes [workflow-scm-step] which includes [git, subversion], so no need to specify git. But you want to know which version you got. Occasionally you may need to explicitly add a dependency to get the latest to avoid a defect, per JENKINS-54018, plugins which were split from Jenkins.
println "Jenkins Instance : " + Jenkins.getInstance().getComputer('').getHostName() + " - " + Jenkins.getInstance().getRootUrl()
println "Installed Plugins: "
println "=================="
Jenkins.instance.pluginManager.plugins.sort(false) { a, b -> a.getShortName().toLowerCase() <=> b.getShortName().toLowerCase()}.each { plugin ->
println "${plugin.getShortName()}:${plugin.getVersion()} | ${plugin.getDisplayName()} "
}
println""
println "Plugins Dependency tree (...: dependencies; +++: dependants) :"
println "======================="
Jenkins.instance.pluginManager.plugins.sort(false) { a, b -> a.getShortName().toLowerCase() <=> b.getShortName().toLowerCase()}.each { plugin ->
println "${plugin.getShortName()}:${plugin.getVersion()} | ${plugin.getDisplayName()} "
println "+++ ${plugin.getDependants()}"
println "... ${plugin.getDependencies()}"
println ''
}
return
I have Jenkins (v2.163) installed and I'm using the plugins email-ext as well as the token macro plugin.
The documentation page of the token macro plugin (https://wiki.jenkins.io/display/JENKINS/Token+Macro+Plugin) tells me that a couple of plugins (that I'm using) like PMD, Warnings, Checkstyle, etc. produce tokens which are also listed on the project configuration page when I click on "Content Token Reference".
Some of the tokens I like to use are CHECKSTYLE_COUNT or PMD_COUNT.
My mail-content for the email-notification looks like this:
Branch: ${GIT_BRANCH}
$DEFAULT_CONTENT
Checkstyle issues: ${CHECKSTYLE_COUNT} (${CHECKSTYLE_NEW} new, ${CHECKSTYLE_FIXED} fixed)
Duplicate code issues: ${DRY_COUNT} (${DRY_NEW} new, ${DRY_FIXED} fixed)
Mess Detector issues: ${PMD_COUNT} (${PMD_NEW} new, ${PMD_FIXED} fixed)
Failed tests:
${FAILED_TESTS}
But all token that are supposed to be filled be the corresponding plugins stay empty (while others are filled), so the resulting mail looks like this:
Branch: origin/stage
PROJECT - Build # 1 - Successful:
Check console output at http(...) to view the results.
Checkstyle issues: ( new, fixed)
Duplicate code issues: ( new, fixed)
Mess Detector issues: ( new, fixed)
Failed tests:
No tests ran.
Am I missing a specific configuration for the tokens to be set?
There are lots of Jenkins pipelines wrappers that do add extra functionality to pipeline and I am wondering if there is a way to write pipelines that do activate these wrappers when the plugins providing them are available but that will also work when the plugins are not available.
To explain this better I do have two examples of wrapper plugins that are very useful but even so, we should not break the pipelines if the plugins providing them are not available. There are too many cases when plugins can be temporary unavailable for some time (days, weeks) and we don't really want to break continuous integration because some optional functionality is broken.
timestamper - which adds timestamps to console output
ansicolor - which enabled output of ANSI color escapes on console output
Current usage pattern, one that does not degrade gracefully:
timestamps {
wrap([$class: 'AnsiColorBuildWrapper', 'colorMapName': 'XTerm', 'defaultFg': 1, 'defaultBg': 2]) {
...
}
}
Why doesn't Nagios have a check_memory as part of the initial set of plugins that come by default? Like check_http, check_load etc?
Thanks!
According to Nagios Plugins the plugins in that project need to meet certain criteria, here is a statement from the FAQ:
There are two types of plugins included in the distribution:
Plugins that are officially maintained by the nagios plugin
developer team
Plugins that are distributed in the contrib directory
Sooner or later all the contrib plugins should be hosted at the
official 3rd Party Nagios plugin page Nagios Exchange
Contrib plugins have to be installed manually. They are not affected
by the build process.
Here is a list of the actively maintained Plugins:
check_apt, check_breeze, check_by_ssh, check_clamd, check_cluster,
check_dhcp, check_dig, check_disk, check_disk_smb, check_dns,
check_dummy, check_file_age, check_flexlm, check_fping, check_ftp,
check_game, check_hpjd, check_http, check_icmp, check_ide_smart,
check_ifoperstatus, check_ifstatus, check_imap, check_ircd,
check_jabber, check_ldap, check_load, check_log, check_mailq,
check_mrtg, check_mrtgtraf, check_mssql, check_mysql,
check_mysql_query, check_nagios, check_netdns, check_nntp,
check_nntps, check_nt, check_ntp_peer, check_ntp_time, check_nwstat,
check_oracle, check_overcr, check_pgsql, check_ping, check_pop,
check_procs, check_radius, check_real, check_rpc, check_sensors,
check_simap, check_smtp, check_snmp, check_spop, check_ssh,
check_ssmtp, check_swap, check_tcp, check_time, check_udp, check_ups,
check_users, check_wave
And these plugins are currently distributed in the contrib directory:
check_adptraid, check_apache, check_apc_ups, check_appletalk,
check_arping, check_asterisk, check_axis, check_backup,
check_bgpstate, check_breeze, check_cluster, check_cluster2,
check_compaq_insight, check_cpqarray, check_digitemp,
check_dlswcircuit, check_dns_random, check_email_loop,
check_fan_cpq_present, check_fan_fsc_present, check_flexlm,
check_frontpage, check_hltherm, check_hprsc,
check_http-with-client-certificate, check_hw,
check_ica_master_browser, check_ica_metaframe_pub_apps,
check_ica_program_neigbourhood, check_inodes, check_inodes-freebsd,
check_ipxping, check_javaproc, check_joy, check_linux_raid,
check_lmmon, check_log2, check_lotus, check_maxchannels,
check_maxwanstate, check_mem, check_ms_spooler, check_mssql,
check_nagios, check_nagios_db, check_nagios_db_pg, check_netapp,
check_nmap, check_oracle_instance, check_oracle_tbs,
check_ora_table_space, check_pcpmetric, check_pfstate, check_qmailq,
check_rbl, check_remote_nagios_status, check_rrd_data, check_sap,
check_smart, check_smb, check_snmp_disk_monitor, check_snmp_printer,
check_snmp_process_monitor, check_snmp_procs, check_sockets,
check_temp_cpq, check_temp_fsc, check_timeout, check_traceroute,
check_traceroute-pure_perl, check_uptime, check_vcs, check_wave,
check_wins
As you can see there isn't an official check_memory plugin included in the maintained plugins, however you can find plugins that do support what you are looking for on Naigos Exchange.
if you install nagios-snmp-plugins from http://nagios.manubulon.com/ then you should have check_snmp_mem.pl and that will do the memory monitoring