I have a batch COBOL program which needs input in the form of a flat file. It is working when I FTP a single file to the batch using a software.
Problem is that , in the final solution , many concurrent users are needed to access the batch program together or individually. For example lets say 10 users need to run the batch.
They can FTP all of the files to a shared directory from where the Mainframe can access the file.
Now the problem comes as to
How the Mainframe Job can be triggered?
since there will be 10 or more files , the JOB needs to run each one of them individually and generate a report.
How should the file names be? for example if two files have same name they will get overridden when they are FTP into the shared directory in the first place. On the other hand if the file names are unique , Mainframe will not be able to differentiate between them .
The user will recieve the report through E-Mail its coded in the Batch program, the ID will be present in the input Flat file.
Previously the CICS functionality was done through excel macro(Screen scrapping). The whole point of this exercise is to eliminate the CICS usage to reduce MIPS
Any help is appreciated.
Riffing off what #SaggingRufus said, if you have Control-M for scheduling you can use CTMAPI to set an auto-edit variable to the name of your file and then order a batch job. You could do this via a web service in CICS using the SPOOLWRITE API to submit the job, or you could try FTPing to the JES spool.
#BillWoodger is absolutely correct, get your production scheduling folks and your security folks involved. Don't roll your own architecture, use what your shop has decided is right for it.
Related
I'm currently working on a Rails application (React frontend) that deals with students submitting zipped directories of Ruby projects. The files are stored through Paperclip and S3 on an Assessment model. Currently, in order to grade student submissions, I must download every student file and test with RSpec individually from the command line.
I've improved the workflow through writing a command line script (in Ruby) that will take a directory of Zips, grade them all at once, and store each individual RSpec output in a scores.txt file.
The goal is to allow students to upload their .zips through a portal, and have them be immediately and automatically tested with RSpec. Is there a way to do this?
I've done a lot of searching, and so far can only find advice on how to test live user input (similar to codecademy, or other sites with integrated IDEs).
Sure this is doable.
Unzip the file, load each file and run tests on the defined methods/classes.
But there are at least two things to consider...
Security
A student could overwrite your test method to return only good grades, or random or whatever. Because you're basically loading their code into the same process that grades them. You can, of course, make the method names not obvious but security through obscurity is just crossing your fingers and hope to be lucky (e.g. a student can make a script that uploads your source code somewhere to inspect later)
I'd solve it with a small API that takes the code, the source file, method to call and returns the output. So instead of loading the file in your space - you ask a separate app to produce output, and in your main app would evaluate/grate it.
Consistency of results
i.e. one student could overwrite definition of Object.to_s and others would use it thinking it's the original implementation - thus making their code failing in producing the desired output.
Solution to this one would be to run each test in a separate process (not just thread)
This unfortunatelly brings us back to security - each script will be able to read and alter the code that's accessible by the app that runs it.
So! What to do then?
The easiest thing I can think of right now is to start a container (i.e. docker, rocket or other) and run the code there. An additional benefit would be that you can share the container image with the students, and they can run their code in the same environment making the results as consistent as you can imagine.
The situation right now:
Every Monday morning I manually check Jenkins jobs jUnit results that ran over the weekend, using Project Health plugin I can filter on the timeboxed runs. I then copy paste this table into Excel and go over each test case's output log to see what failed and note down the failure cause. Every weekend has another tab in Excel. All this makes tracability a nightmare and causes time consuming manual labor.
What I am looking for (and hoping that already exists to some degree):
A database that stores all failed tests for all jobs I specify. It parses the output log of a failed test case and based on some regex applies a 'tag' e.g. 'Audio' if a test regarding audio is failing. Since everything is in a database I could make or use a frontend that can apply filters at will.
For example, if I want to see all tests regarding audio failing over the weekend (over multiple jobs and multiple runs) I could run a query that returns all entries with the Audio tag.
I'm OK with manually tagging failed tests and the cause, as well as writing my own frontend, is there a way (Jenkins API perhaps?) to grab the failed tests (jUnit format and Jenkins plugin) and create such a system myself if it does not exist?
A good question. Unfortunately, it is very difficult in Jenkins to get such "meta statistics" that spans several jobs. There is no existing solution for that.
Basically, I see two options for getting what you want:
Post-processing Jenkins-internal data to get the statistics that you need.
Feeding a database on-the-fly with build execution data.
The first option basically means automating the tasks that you do manually right now.
you can use external scripting (Python, Perl,...) to process Jenkins-internal data (via REST or CLI APIs, or directly reading on-disk data)
or you run Groovy scripts internally (which will be faster and more powerful)
It's the most direct way to go. However, depending on the statistics that you need and depending on your requirements regarding data persistance , you may want to go for...
The second option: more flexible and completely decoupled from Jenkins' internal data storage. You could implement it by
introducing a Groovy post-build step for all your jobs
that script parses job results and puts data of interest in a custom, external database
Statistics you'd get from querying that database.
Typically, you'd start with the first option. Once requirements grow, you'd slowly migrate to the second one (e.g., by collecting internal data via explicit post-processing scripts, putting that into a database, and then running queries on it). You'll want to cut this migration phase as short as possible, as it eventually requires the effort of implementing both options.
You may want to have a look at couchdb-statistics. It is far from a perfect fit, but at least seems to do partially what you want to achieve.
My team is having a debate which is better: a windows service or scheduled tasks. We have a server dedicated to running jobs and currently they are all scheduled tasks. Some jobs take files, rename them and place them in other directories on the network. Other jobs extract data from SQL, modify it, and ship it elsewhere. Other jobs ftp files out. There is a lot of variety, but all in all, they are fairly straightforward.
I am partial to having each of these run as a windows service instead of a scheduled task because it is so much easier to monitor a windows service than a scheduled task. Some are diametrically opposed. In the end, none of us have that much experience to provide actual factual comparisons between the two methods. I am looking for some feedback on what other have experienced.
If it runs constantly - windows service.
If it needs to be run at various intervals - scheduled task.
Scheduled Task - When activity to be carried out on some fixed/predefined schedule. It take less memory and resources of OS. Not required installation. It can have UI (eg. Send reminder mail to defaulters)
Windows Service - When a continue monitoring is required. It makes OS busy by consuming more. Require install/uninstallation while changing version. No UI at all (eg. Process a mail as soon as it arrives)
Use them wisely
Sceduling jobs with the build in functionality is a perfectly valid use. You would have to recreate the full functionality in order to create a good service, and unless you want to react to speciffic events, I see no reason to move a nightly job into a service.
Its different when you want to process a file after it was posted in a folder, thats something I would create a service for, thats using the filesystem watcher to monitor a folder.
I think its reinventing the wheel
While there is nothing wrong with using the Task Scheduler, it is itself, a service. But we have the same requirements where I work and we have general purpose program that does several of these jobs. I interpreted your post to say that you would run individual services for each task, I would consider writing a single, database driven (service) program to do all your tasks, and that way, when you add a new one, it is simply a data entry chore, and not a whole new progam to write. If you practice change control, this difference is can be significant. If you have more than a few tasks the effort may be comperable. This approach will also allow you to craft a logging mechanism best suited to your operations.
This is a portion of our requirments document for our task program, to give you an idea of where to start:
This program needs to be database driven.
It needs to run as a windows service.
The program needs to be able to process "jobs" in the following manner:
Jobs need to be able to check for the existence of a source file, and take action based on the existence or not of the source file. (i.e proceed with processing, vs report that the file isn't there vs ignore it because it is not critical that the file isn't there.
Jobs need to be able to copy a file from a source to a target location or
Copy a file from source, to a staging location, perform "processing", and then copy either the original file or a result of the "processing" to the target location or
Copy a file from source, to a staging location, perform "processing", and the processing is the end result.
The sources and destination that jobs might copy to and from can be disparate: UNC, SFTP, FTP, etc.
The "processing", can be, encrypting/decrypting a file, parsing a data file for correct format, feeding the file to the mainframe via terminal emulation, etc., usually implemented by calling a command line passing parameters to an .exe
Jobs need to be able to clean up after themselves, as required. i.e. delete intermediate or original files, copy files to an archive location, etc.
The program needs to be able to determine the success and failure of each phase of a job and take appropriate action which would be logging, and possibly other notification, abort further processing on failure, etc.
Jobs need to be configured to activate at certain set times, or at certain intervals (optionally during certain set hours) i.e. every 15 mins from 9:00 - 5:00.
There needs to be a UI to add new jobs.
There needs to be a button to push to fire off a job as if a timer event had activated it.
The standard Display of the program should show an operator what is going on and whether the program is functioning properly.
All of this is predicated on the premise that it is a given that you write your own software. There are several enterprise task scheduler programs available on the market, as well. Buying off the shelf may be a better solution for you.
I have a single threaded process, which takes an input file and produces an output file (takes file in and file out paths as inputs). I want to use Erlang to create, manage and close multiple instances of this process.
Basically, whenever the client process need to produce the output file, the client connects to the Erlang server with the input and output path - the server initiates a new process - feeds it the paths, and then once the process is done, terminate the process.
I have basic understanding of how gen_server etc. work, but I want to know whether I can use erlang to create and delete instances of an external process? (e.g. a JAR). What library should I look into?
Look at ports. http://www.erlang.org/doc/man/erlang.html#open_port-2
The os:cmd function is probably the closest, see [http://www.erlang.org/doc/man/os.html1. It does assume that your processes run and then finish - the "deleting" part is not covered.
We have a stored procedure written using a CL and RPG program combination. When called locally on the iSeries all is fine. When called externally (for example from a SQL frontend) the RPG program cannot get a hadle on the spool file it produces because the spool file appears under a different (random?) job number and user.
The jobs run as QUSER in the QUSRWRK subsystem but the spool file gets the user id of which the connection was made externally in the connection pool (i.e USERA).
Is there a way of being able reliably to get a handle on the correct sppol file as the job runs (rather than relying on picking the last spool fiel from that queue etc).
If you're running a stored procedure (running in job QZDASOINIT), you will no be able to access the spooled output via the program status data structure. Those spooled files reside in a job named, user/QPRTJOB, where user is the "current user" running the stored procedure. To access the spooled files, run api QSPRILSP to obtain a structure that points you to the spooled file.
Both the behavior and API is well documented in IBM's infocenter.
A server job (e.g., a database server instance for ODBC/JDBC) runs under a system user profile. For stored procs, the system user will usually be QUSER. Objects created within a job are usually owned by the job user.
Server jobs generally perform work on behalf of other users. You tell the server job which user when you establish a connection. (And note that during its lifetime a given server job might work on behalf of many different users.)
Particularly for spooled output, this is a problem because the spooling subsystem has been around longer than we've had the "Web" and before we had significant numbers of users connecting to remote databases. The behavior of switching around from user to user simply isn't part of the spooling subsystem's makeup, nor can a vendor such as IBM determine when a spooled file should be owned by a particular job user or connection user. (And spooling is not a major element of database connections.)
IBM did adapt how spooling is associated output with users by defaulting "switched users" output to collect in a job named QPRTJOB, but it doesn't quite fit with how you want a later RPG program to handle the output.
However, if you create a stored proc that generates spooled output, the proc can choose who owns the output and thereby choose to keep it within the same job. Consider these example CALLs that can be be pasted into the iSeries Navigator 'Run SQL Scripts' function:
call qsys2.qcmdexc ('OVRPRTF FILE(*PRTF) SPLFOWN(*JOB) OVRSCOPE(*JOB)' , 48);
call qsys2.qcmdexc ('DSPMSGD RANGE(CPF9899) MSGF(QCPFMSG) OUTPUT(*PRINT)' , 51);
call qsys2.qcmdexc ('DLTOVR FILE(*PRTF) LVL(*JOB)' , 28);
If you run them as a set, they create spooled output showing the attributes of the message description for CPF9899. If you check afterwards, you should see that QUSER now owns a spooled file named QPMSGD and that it resides within the QZDASOINIT job that's handling your remote database requests. A RPG program within that job can easily find the "*LAST" spooled file in that case. Also, if you delete the first and last CALL and now run just the middle one, you should find that you own the next spooled file.
(QUSER is the IBM default. If your system uses a different user profile, substitute that user for "QUSER".)
Recommendation:
Change your SP to issue an appropriate OVRPRTF command before spooling output that you need to capture in the job and to issue DLTOVR after the output is generated.
You might use commands similar to the ones shown here to create a test procedure. Experiment with different OVRSCOPE() settings and with FILE(*PRTF) or with specifically named files. Also, create output before and after the override commands to see how differently they behave.
Stay aware that the server job might handle a different user after your SP finishes (or a different SP might be called later in the job), so you'll want to be sure that DLTOVR runs. (That's one reason to keep it close to the OVRPRTF.)
I need a bit more information, but I'll make some assumptions. Please clarify if I assumed wrong.
The QUSER in QUSRWRK behavior is correct. You are now running through the SQL server (or similar server). All connections run under these settings.
There are a couple approaches.
1) Assuming that this all runs in one job. Using '*" for the job information should work.
2) The other option is to use RTVJOBA CURUSER(&ME). The current user is the person that is logged in. USER would not work in this case.
If you can modify the RPG program you can retrieve job information from the Program Status Data Structure while the File Information Data Structure has the spool file number from the open feedback area. However I'm not sure the job information will be for the QUSER job (not what you need) or for the USERA job (what you need). The spool file number could be enough of a handle for subsequent Print API calls.
The job itself knows or can find out (see previous answers) so, if all else fails, modify the program to place a message on a queue that provides the information you need. Read it off at your leisure.
.