Reading these emails belatedly, I was finally motivated to do some timing tests.
I have a handy file for the purpose, only 2700 records but I use it a lot so it was in my mind. I wrote PGM1 to allocate the file, start the timer, process all the input, stop the timer and then free the DD and display the results; this way the overhead affects the time as little as possible. I used three methods:
1) Write <n> records at a time to a stem, and drop each record from the stem one at a time.
2) Write <n> records to a stem, then drop the entire stem.
3) Write <n> records to the stack, then parse pull each record.
I wrote PGM2 to call PGM1 eight times and display the average time. I ran PGM2 for 1 record at a time, 5, 20 and *. Here's what I get:
--Read <n> records at a time: 1 5 20 *
Stem, drop each line individually: 0.087968875 0.075949 0.073918375 0.073024125
Stem, drop stem in batches: 0.087492125 0.074802 0.072333375 0.072098625
Stack, using parse pull: 0.089421375 0.07645475 0.073615125 0.07308325
a) As expected, reading one record at a time is slower than *. But there isn't as much difference as I expected, between 20% and 22% more.
b) As expected, dropping the stem as a group between EXECIO statements is faster than doing it one line at a time, but by a much smaller margin.
c) Contrary to my expectations, the stem is very slightly faster than the stack. Not faster enough to outweigh other considerations.
Notice that although there's 20% difference between reading one record at a time and the whole file, most of that difference occurs between one record and five records at a time. That is, 1 rec/read takes 17% more time than 5 rec/read, but 5 rec/read takes only 3½% more time than 20 rec/read and 20 rec/read only 0.3% more than 2700.
***@gmail.com, cell 336 382-7313
/* Even the most marginal NBA player is an absurdly better athlete than an ordinary person. When basketball people say that Grant Long can't shoot, can't pass, can't dribble, what they mean is: He can shoot, pass and dribble better than you, better than anybody you know, better than all but a few hundred people in the world. -from _Why the NBA Isn't As Offensive As You Think_ by Dave Barry */
From: TSO REXX Discussion List [mailto:TSO-***@VM.MARIST.EDU] On Behalf Of Jesse 1 Robinson
Sent: Thursday, July 20, 2017 17:00
IIRC EXECIO for one record at a time was introduced for TSO/E. My VM colleagues had never heard of it. My *impression* was that its purpose was to simplify converting CLIST to REXX. Since CLIST could only read one record at a time, CLIST logic flow was built around that capability.
EXECIO for a whole file in one shot is super-fast. Of course there are memory limitations, but these days virtual is way bigger than it was back then.
From: TSO REXX Discussion List [mailto:TSO-***@VM.MARIST.EDU] On Behalf Of Hobart Spitz
Sent: Thursday, July 20, 2017 12:20 PM
I have not tested it myself, but it's my understanding the EXECIO is horribly slow. Anyone who cares about performance, will avoid "EXECIO 1 ...", and process records in bunches. It may be due to the overhead of the TSO host command interface; I know that ISPF uses it's own CLIST interpreter, probably for that reason. It's too bad that ISPF doesn't do something similar for REXX.
LINEIN() should be available everywhere, and does not suffer from this performance issue. It does not have to go thru the TSO host command interface. Many of the TSO routines (IKJVC441, variable access, is another), are old and slow and have not been updated (or cannot be for compatibility reasons) since the original days of TSO.
--- On Thu, Jul 20, 2017 at 3:10 PM, Bob Bridges <***@gmail.com> wrote:
> I'm sure you're right. But the coding for <n> records at a time is
> more complex, requiring an extra layer; if you do just one record at a
> time, it's simpler. That is, for output; for input the coding
> difference is smaller.
> I'm assuming that doing it one record at a time is ~slightly~ less
> efficient than doing it in batches as you suggest, but only slightly.
> There are two other possibilities, and I just don't know (and haven't
> tested) which of the three is correct:
> 1) Maybe it makes no difference whatsoever; the OS puts the records in
> RAM and sends them to DASD when a full block has been written, and
> trying to batch them up in the program changes the timing either not
> at all or so little as to be unmeasurable. That'd be nice, wouldn't
> it? :)
> 2) Or maybe I'm all off and it makes a huge difference, for some reason.
> But about a decade ago, I vaguely recall that I ~did~ time that a
> little, and decided this isn't true. Mind you, I didn't really time
> it; I ran the program two different ways and felt that they ran in
> about the same amount of time. Probably I was going to stick a timer
> in the program, but got distracted by some other priority and never
> came back to it, but I don't remember.
> Now that you've brought it up, maybe I'll finally get around to trying
> it out.
> -----Original Message-----
> From: TSO REXX Discussion List [mailto:TSO-***@VM.MARIST.EDU] On Behalf Of Jeremy Nicoll
> Sent: Wednesday, July 19, 2017 06:17
> If that works, surely it'd be better to write ten or a hundred or a
> thousand records at a time, and then whatever's left in the final chunk?
> Then some experiment (based on record length & number of records)
> would tell you what a good compromise between far too many chunks and
> not enough would be?
> --- On Tue, 18 Jul 2017, at 21:38, Bob Bridges wrote:
> > I don't know the answer to your question now any more than I did
> > last week, but now that you've said it's for loading into DB2 I do
> > have another idea, although it involves a little extra rewriting. Like this:
> > 1) Write the data one record at a time (EXECIO 1 DISKW) to a dataset
> > 2) Connect with QMF
> > 3) In QMF, run a procedure that loads the raw data into DB2
For TSO-REXX subscribe / signoff / archive access instructions,
send email to ***@VM.MARIST.EDU with the message: INFO TSO-REXX