2

I need to generate a fixed width file with few of the columns in packed decimal format and few of the columns in normal number format. I was able to generate. I zipped the file and passed it on to the mainframe team. They imported it and unzipped the file and converted to EBCDIC. They were able to get the packed decimal columns without any problem but the normal number fields seemed to have messed up and are unreadable. Is there something specific that I need to do while process/zip my file before sending it to mainframe? I am using COMP3 packed decimal. Currently working on Windows XP but the real production will be on RHEL.

Thanks in advance for helping me out. This is urgent.


Edited on 06 June 2011:

This is how it looks when I switch on HEX.

. . . . . . . . . . A . .
333333333326004444
210003166750C0000

The 'A' in the first row has a slight accent so it is not the actual upper case A.

210003166 is the raw decimal. The value of the packed decimal before comp3 conversion is 000000002765000 (we can ignore the leading zeroes if required).


UPDATE 2 : 7th June 2011 This how I am converting creating the file that gets loaded into the mainframe: File contains two columns - Identification number & amount. Identification number doesn't require comp3 conversion and amount requires comp3 conversion. Comp3 conversion is performed at oracle sql end. Here is the query for performing the conversion:

Select nvl(IDENTIFIER,' ') as IDENTIFIER, nvl(utl_raw.cast_to_varchar2(comp3.convert(to_number(AMOUNT))),'0') as AMOUNT from TABLEX where IDENTIFIER = 123456789

After executing the query, I do the following in Java:

String query = "Select nvl(IDENTIFIER,' ') as IDENTIFIER, nvl(utl_raw.cast_to_varchar2(comp3.convert(to_number(AMOUNT))),'0') as AMOUNT from TABLEX where IDENTIFIER = 210003166"; // this is the select query with COMP3 conversion


ResultSet rs = getConnection().createStatement().executeQuery(sb.toString());
sb.delete(0, sb.length()-1);
StringBuffer appendedValue = new StringBuffer (200000);
while(rs.next()){
appendedValue.append(rs.getString("IDENTIFIER"))
.append(rs.getString("AMOUNT"));
}


File toWriteFile = new File("C:/transformedFile.txt");
FileWriter writer = new FileWriter(toWriteFile, true);
writer.write(appendedValue.toString());
//writer.write(System.getProperty(ComponentConstants.LINE_SEPERATOR));
writer.flush();
appendedValue.delete(0, appendedValue.length() -1);

The text file thus generated is manually zipped by a winzip tool and provided to the mainframe team. Mainframe team loads the file into mainframe and browses the file with HEXON.

Now, coming to the conversion of the upper four bits of the zoned decimal, should I be doing it before righting it to the file? Or am I to apply the flipping at the mainframe end? For now, I have done the flipping at java end with the following code:

public static String toZoned(String num) {
    if (num == null) {
        return "";
    }
    String ret = num.trim();

    if (num.equals("") || num.equals("-") || num.equals("+")) {
        // throw ...
        return "";
    }

    char lastChar = ret.substring(ret.length() - 1).charAt(0);
    //System.out.print(ret + " Char - " + lastChar);
    if (lastChar < '0' || lastChar > '9') {
    } else if (num.startsWith("-")) {
        if (lastChar == '0') {
            lastChar = '}';
        } else {
            lastChar = (char) (lastChar + negativeDiff);
        }
        ret = ret.substring(1, ret.length() - 1) + lastChar;

    } else  {
        if (num.startsWith("+")) {
            ret = ret.substring(1);
        }

        if (lastChar == '0') {
            lastChar = '{';
        } else {
            lastChar = (char) (lastChar + positiveDiff);
        }
        ret = ret.substring(0, ret.length() - 1) + lastChar;
    }
    //System.out.print(" - " + lastChar);

    //System.out.println(" -> " + ret);
    return ret;
}

The identifier becomes 21000316F at the java end and that is what gets written to the file. I have passed on the file to mainframe team and awaiting the output with HEXON. Do let me know if I am missing something. Thanks.


UPDATE 3: 9th Jun 2011

Ok I have got mainframe results. I am doing this now.

 public static void main(String[] args) throws FileNotFoundException {
            // TODO Auto-generated method stub
            String myString = new String("210003166");
            byte[] num1 = new byte[16];
            try {
                PackDec.stringToPack("000000002765000",num1,0,15);
                System.out.println("array size: " + num1.length);
            } catch (DecimalOverflowException e1) {
                // TODO Auto-generated catch block
                e1.printStackTrace();
            } catch (DataException e1) {
                // TODO Auto-generated catch block
                e1.printStackTrace();
            } 
            byte[] ebc = null;
            try {
                ebc = myString.getBytes("Cp037");
            } catch (UnsupportedEncodingException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }

            PrintWriter pw = new PrintWriter("C:/transformationTextV1.txt");
            pw.printf("%x%x%x%x%x%x%x%x%x",ebc[0],ebc[1],ebc[2],ebc[3],ebc[4], ebc[5], ebc[6], ebc[7], ebc[8]);
            pw.printf("%x%x%x%x%x%x%x%x%x%x%x%x%x%x%x",num1[0],num1[1],num1[2],num1[3],num1[4], num1[5], num1[6], num1[7],num1[8], num1[9],num1[10], num1[11],num1[12], num1[13], num1[14],num1[15]);
            pw.close();
        }

And I get the following output:

Á.Á.Á.Á.Á.Á.Á.Á.Á.................Ä
63636363636363636333333333333333336444444444444444444444444444444444444444444444
62616060606361666600000000000276503000000000000000000000000000000000000000000000

I must be doing something very wrong!

UPDATE 4: 14th Jun 2011

This query was resolved after using James' suggestion. I am currently using the below code and it gives me the expected output:

    public static void main(String[] args) throws IOException {
        // TODO Auto-generated method stub
        String myString = new String("210003166");
        byte[] num1 = new byte[16];
        try {
            PackDec.stringToPack("02765000",num1,0,8);
        } catch (DecimalOverflowException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        } catch (DataException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        } 
        byte[] ebc = null;
        try {
            ebc = myString.getBytes("Cp037");
        } catch (UnsupportedEncodingException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        FileOutputStream writer = new FileOutputStream("C:/transformedFileV3.txt");
        writer.write(ebc,0,9);
        writer.write(num1,0,8);
        writer.close();
    }
Bill Woodger
  • 12,968
  • 4
  • 38
  • 47
SJoe
  • 319
  • 2
  • 6
  • 14
  • What do you mean by "normal" number format? What do you mean by "messed up"? If they are not packed how is it important that you use COMP3 for the numbers which work? – Howard Jun 02 '11 at 15:45
  • By normal format I mean that I do not need to apply a comp3 conversion on to that column. Assume my first column is 12345678 and then the second column is 87654321, I want to apply comp3 conversion only for the second column and retain the first column as is. This works. I am using a java application to invoke a comp3 conversion program that does the conversion for the column I specify. Then in the mainframe, I see that the unconverted number 12345678 becomes all scrambled and unreadable with some weird special characters. Does this help or am I still not making sense? – SJoe Jun 02 '11 at 16:16
  • If you create a text file on windows containing "12345678", and view it on the mainframe, do you see the same special characters? Those are the EBCDIC control characters with hexadecimal values 0x30-0x39. – Heath Hunnicutt Jun 02 '11 at 16:52
  • @Heath Hunnicutt - I could not look at how it looks at the mainframe end as I have no access to the mainframe system. However, the support person on the mainframe system says that "12345678" become unreadable every time he views it. – SJoe Jun 03 '11 at 16:03
  • @SJoe - Of course it becomes unreadable, but is the unreadable result made of the same-looking characters as the support person sees when viewing your output? – Heath Hunnicutt Jun 03 '11 at 16:26
  • @Heath Hunnicutt - it becomes a series of dots( ..... ) – SJoe Jun 03 '11 at 17:03
  • "I must be doing something wrong..." I think you are having datatype problems. Java uses 2 bytes per character for string variables. The only way you will ever get this to work is with single bytes. Make sure all of the data manipulations you do are with byte arrays. Read your input as a byte array, manipulate it as a byte array and write it as a byte array. Somewhere along the line you are stuffing a byte array into a character string (or the other way round) and it is getting messed up. Find some way to view your output in HEX before sending to the mainframe to ensure it is correct. – NealB Jun 10 '11 at 01:01

4 Answers4

2

As you are coding in Java and you require a mix of EBCDIC and COMP-3 in your output you wiil need to do the unicode to EBCDIC conversion in your own program.

You cannot leave this up to the file transfer utility as it will corrupt your COMP-3 fields.

But luckily you are using Java so its easy using the getBytes method of the string class..

Working Example:

package com.tight.tran;

import java.io.*;

import name.benjaminjwhite.zdecimal.DataException;
import name.benjaminjwhite.zdecimal.DecimalOverflowException;
import name.benjaminjwhite.zdecimal.PackDec;

public class worong {

    /**
     * @param args
     * @throws IOException 
     */
    public static void main(String[] args) throws IOException {
        // TODO Auto-generated method stub
        String myString = new String("210003166");
        byte[] num1 = new byte[16];
        try {
            PackDec.stringToPack("000000002765000",num1,0,15);
            System.out.println("array size: " + num1.length);
        } catch (DecimalOverflowException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        } catch (DataException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        } 
        byte[] ebc = null;
        try {
            ebc = myString.getBytes("Cp037");
        } catch (UnsupportedEncodingException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        FileOutputStream writer = new FileOutputStream("C:/transformedFile.txt");
        writer.write(ebc,0,9);
        writer.write(num1,0,15);
        writer.close();
    }

}

Produces (for me!):

0000000: f2f1 f0f0 f0f3 f1f6 f600 0000 0000 0000  ................
0000010: 0000 0000 2765 000c 0d0a                 ....'e....
James Anderson
  • 27,109
  • 7
  • 50
  • 78
  • @James - Am I to do this while creating the file - at the Java end? Please see my update to my question. Let me know if there is really something wrong. On a seperate note, is there a way to switch off EBCDIC conversion at the mainframe end? I may not be making any sense but the problem seems to occur because my non-packed values are getting corrupted because of this EBCDIC conversion. – SJoe Jun 07 '11 at 11:58
  • @james - And James, this example seems to be using a class import name.benjaminjwhite.zdecimal.PackDec. Can you please share the same? – SJoe Jun 07 '11 at 12:01
  • 1
    google for Zdecimal, its available from IBM and SourceForge. But as you already have the code for converting numbers to packed you may as well stck with whatever method you are currently using. The imporant bit is ' mystring.getBytes("Cp037")' – James Anderson Jun 08 '11 at 01:52
  • @James - your sample helped. Have recreated the files. Waiting on the mainframe to get back with the results. – SJoe Jun 08 '11 at 15:31
  • @James - please see the update to my question with the changes and output thus obtained. Something is wrong. – SJoe Jun 09 '11 at 17:32
  • 1
    @SJoe -- my hex print output was confusing you. Ammended example to write bytes direct to file. – James Anderson Jun 10 '11 at 02:31
  • @James - it worked. Thanks a million!!! I used the code provided by you and I get the expected result. – SJoe Jun 10 '11 at 10:23
1

"... converted to EBCDIC..." may be part of the problem.

Unless the mainframe conversion process is "aware" of the record layout it is working with (ie. which columns contain binary, packed and/or character data), it is going to mess something up because the mapping process is format dependant.

You have indicated the COMP-3 data are ok, I am willing to bet that either the "converted to EBCDIC" doesn't do anything, or it is performing some sort of ASCII to COMP-3 conversion on all of your data - thus messing up non COMP-3 data.

Once you get to the mainframe, this is what you should see:

COMP-3 - each byte contains 2 digits except the last (right most, least significant). The least significant byte contains only 1 decimal digit in the upper 4 bits and the sign field in the lower 4 bits. Each decimal digit is recorded in hex (eg. 5 = B'0101')

Zoned Decimal (normal numbers) - each byte contains 1 decimal digit. The upper four bits should always contain HEX F, except possibly the least most significant byte where the upper 4 bits may contain the sign and the lower 4 bits a digit. The 4 bit digit is recored in hex (eg. 5 = B'0101')

You need to see what the un-zipped converted data look like on the mainframe. Ask someone to "BROWSE" your file on the mainframe with "HEX ON" so you can see what the actual HEX content of your file is. From there you should be able to figure out what sort hoops and loops you need to jump through to make this work.

Here are a couple of links that may be of help to you:

Update: If the mainframe guys can see the correct digits when browsing with "HEX ON" then there are two possible problems:

  • Digit is stored in the wrong nibble. The digit should be visible in the lower 4 bits. If it is in the upper 4 bits, that is definitely a problem.
  • The non-digit nibble (upper 4 bits) does not contain HEX 'F' or valid sign value. Unsigned digits always contain HEX 'F' in the upper 4 bits of the byte. If the number is signed (eg. PIC S9(4) - or something like that), the upper 4 bits of the least most significant digit (last one) should contain HEX 'C' or 'D'.

Here is a bit of a screen shot of what BROWSE with 'HEX ON' should look like:

   File  Edit  Edit_Settings  Menu  Utilities  Compilers  Test  Help            

 VIEW       USERID.TEST.DATA - 01.99                        Columns 00001 00072 
  Command ===>                                                  Scroll ===> CSR  
  ****** ***************************** Top of Data ******************************  
 000001 0123456789                                                              
        FFFFFFFFFF44444444444444444444444444444444444444444444444444444444444444  
        012345678900000000000000000000000000000000000000000000000000000000000000  
 ------------------------------------------------------------------------------   
  000002  |¬?"±°                                                              
        012345678944444444444444444444444444444444444444444444444444444444444444  
        FFFFFFFFF000000000000000000000000000000000000000000000000000000000000000  
 ------------------------------------------------------------------------------   
  000003  àíÃÏhr                                                              
        012345678944444444444444444444444444444444444444444444444444444444444444  
        012345678900000000000000000000000000000000000000000000000000000000000000   
 ------------------------------------------------------------------------------    

The lines beginning with '000001', '000002' and '000003' shows 'plain' text. the two lines below each of them show the HEX representation of the character above it. The first line of HEX shows the upper 4 bits, the second line the lower 4 bits.

  • Line 1 contains the number '0123456789' followed by blank spaces (HEX 40).
  • Line 2 shows junk because the upper and lower nibbles are flipped. The exact silly character is just a matter of code page selection so do not get carried away with what you see.
  • Line 3 shows similar junk because both upper and lower nibbles contain a digit.

Line '000001' is the sort of thing you should see for unsigned zoned decimal numbers on an IBM mainframe using EBCDIC (single byte character set).

UPDATE 2

You added a HEX display to your question on June 6th. I think maybe there were a couple of formatting issues. If this is what you were trying to display, the following discussion might be of help to you:

..........A..
33333333326004444
210003166750C0000

You noted that this is a display of two "numbers":

  • 210003166 in Zoned Decimal
  • 000000002765000 in COMP-3

This is what an IBM mainframe would be expecting:

210003166    :Á :  <-- Display character  
FFFFFFFFF00002600  <-- Upper 4 bits of each byte  
2100031660000750C  <-- Lower 4 bits of each byte  

Notice the differences between what you have and the above:

  • The upper 4 bits of the Zoned Decimal data in your display contain a HEX '3', they should contain a HEx 'F'. The lower 4 bits contain the expected digit. Get those upper 4 bits fixed and you should be good to go. BTW... it looks to me that whatever 'conversion' you have attempted to Zoned Decimal is having no affect. The bit patterns you have for each digit in the Zoned Decimal correspond to digits in the ASCII character set.
  • In the COMP-3 field you indicated that the leading zeros could be truncated. Sorry, but they are either part of the number or they are not! My display above includes leading zeros. Your display appears to have truncated leading zeros and then padded trailing bytes with spaces (HEX 40). This won't work! COMP-3 fields are defined with a fixed number of digits and all digits must be represented - that means leading zeros are required to fill out the high order digits of each number.

The Zoned Decimal fix should be pretty easy... The COMP-3 fix is probably just a matter of not stripping leading zeros (otherwise it looks pretty good).

UPDATE 3...

How do you flip the 4 high order bits? I got the impression somewhere along the line that you might be doing your conversion via a Java program. I, unfortunately, am a COBOL programmer, but I'll take a shot at it (don't laugh)...

Based on what I have seen here, all you need to do is take each ASCII digit and flip the high 4 bits to HEX F and the result will be the equivalent unsighed Zoned Decimal EBCDIC digit. Try something like...

public static byte AsciiToZonedDecimal(byte b) {
        //flip upper 4 bits to Hex F... 
        return (byte)(b | 0xF0)
};        

Apply the above to each ASCII digit and the result should be an unsigned EBCDIC Zoned Decimal number.

UPDATE 4...

At this point the answers provided by James Anderson should put you on the right track.

James pointed you to name.benjaminjwhite.zdecimal and this looks like it has all the Java classes you need to convert your data. The StringToZone method should be able to convert the IDENTIFIER string you get back from Oracle into a byte array that you then append to the output file.

I am not very familiar with Java but I believe Java Strings are stored internally as Unicode Characters which are 16 bits long. The EBCDIC characters you are trying to create are only 8 bits long. Given this, you might be better off writting to the output file using byte arrays (as opposed to strings). Just a hunch from a non Java programmer.

The toZoned method in your question above appears to only concern itself with the first and last characters of the string. Part of the problem is that each and every character needs to be converted - the 4 upper bits of each byte, except possibly the last, needs to be patched to contain Hex F. The lower 4 bits contain one digit.

BTW... You can pick up the source for this Java utility class at: http://www.benjaminjwhite.name/zdecimal

NealB
  • 16,670
  • 2
  • 39
  • 60
  • Thank you for these inputs. I am novice when it comes to Mainframe so I am going to pass this on to the guy who works on mainframe. The mainframe guy feels that my zipping process could be the issue. An additional information - if I send the file without zipping to mainframe, the zoned decimals (normal numbers) appear okay in mainframe and comp 3 converted numbers get muddled up. That is why he suspects that zipping could be causing the trouble. – SJoe Jun 02 '11 at 21:29
  • @NealB - When they switch on "HEX ON" they can see the number "12345678" but when they switch it off, it become dots. I am yet to recieve the exact representation in writing. But this is just the initial information I recieved. Do you think this can help in determining anything at all? – SJoe Jun 03 '11 at 14:34
  • 1
    @SJoe See update above... Never mind what the display looks like (ie. dots), this tells you next to nothing. Find out what the HEX values (ie. bit patterns) are and go from there. I am still a bit amazed that the COMP-3 data come up ok but the zoned decimals don't. – NealB Jun 03 '11 at 23:32
  • @NealB - Thanks. I will have asked the guy to share the screenshot with me. Hope I can interpret it well. Is there a way I share a screenshot here? How do I attach it here? – SJoe Jun 04 '11 at 22:09
  • @NealB - that means, even if it looks goofed up, if the HEX values are shown right, the values should be right and I need not worry about the junk characters (in my case just dots)? – SJoe Jun 04 '11 at 22:11
  • @NealB - I managed to get a screenshot of the HEX ON output. Refer to the update in my question. – SJoe Jun 06 '11 at 15:02
  • @NealB - I can fix the leading zero issue for COMP3. Not a problem. But how to fix the upper 4 bits of Zoned Decimal? Any pointers would be great. Thanks Neal. – SJoe Jun 06 '11 at 17:19
  • @NealB - Thanks Neal. I have updated my question, please let me know if I am missing out anything. – SJoe Jun 07 '11 at 12:02
  • @NealB, thanks for the links. It helped. I have recreated the files as per James & your recommendation. I have passed it on to the mainframe team to see the results... fingers crossed. – SJoe Jun 08 '11 at 15:30
  • @NealB - please see the update to my question with the result obtained after making the changes. – SJoe Jun 09 '11 at 17:32
1

"They were able to get the packed decimal columns without any problem but the normal number fields seemed to have messed up " would seem to indicate that they did not translate ASCII to EBCDIC.

ASCII zero x'30' should translate to EBCDIC zero x'F0'. If this was not done then (depending on the EBCDIC code page) then x'30' does not map to a valid character on most EBCDIC displays.

However even if they did translate you will have different problem as all or some of your COMP-3 data will be corrupted. The simple translate programs have no way to distinguish between character and comp-3 so they will convert a number such as x'00303C' to x'00F06E' which will cause any mainframe program to bomb out with the dreaded "0C7 Decimal Arithmetic Exception" ( culturally equivalent to "StackOverflow").

So basically you are in a lose/lose situation. I would suggest you ditch the packed decimals and use plain ASCII characters for your numbers.

The zipping should not cause you a problem, except, the file transfer utility was probably doing ASCII to EBCDIC on the plain text file, but, not on the zipped file.

James Anderson
  • 27,109
  • 7
  • 50
  • 78
  • "ditch the packed decimals and use plain ASCII characters for your numbers." is not an option because this a migration from legacy system and many downstream applications (no one knows how many because they are at the customer end) will be affected if we removed packed decimal. SO we need to somehow, find a solution. I can ask them to check if they did or did not translate ASCII to EBCDIC. But you say that even this would corrupt the packed decimal structure... looks like roadblock, especially when I have no clue about mainframe systems. – SJoe Jun 03 '11 at 07:20
  • In taht case you will need to do the ASCII -> EBCDIC conversion field by field yourself. Which language is your program coded in? – James Anderson Jun 06 '11 at 01:39
  • I have used Java & SQL to convert decimals to pack decimals. I do COMP3 conversion. EBCDIC conversion of file happens in mainframe. – SJoe Jun 06 '11 at 07:47
  • EBCDIC conversion can't happen on the mainframe (or more accurately during file transfer to the mainframe) without corrupting your COMP-3 fields. See my second posting for a how-to. – James Anderson Jun 06 '11 at 09:02
0

It sounds like the problem is in the EBCDIC conversion. The packed decimal would use characters as byte values, and isn't subject to the transliterations EBCDIC <-> ASCII.

If they see control characters (or square markers on Windows), then they may be viewing ASCII data as EBCDIC.

If they see " ñòóôõö øù" in place of "0123456789" then they are viewing EBCDIC characters in a viewer using ANSI, or extended ASCII.

Heath Hunnicutt
  • 18,667
  • 3
  • 39
  • 62