Monday 27 March 2017

ORA-00600: internal error code multiple time and struggled to debug the issues

I have faced ORA-00600: internal error code multiple time and  struggled to debug the issues,
so I thought to share this info to all, so that it will be useful if anyone face similar issues.

Generic internal error code message will be display like “ORA-00600 internal error code, arguments:
[string], [string], [string], [string], [string], [string], [string], [string]”.
The first argument is the internal message number. Other arguments are various numbers,
 names, and character strings. The numbers may change meanings between different versions of Oracle.

Causes of this error message include:
- timeouts
- file corruption
- failed data checks in memory
- hardware, memory, or I/O errors
- incorrectly restored files

If you are facing this error we have to check in trace file, alert logs,
Metalink account or we have to do some trial and error methods to resolve the issues,
 I have don’t trial and error methods to resolve below errors because I didn’t get any proper
 solution from any of the sites, most of the sites are given solutions like it is oracle bug ,
we have to check in Metalink or raise a ticket to support.

Below are the internal code errors I have faced.

Error 1:  SQL Error: ORA-00600: internal error code, arguments: [kxtotolc_lobopt], [], [], [], [], [], [], [], [], [], [], []
00600. 00000 -  "internal error code, arguments: [%s], [%s], [%s], [%s], [%s], [%s], [%s], [%s]"
*Cause:    This is the generic internal error number for Oracle program
           exceptions.             This indicates that a process has encountered an
           exceptional condition.
*Action:   Report as a bug - the first argument is the internal error number


Root Cause:
If we are inserting/updating CLOB column value as null using merge statement, and merge statement table has trigger,
 same null CLOB column value if we try to insert/update CLOB value in another table using trigger then we will get the above error,
 also session is automatically closed, we got this error in both Oracle 11g and 12c.

Example :

1) create table test_clob_merge(sno number, c_clob clob);

2) create table test_clob_merge_stg as select * from test_clob_merge;

3) Create a trigger on test_clob_merge table and insert data in staging table test_clob_merge_stg using trriger

CREATE OR REPLACE trigger test_clob_merge_trg AFTER INSERT ON test_clob_merge
FOR EACH ROW
DECLARE
BEGIN
INSERT INTO test_clob_merge_stg VALUES(:new.sno,:new.c_clob);
EXCEPTION
WHEN others THEN
NULL;
END;
/
4) Run the below query.
MERGE INTO test_clob_merge a USING (
select 1 AS sno, null AS C_CLOB from dual) b
ON (a.sno = b.sno)
WHEN MATCHED THEN
UPDATE SET C_CLOB = b.C_CLOB
WHEN NOT MATCHED THEN
INSERT (a.sno,a.C_CLOB) VALUES (b.sno,b.C_CLOB);


Error 2: SQL Error: ORA-00600: internal error code, arguments: [15851], [3], [3499], [2], [1], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [15851], [3], [3499], [2], [1], [], [], [], [], [], [], []
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at "KAISER12C.MS_CCM_UTILITIES", line 2866
ORA-04088: error during execution of trigger 'KAISER12C.MS_CCM_TI_INVESTIGATION'
00600. 00000 -  "internal error code, arguments: [%s], [%s], [%s], [%s], [%s], [%s], [%s], [%s]"
*Cause:    This is the generic internal error number for Oracle program
           exceptions.            This indicates that a process has encountered an
           exceptional condition.
*Action:   Report as a bug - the first argument is the internal error number

Root Cause : one of the function used in trigger has variable size issue due to this we got above error,
 this trigger created on merge statement table used in “_P” pacakage. This error we will not track in any
of the error log tables until if you run each and every statement outside of the package, we got socket error in si_application_error_log.

Error 3 : SQL Error: ORA-00600: internal error code, arguments: [13013], [5001], [158796], [30437895], [0], [30437895], [17], [], [], [], [], []
We have created index on one table to improve performance of query, on same table we are updating data in some stages.
It worked fine and improved performance  in local instances and customer testing instance, but we got above error
 in UAT after applying patch, whenever they has done update statement stage flow. After 3 days of debugging
 we suspect some data blocks are corrupted in UAT due that we got the error and then we dropped the newly index and check the flow, it worked fine.

We got the below update statement highlighted in trace file with error message, it helped us to debug the issue
and also we will get some more information in alert log for any internal error.

UPDATE MS_ITC_AUDIT_V SET STATUS = '38', AUDIT_STATUS = '38' WHERE AUDIT_ID = :B1
AND INSTANCE_ID = (SELECT MAX(INSTANCE_ID) FROM MS_ITC_AUDIT_V WHERE AUDIT_ID = :B1 )