Month: March 2014

ETL Testing and Teradata Control Framework widely called as TCF Framework

Posted on

Hello All,

Hope you have tested yourself on ETL Testing skills @ theoretical level at least. In this post let me explain about the Teradata Control Framework and the significance of testing role in it.

I am thankful to all the viewers from 52 countries and I request you people to comment how my blog is useful to you guys and is it Good / Bad / Ugly ?

The basic difference between conventional ETL Framework and TCF framework is, TCF is fully automated. Everything that we wanted will be given by a macro written in Excel sheet. TCF is very much helpful for all the developers because it reduces the coding effort.

TCF has three patterns and based on our requirement we need to choose which pattern will fit for our solution.

  1.        Full Apply Patter
  2.        Delta Apply Pattern
  3.        Insert & Append patter

The solution architect will decide what pattern they want to follow and cascade them to Data designers.

  •   If your target tables are not time variant and you are going to have only inserts then Full Apply    pattern will be applicable.
  •   If your target table is time variant then Delta Apply pattern will be applicable
  •   Insert & Append will be applicable for the target tables where you will not do delta, just append the new values upon the old values and just inserts

We as tester’s needs to understand the patterns and test the patterns are working as expected. We have Oracle control tables to maintain and control the ETL process likewise we have many control tables in TCF to maintain and control the ETL Process.

We have around 6 columns will be loaded by this TCF control tables in the matter of maintain the TCF process. We as testers need to verify these column values are populated from corresponding control tables.

In conventional ETL architecture we have [File to Staging table to Intermediate table to Target table], and we will maintain VARCHAR in all Staging and Intermediate tables but in TCF we will have [File to Staging table-1 to Source Image table to Intermediate table to target table]. In staging table we will do the Type Casting – changing the Data Types as per the Target table. So in TCF except Staging tables we will not keep all the columns as VARCHAR. As testers we need to verify the data type conversion is achieved as per the specification. These Data typing SQL are part of the TCF and will be generated using the TCF macros.

Unknown records are now will be updated in the Source Image tables only, so from Source Image table to IM table transformation, we do not want to verify for Null, Blank, Space, 0s.

  1.  Source Image table DDLs are generated by the Excel macro, so we need to check all the data types and the length of any columns are as per specifications.
  2.  We need to ensure the ETL is working as per the Pattern chosen – like we chose Fully Apply and if  the records are loaded for Delta Apply then it’s a mistake.
  3.  In conventional staging tables we do not keep historical records, but if you would have chosen delta apply pattern then Source Image tables will have Expiry and Inserts.
  4.  In TCF, source Image tables the copy records are treated as Delta records in Delta Apply patterns
  5.  If you are testing Soft Delete (Logical Delete) then please refer a column called – Record Delete Flag = 1
  6. In conventional ETL, if you do logical delete then you will keep Expiry Date= ETL Date – 1, but in TCF we will have the record active based on  EFFT_D and EXPY_D by record deleted flag will be set to 1
  7.  Date and Time validations will be done in Source Image tables only, if you send invalid Date or Time the Type Casting SQLs will load as Null in source image tables.
  8. As the Source Image tables are not SET tables, we need to run queries to get the duplicates out of it.
  9.    We cannot use TCF to load the data to Existing tables

Cheers – Asik

 

Advertisements

ETL Testing Roll Back post – Ask yourself and get ready to move to ETL Testing world

Posted on Updated on

Hello Folks

It had been long time I posted something through my Blog L , I’m really sorry for that 😦

jobinterview

I have met a person unknowingly in a common place, we were discussing about ETL testing and other stuffs and I told him I’m writing blogs about ETL Testing.  He was surprised and said that he is one visitor of my Blog. I am happy that I have met a unknown person who reads and benefited with my blog.

I have got few requests ‘Can you please share some of Interview Q & A?’

I replied them

“If you have hands on experience and if you would have mentioned in your CV about you skills and responsibilities then why you wanted to know about Q&A in ETL Testing”

Guys who are trying in ETL Testing domain, are thinking that if they know how to write SQL can clear the Interview and can get the job. Folks that’s not possible. I request you to think that ETL testing is not a different entity; your job is testing on whatever technology it is.

If you are taking an interview, please go through your CV and be strong on what you have mentioned. If you are asked other than what you have mentioned then please raise your hand ‘I’m sorry, I haven’t worked in it’

Ok let me come subject what I wanted to discuss here!!!

Here I’m going ask questions to myself to test my technical capability, if I win myself then I can win anywhere in the Globe, right guys?

So try to ask as much questions as possible to yourself!!!!

Q1

INTRV: Asik, Can you please let us know your roles and responsibilities in your previous project?

CAND: Hello, I’m from here and blaw, blaw, blaw………

Q2

INTRV: Asik, How you can scale yourself in writing in SQL from 1-5?

CAND:  [—-Fill your Best Answer here——]

Q3

INTRV: Asik, What all are the tools that you used in your ETL Projects?

CAND: [—-Fill your Best Answer here——]

Q4

INTRV: Asik, Can you please explain me the ETL architecture?

CAND: [—-Fill your Best Answer here——]

Q5

INTRV: What all are the Joins that you have worked so far? And can you please write down joins using two tables T1 and T2?

CAND: [—-Fill your Best Answer here——]

Q6

INTRV: Asik, what is the difference between Inner join and Outer Join?

CAND: [—-Fill your Best Answer here——]

Q7

INTRV: Asik, can you please explain me about the significance of Data Types in ETL Testing?

CAND: [—-Fill your Best Answer here——]

Q8

INTRV: Asik, I have a column as VARCHAR from source but corresponding target column is INTERGET, how you can write a query to check the target column?

CAND: [—-Fill your Best Answer here——]

Q9

INTRV: Asik, what is the significance of the Staging tables in the ETL loads?

CAND: [—-Fill your Best Answer here——]

Q10

INTRV: Asik, what is the difference between, copy records and duplicate records?

CAND: [—-Fill your Best Answer here——]

Q11

INTRV: Asik, what is the significance of Oracle Control tables in ETL Testing?

CAND: [—-Fill your Best Answer here——]

Q12

INTRV: Asik, what does it mean if you get a NULL, Blank, Space or 0 from Source?

CAND: [—-Fill your Best Answer here——]

Q13

INTRV: Asik, I have a VARCHAR column from source, this column can allow 0, suppose if source is sending 0000 instead of 0 , and the rule is we should not allow this record into target table. Does our SQL will stop this record if we write the condition as [Column <> 0]

CAND: [—-Fill your Best Answer here——]

Q14

INTRV: Asik, what is the difference between the Primary Index and Primary Key?

CAND: [—-Fill your Best Answer here——]

Q15

INTRV: Asik, what could be consequence if the reconciliation failed in the Day_01 load or Day_02 load?

CAND: [—-Fill your Best Answer here——]

Q16

INTRV: Asik, Why you wanted to test the DDL of any table?

CAND: [—-Fill your Best Answer here——]

Q17

INTRV: Asik, what is the difference between Data Definition Language and View Definition?

CAND: [—-Fill your Best Answer here——]

Q18

INTRV: Asik, as a tester what all are the access you required? And do you know the SQL query syntax to get the access?

CAND: [—-Fill your Best Answer here——]

 Q19

INTRV: Asik, what is the difference between DB level access, Table level access and view level access?

CAND: [—-Fill your Best Answer here——]

Q20

INTRV: Asik, What all are the steps you take to validate the Source Files? What is significance of Source File Names, Header, Trailer and Detailed Records?

CAND: [—-Fill your Best Answer here——]

Q21

INTRV: Asik, if you could not able to open a file manually by double click due to the file size? How you will access the file and look at the records?

CAND: [—-Fill your Best Answer here——]

Q22

INTRV: Asik, What is the High Severity defect that you raised so far in any ETL project and why you have classified as Sev-01?

CAND: [—-Fill your Best Answer here——]

Q22

INTRV: Asik, What is the High Priority defect that you raised so far in any ETL project and why you have classified as Sev-01?

CAND: [—-Fill your Best Answer here——]

Q23

INTRV: Asik, Explain about Inserts, Updates, Logical Deletes in any given ETL Projects?

CAND: [—-Fill your Best Answer here——]

Q24

INTRV: Asik, Why and what is necessity of Reference tables in any ETL Projects?

CAND: [—-Fill your Best Answer here——]

 Q25

INTRV: Asik, Can you modify Reference table data if you have Update access to the reference tables?

CAND: [—-Fill your Best Answer here——]

Q26

INTRV: Asik, What all are the Steps by which you can achieve the test data for a given scenarios?

CAND: [—-Fill your Best Answer here——]

Q27

INTRV: Asik, What is the difference between the Time Variant and Non time variant tables in any ETL Testing projects and how you will test it?

CAND: [—-Fill your Best Answer here——]

Q28

INTRV: Asik, Can you delete a table or Drop a table? What happened if we Drop a tables?

CAND: [—-Fill your Best Answer here——]

Q29

INTRV: Asik, What is the significance of Views in any ETL testing projects?

CAND: [—-Fill your Best Answer here——]

Q30

INTRV: Asik, Does a table contain more than one Primary Key? And Does a table contain more than one Primary Index?

CAND: [—-Fill your Best Answer here——]

Q31

INTRV: Asik, DBA has granted Select access to DB, does it mean that you have select access to all the tables sits in that DB?

CAND: [—-Fill your Best Answer here——]

 Q31

INTRV: Asik, if the target column is Integer (30) can you load a Varchar (30) into it?

CAND: [—-Fill your Best Answer here——]

Q32

INTRV: Asik, target Date column is constrained as DDMMYYYY but from source we are getting it as MMDDYYYY? What will happen to load job?

CAND: [—-Fill your Best Answer here——]

Q34

INTRV: Asik, What does it mean by Full-Snap shot and historical records?

CAND: [—-Fill your Best Answer here——]

Q35

INTRV: Asik, What does it mean Catch up logic involved in loads and how you will test it?

CAND: [—-Fill your Best Answer here——]

Q36

INTRV: Asik, What is the difference between Load projects and Extract projects?

CAND: [—-Fill your Best Answer here——]

Q37

INTRV: Asik, if the target column is defined as Decimal (4, 3), can you load the values as Decimal (4, 5)? Or can you load the values as Decimal (4, 3)

CAND: [—-Fill your Best Answer here——]

Q37

INTRV: Asik, What is the difference between Union and Union All? And where we want to use Union and where we want to use Union All?

CAND: [—-Fill your Best Answer here——]

 Q38

INTRV: Asik, What is the use of Cross Join? And what circumstances you use the Cross Join?

CAND: [—-Fill your Best Answer here——]

Q39

INTRV: Asik, What is the AutoSys comment to trigger the job, abort the job, hold the job?

CAND: [—-Fill your Best Answer here——]

Q40

INTRV: Asik, What is the significance of Group by functions in SQL and where we will use the Group by functions?

CAND: [—-Fill your Best Answer here——]

Q41

INTRV: Asik, How you can find the duplicates in the Staging tables?

CAND: [—-Fill your Best Answer here——]

Q42

INTRV: Asik, What is the difference between SET table and Multi Set tables? From where we can get these details?

CAND: [—-Fill your Best Answer here——]

Q43

INTRV: Asik, Can one View table refer another view table to create a new view table?

CAND: [—-Fill your Best Answer here——]

Q44

INTRV: Asik, Can one View table refer another view table to create a new view table?

CAND: [—-Fill your Best Answer here——]

 Q45

INTRV: Asik, What all are the common SQL Syntax errors you encountered and how you are resolved it?

CAND: [—-Fill your Best Answer here——]

Q46

INTRV: Asik, Do you have any idea what is Data Modeling?

CAND: [—-Fill your Best Answer here——]

Q47

INTRV: Asik, What is the difference between the Physical Design and Logical Design?

CAND: [—-Fill your Best Answer here——]

Q48

INTRV: Asik, What all are the Items available in a generic S2T?

CAND: [—-Fill your Best Answer here——]

Q49

INTRV: Asik, what does it mean if a column is mentioned as Value column in the S2T?

CAND: [—-Fill your Best Answer here——]

Q50

INTRV: Asik, Have I answered all of my basic questions to go to advanced questions?

CAND: I am still thinking and trying to answer few questions J