DWBI IT companies' review for Laterals

You are a 7+ years dwbi professional and you are looking out for your next switch. Many industry pundits would argue like product based companies and consultancies have better jobs , both in terms of the CTC and the kind of work, In this article I would like to emphasize why an IT professional shall stick to essentially IT companies while switching and which IT companies excel in the DWBI space and why an IT professional shall consider switching\being into these IT companies.

Please note this is just an opinion and the information below is based on what we hear in cafe discussions. This is not legal advise :)

It's always better to be in the IT service industry as it offers what is most important -

1. Job security.
2. A wide range of projects to move into from time to time.
3. Ability to work overseas and in India accordingly.
4. Though product based companies do offer high packages, most of their projects are internal, the exposure of which is nothing when compared to that of traditional billable projects.

Mid sized IT companies generally hire 7-8 years experience IT professionals as Project manager and leads for once in a blue moon client projects which they are able to bag.The upside is that there is tremendous opportunity for you in terms of CTC and client interaction. You can own the entire project. The only downside is job security as it is a project specific hiring.

Some of the top players in dwbi domain globally are as follows:

1. Cognzant - Every dwbi professional should definitely consider this company to work for. The work experience earned here is globally recognized in the dwbi space.

2. Wpro - Many wpro alumni have moved in dwbi product leaders like Informatica and are running the show there.Great company to work in new technologies.

3. IBbM - Is a market leader in analytics and every other thing. There is no dearth of projects but again that could change as per the geography.

4. Acenture- Another very nice to company to work for. Takes care of its employees and provides good CTC. Good number of projects across all domains.

5. Infy - It's great to see infy in the new avtaar. This company has managed to put back leadership issues and is now going great guns. A good option to consider.

H1B visa interview experience

Date\Venue – 11^th December 2015 – US Embassy, New Delhi

You may consider reaching 45 minutes before your time slot. It would be a good idea, if someone would accompany you to the embassy and wait outside, so that you may handover your mobile phones to him.

The place is bit crowded but the crowd is extremely well managed and things at the embassy go smooth.

Check 1 – Outside the embassy-  I was carrying my original documents along with the DS-160,Application confirmation and Passport from the VAC.I was told to produce my I-797 and it was stapled along with my passport.

Check 2 – Security Check. Its similar to that of the airport. I had to take off my jacket and wallet, and kept all my stuff in the tray which underwent a scanner. Simultaneously the guard scanned me with the machine.

We were made to wait in a waiting area and the entire line was told to move to the next stage inside the embassy.

Check 3-The gates of the main section opened up and I could see the portraits of American leaders .Nervous energy occupied the air and we were made to submit our finger prints. An officer was validating the same in the system and after a long pause confirmed my fingerprints.

Now we entered the main area, I could see around 7 to 8 counters with visa officials interviewing the candidates. The description of the visa officers were as follows:

1.       The sweet guy – Wearing a bow tie and sweetly asked the questions.

2.       The rude guy – Was asking questions in an aggressive way.

3.       The shrewd silent guy – This person meant business and was silent but extremely observant.

4.       The normal guy – Was asking questions in a normal tone and pretended to be confused at times.

5.       A young lady

6.       An old lady

There were some candidates that had chosen to be interviewed in Hindi like athletes and the American visa officials were asking questions to them in Hindi. I was quite impressed seeing this.

There was an attendant that was assisting the crowd to go the counters as per availability was directed to the Normal guy’s counter but was asked to shift to the ‘shrewd guy’s counter as it had a vacant space.

I greeted the Visa Officer and handed over the Passport and the I-797 to him.

VO – Whom do you work for?

Me- said my Employer’s name (only)

VO-Who is the client you shall work for?

Me – There is no client, the petitioner is XX.

VO – Where shall you be going in the US

Me – City and State

VO – Can you tell me something about the internal project that you shall be working on?

Me – I shall be participating in all phases of the SDLC starting from requirements analysis,design….

VO – Interrupted me ….No please specifically tell me about the product you shall be working on.

Me – I got extremely nervous here , I guess the VOs tone made me more nervous.I told him” We are building a product in the informatica product suite, that shall read data from an application databse and load the data in the cloud” (just said this line , although I could have carried on explaining but I decided to stop at this point)

There was a pause of 2 seconds.As though the VO wanted to hear more, but I decided not to explain    un-necessarily unless required.

VO – Your visa is approved sir.

Me – I stood there for some white receipt, I had read about in journals on internet.But he did;t give me any.

VO – Your Visa is approved sir, have a nice day.

Me-Thank you.

I came out of the embassy and didn’t stop for the courier services. Was too excited to stop.

VAC experience (US Visa Application Center)

Date\Venue - December 10^th 2015 – VAC New Delhi

My VAC was scheduled on 10-Dec-2015. I reached American Plaza ,Nehru Place around 1 hour earlier which was not really required. Everyone who has been given a particular time slot let’s say 12:15 pm (as in my case) may reach at least 20 mins earlier. The processing went smooth. As required, I carried my DS-160 and Application confirmation page along with passport. (Only these 3 docs required). No photographs are required as they shall be taken at the Visa Application Center (VAC).

Mobile phones in Switched off mode were allowed for the VAC.

Level 1 Check – At the entrance- DS-160, Application Confirmation page and Passport information cross checked. I was carrying a black and white printout of DS-160 and Application Confirmation page with clear bar codes. The Application Confirmation page was stamped with some seal. As the seal was dim, I asked the person to stamp my printout again.

Level 2 Check – I entered the VAC office and the guard asked me to switch off my mobile phone. He then checked me for any banned items and allowed me to enter.

Level 3 – I joined a queue inside the office and on reaching the counter, the officer checked my docs i.e. DS-160, Application Confirmation and the Passport. Then she pasted a sticker at the back of my passport which was earlier generated by a machine and had the same DS-160 id and passport number as mine. Please do not forget to get your Passport pasted by this sticker.

Level 4- I then collected a token from a different counter (like we do in banks) and sat in the waiting area. It’s not a very big place ,but a single hall where everything is happening ,just like a bank so no need to get nervous. I waited for my token number to be displayed on the screen

Level 5 – After waiting for 10 mins , my token number flashed and I went to the counter and submitted the three docs. The person asked me your name is XXX and your date of birth is XXX I confirmed.

Then he asked me to take off my specs and clicked my pic.

Then he asked me to put my left hand (4 fingers) on the scanner, and clean it with the cloth then after.The same process was repeated for the right hand and thumbs.

He then stamped my DS-160 form , returned my docs and wished me best of luck for the H1B interview.

Questions regarding existing BI reports to Business Users.

• How do you access these existing reports?
• Whats the existing frequency of report updates and data updates?
• What reports go together to make them relevant?
• What other tools do you use for analyzing the data out of these reports?
• What are the report delivery mechanisms existing right now?
• Are these refreshed on a batch process or based on demand from the users?
• Any specific features in the existing report that you would like most and does not see loosing it which may cause potential business implications?
• What are the existing bottlenecks with each of these reports?
• Are the goals in existing reports still relevant?
• What are the different parameters used in these reports? Whats the frequency of each parameter being used?
• What reports do you cross check to make sure your reports are upto date and are valid?
• What reporting metrics/KPI are being created in these reports? Are there special formulae being set in these reports as we bring data from the data warehouse?
• How do you perform your routine analysis? How does the existing reports aid in getting your job done faster? Do you see any trends in what kinds of routine analysis are you being tasked with?
• How complicated can these ‘one off ‘ reports being created? How long does it take to create these reports? How often do you think the data is already available in data warehouse but not in reports?
• Who are the power users for each of these reports? Organizing these users by department, by task can give you a very good insight of what reports can be grouped together in an OBIEE dashboard.
• Is there report usage tracking mechanism already available? Do we like to see this moving further into OBIEE?
• Open up your c-level executive’s high level requirements and see whether they are being already covered by these existing reports.
• Is there a backlog of users requests for enhancing the existing reports, create a new flavor of report based on the existing report?
• Can all reports be viewed by every one? How are the users segmented so that groups can be created to access these reports?
• Does the report format change based on the user? Is there any row level security involved? Is there any column level security involved in these reports? Simply ask ‘Who can see what’ for each of these reports.
• Do you analyze information or conduct analysis quarter over quarter or year over year? How far back do you go to support your analysis?
• Ask for access to the recent support ticket logs and see whether there is any trend of problem areas with the reports.
• Organizing the reports by business importance i.e. mission critical every day, end of month report to be ready by 2nd fiscal week of next month etc. etc..
• What is the existing support structure for these BI reports? Ex: When a user has problem, he goes to his assigned power user, if power user cant resolve it, create a ticket for BI team.
• Identify the existing drill down capabilities in the existing reports. Is this still valid? How often does these hierarchies change? Ex: A Product Hierarchy is created every month, we create a new IT request to include this new product hierarchy and it gets added.
• Identify the common dimensions and hierarchies used in most of these reports. Ex: Time, Product, Customer
• Identify the common set of filters that are being built across all these dimensions.
• What are the known gotchas with the existing data in the data warehouse. Do we have reports where multiple versions of truth is always a problem just because the way these reports are set up?
• Are there any decodes, if this is ‘x’ then ‘y’ kind of expressions developed as part of the reports?
• Are we happy with the existing refresh schedules?
• What output formats are being supported by these reports? Ex: Excel, PDF, web-based always.
• Identify any enhancements that would really delight the customer based on an existing report.
• Identify opportunities where previous attempts failed and the it did not make any financial sense at that time. Review these with your users and see whether that it is still the same.
• Document on how each of these reports fits strategically to the organization’s goals and performance monitoring. What are the success factors for your organization? Are these factors quantifiable? How do you know you are on track to achieve your results? How often does the department’s/organization’s goals change?
• Does any of the existing reports help you predict problems? Are there any alerting mechanisms built into these reports? Who gets these alerts?
• Check the reports for the visualization techniques used in the existing reports. Does it make sense to replicate them as is or see whether you can add new web 2.0 widgets for data visualizations. Ex: There was never a goal against this KPI before. Now, we have one. We like a dial chart on the first page of our report to show how we are performing weekly, quarterly, yearly.
• Identify users and see what level of familiarity do they have with this data. Segment these users based on criteria like influence, authority, participation levels. You always need high influencial, high authoritative and high participative users. Deal slowly and patiently with low influential and highly pessimistic users.

Identify the BI technology trends in their organization. Ex: We used to have these nice and fancy all excel based reports. Then the IT has changed our strategy 5 years ago and asked us to start using Micro Strategy tool. Now, we are going to go OBIEE. Check the pulse of your users to see how excited they are. As the technology changed, did something worked better? Did something get lost as part of the transition. Look for those transition pain points and make sure you have/develop some strategies on how to deal with them if you encounter one.

H1B visa- Facts and Myths

Myth 01 - I cannot have H1B and B1 visa at the same time.

H1B visa application is a 1 year long process. You can have both H1B and B1 visa.The only condition is that if H1B is filed for client A. You may not file B1 visa for the same client A, it has to be anyother client other than client A.

Myth 02 - The client for which my H1B was filed, is no more with the company and I need to go for stamping.

You will have to apply for amendment of LCA , the client should be a current client of the organisation.Once done you can go for stamping.

Myth 03- Dependent Visa should be applied while stamping of the concerned H1B visa person.

Dependent visa stamping can be done later too.

Myth 04 - If I switch the company my current H1B visa will expire - False, H1B and B1 visa can be transferred from one company to other.L1 visa cannot be transferred.

H1B visa - High level process flow

Step 01 – Your name gets nominated for the H1B visa from your organization. You would need the following:

1.       Employee Salary Structure(ESS)

2.       Client Letter

3.       Letter from Onsite Manager

4.       LCA letter approved from Department of Labour-USA.

5.       Educational Certificates

6.       Work Experience Documents.

7.       If you have travelled on other kinds of visas to US, you would be required to provide the information of dates and places you have lived.

Step 02 – The Immigration services will process your LCA and once done will flag you. The LCA is done so that Department of Labour is sure that the company is not under paying you.

Step 03 – Your name shall be given to the USCIS for the H1B selection. If initial slot of 20000 is reached (which happens in most cases) additional 65000 visas are made available via lottery known as H1B-CAP for graduate degree holders.

Step 04 – If your name is selected then you will have to go for stamping of your passport with H1B visa to the counsellet.

Step 05 – Since the process takes over a year,the project\client from which the LCA would be processed may change, hence it is advisable to apply for LCA amendment with the new project\client. To achieve the same a new ESS letter and Transfer letter based on client location needs to be generated.

But we can parallely go and get the stamping done on the old LCA to save time.

Once the LCA is amended the H1B visa will also have to be amended.

Step 06 – Face the visa interview and if you pass you are ready to fly.

Disclaimer - This is not legal advice, I'm just jotting down my limited knowledge.

Batch,Real Time and CDC -Data Extraction Techniques

1. Batch - {On-Demand, User initiated,Loss of transactional Lineage,Can be scheduled}
2. Real Time-{Maintains transactional Lineage based on architechture,Implemented to meet tighter SLAs}
3. CDC-Change Data Capture-Continuous extraction.Transactrional Lineage is maintained,Delete data is also captured in form of soft deletes.

-For CDC implementation we need to capture CreatedDate and LastModifiedDate of records

Difference between Kimball and Inmon approach

The Inmon approach is said to have the Hub and Spoke architecture. The primary feature of the same is the Enterprise Data Warehouse (EDW) which is maintained as a 3NF database. It is slightly different from the Operational database as it is:

1. Subject Oriented
2. Non-volatile - Data doesn't change . Soft delete is present by the use of timestamping.
3. Time variant - Maintains historical data
4. Integrated.

It does not have fact or dimensions, and has a set of functional data marts pulling data out of it.

OLTP  > EDW (3N) >Datamart (3N) >OLAP Cube

The Kimball approach is said to have the Bus Architecture. It contains set of stars i.e de-normalized groups of fact and dimension tables connected via conformed dimension (dimensions common to two facts)

OLTP  > Star Schema (Dimensional Model) >OLAP Cube

Definition of a DWH

A DWH is a

1. Subject Oriented i.e. Has the power to mould the transactional data around a particular subject.If we want to answer questions around the sales of a particular product, the DWH would be around the Subject-sales.etc.
2. Integrated i.e. Integrates data from various sources and keeps in form of dimensional model.
3. Non-volatile i.e. -Data once entered in DWH does not change.
4. Time variant - DWH maintains historical data.

collection of data.

Why is a DWH required in an organization?

1.       Data is an asset and has to be retained as per organizational needs.DWH helps us in maintaining          large volumes of data.

2.       To reduce IT involvement in report creation and helps the business to custom build the reports.

3.       MDM is used to maintain the single version of the truth.

Categories of Enterprise Data

1.    Master data -   Business entities of the organization.Data revolves around these business entities. Customer,Employee etc. This is maintained as part of the MDM solution.

2.       Transactional Data – Generated as part of day to day business processes. Also known as the OLTP system Online transactional processing systems)

3.       Analytical Data – Typically related to Key Performance Indicators (KPIs).Data Warehousing systems are used to store and retrieve analytical data.

4.       Meta Data-  Data about data.Describes IT systems, Business rules etc.

5.       Reference Data- Static data elements, sometime transient like currency ,industry standards. A reference data hub is maintained for transaction processing.

6.       Un-Structured data- All the above mentioned data would be classified as structured data.Examples of unstructured data would be Emails,Social Media data i.e. Big Data. Hadoop is a platform used to manage un-structured data.

Product Dimension

The Product dimension has the following hierarchy.

Merchandise hierarchy, important hierarchy of the product dimension
Department > Categories > Brands > SKU (Stock keeping Units)-identifies individual product.

To simplify the above hierarchy the following example could be used. Hierarchy in Flipkart

Men (Department) > Clothing (Categories) >T-Shirts>UCB(Brand)>Blue colored shirt.

Some of the attributes of the Product dimension are as follows:

1. SKU Number (Natural Key\Business Key)
2. Product Key (PK)
3. Category Description
4. Department Description etc.

Date Dimension

Why do we need a separate date dimension, if we can insert the date from the transnational data directly,as is?

A date dimension has 365 rows ,each row representing a particular date and a Date ID. This Date ID is maintained in the fact table rather than the date from the transnational table.The date dimension would have the following attributes:

1. Day of the week

2. Week of the month

3. Month

4. Year

5. Holiday indicator

6. Season {Thanksgiving,Christmas etc.}

7. Fiscal year (Financial year of US)

8. Fiscal month

9. Fiscal Week etc.

All these attributes are very relevant at report level, and could give a whole lot of information combined with the fact, as compared to a single date value coming from the transnational system.

Hierarchy for Date Dimension

Year > Month >Week>Day>Hour>Minute>Second.

Stories of Big Daddies of the industry

Larry Ellison (Oracle) - Never gradated from college, learning was primarily based on Self study. He just picked up a book and started programming.

Bill Gates (Microsoft) - Never completed college

Steven Jobs (Apple) - Epic Misfit!!

The reason I need to write the same in my blog, that every s/w engineer needs to remind himself that he is working in an industry where degrees and certifications do not matter.What matters is your intellect, competence and aptitude. It gives a fair platform to a private engineering college or a correspondence student to compete against the ones coming from the elites institutions.So if you are good enough you are bound to make it big.The question is -Do you have it in you?

Difference between Delta Load and Incremental Load

Scenario:

We have completed the first load into the target table, and wish to implement the Delta load and the Incremental load.

• Delta Data is the parent of Incremental Data.
• What is Delta Data?
• Assuming that in our scenario, the first load happened on 1-Jan-2015. The data present in the source on 2-Jan -2015 is the delta data. 
• How to read delta data?
• Add the date filter in the source query i.e. WHERE load_date>1-Jan-2015
• What is incremental data?
• The data that we have read , as delta data, needs to be compared with the target data for key columns. This is generally done through SCD(Slowly Changing Dimensions) algorithms. Incremental data is the changed or new data found while the delta and target data are compared via SCD algorithms.

MSBI - SSIS,SSRS,SSAS - Interview experience-HCL

DWH and SSAS

1. How is Surrogate key different from Primary key? - It is independent of business data.
2. What is a conformed dimension? - Dimension common to more than one fact table
3. Whats is a degenerate dimension? - Dimension too small to be made into a table, hence included in fact. eg: Attendance, Boolean values.
4.Junk Dimension?- Grouping meaningful flags and texts in a single dimensional table.
5. Aggregate fact table use? fact table containing pre-calculated value,that feeds the report for faster results.
6. SCD? -
Slowly changing dimension 01 - update
Slowly changing dimension 02 - update and insert - History maintained
Slowly changing dimension 03 - First and Latest record only.
7. In a bottom up approach we always have a datamart at the bottom then the DWH and then OLAP at topmost.
8. What is a cube? - Data modelling technique to do multidimensional analysis.
9. What is a Hypercube?-To analyse more than 3 dimensions using a cube,the dimensions are merged representing single edge of the cube.Such cubes are called hyper cubes.
10.What is faster ROLAP or MOLAP? - MOLAP it is faster and is more complex as well.Supports Slice and Dice features which are not supported in ROLAP.
11.What is Slice and Dice feature of cube? - If each edge represents a single dimension of the cube, the page displayed is a slice or 2-dimensional plane of the cube.With each rotation the dimensions on the edges of the cube changes and so does the plane giving user the liberty to view and analyse the data from different angles.
12. Difference between relational model and dimensional model?

SSIS

1. What is the difference between Merge,Merge Join and Union all?
2. How can we trigger a SSRS report once a job in SSIS is completed? --sp_start_job
3. Name some native connections in SSIS? - ADO.NET,OLEDB,flatfiles,excel
4.Difference between Fuzzy lookup and lookup?
5. Container Types ={For each loop container,For loop container,Sequence container,task host}

SSRS
1. Difference between SSRS and Crystal reports.
2. How caching works in SSRS reports?
3. Deployment of SSRS in Native and SharePoint modes?
4. Is it safe to cache the report in report manager if we are required to save the credentials? - Yes as the credentials are starred.

Scrum Implementation : Live project experience

So we completed a 3 – Sprint DWH project for a FMCG US based company.

It was a roller coaster ride and we stretched every bit of our mental and physical skin to deliver.

Technical Streams: The project had the following technical streams

1.       Data Modeling – The data modelers started off with their part of analysis and prepared the logical data models of the Salesforce ERP source system.

a.       The analogous tables were physicalised in the Oracle relational dbse which was called as the first layer. The business key was being used as the primary key in this layer.

b.      The second layer was the staging layer where the dimensional model existed(ODS). Facts and dimensions were identified by the onsite teams and the same were incorporated in the logical data model and implemented in the physical data model in the 2^nd layer of the DWH.Here the Surrogate key was the primary key called as the SID or the source identifier.

c.       The presentation layer (BIDW)was the third and the final layer which had the warehouse identifier (WID) as the primary key , it also had the SID.The tables were a mirror image of their siblings in the ODS table.

2.       ETL – Informatica 9.6 on Linux was being used for this project.

3.       Reporting – Cognos 10.1 clients were used.

DataWarehouse and Data Marts

Data Warehouse (Top – down approach)

• -          Union of all the data mart
• -          Data Received from staging area
• -          Corporate enterprise wide data
• -          Data Received from staging area

Data Marts (Bottom up approach)

• -          Departmental view of data
• -          Star Join (Facts and Dimension)
• -          A single business process