I am using Microsoft SQL Server Data Tools for Visual Studio 2017 (SSDT) to develop SSIS packages. Recently i started getting “Object reference not set to an instance of an object” error on commits to Azure DevOps. This is what i did to resolve my errors;
Close Visual Studio
Navigate to “C:\Users\[User Folder]\AppData\Local\Microsoft\Team Foundation\7.0\Cache” folder. Delete all contents.
If you are not able to solve External table is not in the expected format error, consider changing driver. try NPOI or ExcelDataReader.
ExcelDataReader Library
This library doesn’t work with .NET core.
EPPlus Library
EPPlus library works with .NET core.
Working with Excel Files with the help of Script Task
Integration Services provides the Excel connection manager, Excel source, and Excel destination for working with data stored in spreadsheets in the Microsoft Excel file format. The techniques described in this topic use the Script task to obtain information about available Excel databases (workbook files) and tables (worksheets and named ranges).
Common variables and Namespaces used to read Excel files
Variables. Open the Variables window and define the following variables: 1- ExcelFile, of type String. Enter the complete path and filename to an existing Excel workbook. 2 -ExcelTable, of type String. Enter the name of an existing worksheet or named range in the workbook named in the value of the ExcelFile variable. This value is case-sensitive. 3 – ExcelFileExists, of type Boolean. 4 – ExcelTableExists, of type Boolean. 5 – ExcelFolder, of type String. Enter the complete path of a folder that contains at least one Excel workbook. 6 – ExcelFiles, of type Object. 7 – ExcelTables, of type Object.
Imports statements You will need following .NET Framework namespaces; 1 – System.IO, for file system operations. 2- System.Data.OleDb, to open Excel files as data sources.
References. The code that read schema information from Excel files require an additional reference in the script project to the System.Xml namespace.
If the requirement is to run dynamic query on each row, you might get inside Script comoponent. Here is how you can grab SQL connection and run your queries;
Your connection is defined here on script component;
public class ScriptMain : UserComponent
{
IDTSConnectionManager100 connMgr;
SqlConnection sqlConn;
SqlDataReader sqlReader;
public override void AcquireConnections(object Transaction)
{
connMgr = this.Connections.MyConn;
sqlConn = (SqlConnection)connMgr.AcquireConnection(null);
}
public override void ReleaseConnections()
{
//base.ReleaseConnections();
connMgr.ReleaseConnection(sqlConn);
}
public override void PreExecute()
{
base.PreExecute();
string KeyValue = Variables.vFileOutputString;
String vSqlStatement = "SELECT SqlStatement FROM [DataHub].[vw_LastStepForRglData] WHERE ADNumber = '" + KeyValue + "' ";
SqlCommand cmd = new SqlCommand(vSqlStatement, sqlConn);
sqlReader = cmd.ExecuteReader();
}
public override void PostExecute()
{
base.PostExecute();
sqlReader.Close();
}
public override void CreateNewOutputRows()
{
while (sqlReader.Read())
{
{
DynamicSqlBuffer.AddRow();
DynamicSqlBuffer.SqlStatement = sqlReader.GetString(0);
}
}
}
}
I ma using Dapper and SqlClient libraries ASP.NET Core and Console application to access data from Database.
SSIS application;
Here is how this will work in SQL Server Integration services;
ADO.NET Source
SELECT DocumentId, [FileName], [ContentType], [FileStreamCol]
FROM [dbo].[Document]
WHERE 1=1
SC-Transform data
This is a SSIS Transformation script component. This prepares the data that will be used later on in Export Column task;
Here is the code block;
private byte[] fileContent = null;
private string fileExtension = String.Empty;
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
//determine content type
switch (Row.ContentType)
{
case "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet":
case "application/vnd.ms-excel":
case "application/msexcel":
fileExtension = "xlsx";
break;
case "application/vnd.ms-excel.sheet.macroEnabled.12":
fileExtension = "xlsm";
break;
default:
break;
}
Row.fileContentName = String.Concat(Row.DocumentId, ".", fileExtension);
//A VB6 application saves excel files as BLOB objects. For download it subtracts one element from
//byte array. Following same dance here
var blobLength = Convert.ToInt32(Row.FileStreamCol.Length);
fileContent = Row.FileStreamCol.GetBlobData(0, blobLength - 1);
Row.fileContent.AddBlobData(fileContent);
}
Derived column transform;
Export column settings;
Checking Force Truncate overwrites extracted files.