Get/SetVariable value from SSIS VariableDispenser

Here is some code that allows you to Get/Set a variable (object) value from/on the VariableDispenser in an SSIS package:

public static T GetVariable<T>(this VariableDispenser variableDispenser, string scopedVariableName)
{
 Variables variables = null;

 try
 {
  variableDispenser.LockForRead(scopedVariableName);
  variableDispenser.GetVariables(ref variables);
  return (T)variables[0].Value;
 }
 finally
 {
  if (variables != null) variables.Unlock();
 }
}

public static void SetVariable<T>(this VariableDispenser variableDispenser, string scopedVariableName, T value)
{
 Variables variables = null;

 try
 {
  variableDispenser.LockForWrite(scopedVariableName);
  variableDispenser.GetVariables(ref variables);
  variables[0].Value = value;
 }
 finally
 {
  if (variables != null) variables.Unlock();
 }
}

Consume custom SSIS tasks without GAC installation

For a while i thought that in order to consume a custom SSIS task you had to install the assembly in the GAC. Now i know better ;)

For the designer (BIDS) you have to copy the files to:

C:\Program Files\Microsoft SQL Server\100\DTS\Tasks
C:\Program Files\Microsoft SQL Server\100\DTS\PipelineComponents

For the runtime (BIDS) you have to copy the files to:

C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies

And for dtexec you have to copy the files to:

C:\Program Files\Microsoft SQL Server\100\DTS\Binn

Here is a small powershell script that gives you the paths where you want to copy your assembly to:

# Lookup Tasks, PipelineComponents and DtExec paths
$DtsPath = (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\100\DTS\Setup').SQLPath;
$DtsTasksPath = (Resolve-Path "$DtsPath\Tasks");
$DtsPipelineComponentsPath = (Resolve-Path "$DtsPath\Pipelinecomponents");
$DtExecPath = (Resolve-Path "$DtsPath\Binn");

# Lookup VS2008/Bids path
$Vs2008Path = (Get-ItemProperty 'HKLM:\Software\Microsoft\VisualStudio\9.0').InstallDir;
$Vs2008PrivateAssembliesPath = (Resolve-Path "$Vs2008Path\PrivateAssemblies");

Exploring messaging at lower levels…

Yesterday a colleague of mine, Neil Robbins, asked me how a piece of code would look like if I apply the Hollywood principle on it (Don’t call us, we’ll cal you).

Let me start with setting the scene: The purpose of the code is to provide items via provider and to consume those items via a consumer.

Here is how my oldskool function signatures would look like:

class ItemProviderFactory {
 public ItemProvider Create() { .. }
}

class ItemProvider {
 public IEnumerable<Item> Provide() { .. }
}

class ItemConsumerFactory {
 public ItemConsumer Create() { .. }
}

class ItemConsumer {
 public void Consume(IEnumerable<Item> items) { .. }
}

My Hollywood style function signatures look like the following:

class ItemProviderFactory {
 public void WithItemProvider(Action<ItemProvider> action) { .. }
}

class ItemProvider {
 public void Provide(Action<IEnumerable<Item>> action) { .. }
}

class ItemConsumerFactory {
 public void WithItemConsumer(Action<ItemConsumer> action) { .. }
}

class ItemConsumer {
 public void Consume(IEnumerable<Item> items) { .. }
}

And now I am able compare the code that glues everything together:

void OldStyle(OldStyle.ItemProviderFactory itemProviderFactory, OldStyle.ItemConsumerFactory itemConsumerFactory) {
 var provider = itemProviderFactory.Create();
 var items = provider.Provide();
 var consumer = itemConsumerFactory.Create();
 consumer.Consume(items);
}
void HollywoodStyle(HollywoodStyle.ItemProviderFactory itemProviderFactory, HollywoodStyle.ItemConsumerFactory itemConsumerFactory) {
 itemProviderFactory.With(provider =>
  provider.Provide(items =>
    itemConsumerFactory.With(consumer
      => consumer.Consume(items))));
}

Let me refactor this Hollywood code a bit:

// An itemconsumer consumes items as following:
Action<HollywoodStyle.ItemConsumer, IEnumerable<Item>> consumerAction = (consumer, items) => consumer.Consume(items);

// As soon as I have items, I want a consumer to consume them:
Action<IEnumerable<Item>> itemsAction = (items) => itemConsumerFactory.With(consumer => consumerAction(consumer, items));

// I can get items as following:
Action<HollywoodStyle.ItemProvider> providerAction = (provider) => provider.Provide(itemsAction);

// I can get an ItemProvider as following:
itemProviderFactory.With(providerAction);

I think that most colleagues are quite thankful that i’m not a Hollywood star ;)

Polling the existence of a file with PowerShell

Sometimes you run into a situations where a given task spawns a separate thread and completes it’s work on that separate thread. Eg: sending a bit XMLA to SQL Server Analysis Services with Microsoft.AnalysisServices.Deployment.exe and then waiting for the processing to be completed. Anyway, here is a simple function that will wait untill a given file exists:

function WaitForFile($File) {
 while(!(Test-Path $File)) {
  Start-Sleep -s 10;
 }
}

Get current file in PowerShell

A while ago i wrote a small script to take care of deployment. Configuring the source folders went as following:

param(
	$BaseDir = (Get-Location).Path,
	$WebDir = (Resolve-Path "$BaseDir\web"),
	$DatabaseDir = (Resolve-Path "$BaseDir\database")
)

The problem with this code is that it only works when your current working directory is set to the location of this script. An administrator (or build system) invokes the script as following:

PS C:\Users\Admin>& 'D:\Deployments\20110124\Deploy.ps1';

Because we don’t want to annoy the consumer of our script with the burden of making sure he is in the correct directory we modified our code as following:

param(
	$BaseDir = (Split-Path $MyInvocation.MyCommand.Definition),
	$WebDir = (Resolve-Path "$BaseDir\web"),
	$DatabaseDir = (Resolve-Path "$BaseDir\database")
)

A quick win :)

Get entire message body from an Intent

I recently started programming the Android and noticed that most examples for processing an incoming SMS are not entirely correct.

An SMS message is limited to 160 characters. Current mobile phones break up a larger message in multiple messages transparently for the user. When Android notifies you about an incoming SMS it has all parts (of that large message) available. So here is how you reconstruct the entire message body from an Intent

Bundle bundle = intent.getExtras();
if (bundle == null) return;

StringBuilder message = new StringBuilder();
Object[] pdus = (Object[]) bundle.get("pdus");

// Rebuild this entire message from the multi part smses/pdus
for (Object pdu : pdus){
 // Notice that i use the deprecated android.telephony.gsm.SmsMessage
 // android.telephony.SmsMessage throws when i call createFromPdu
 SmsMessage msg = SmsMessage.createFromPdu((byte[])pdu);
 message.append(msg.getMessageBody().toString());
}

Techniques learned in SQLtopia: Look Ma, no loops!

I have been programming in c-based languages for more than 10 years now. Lately i have spent quite a bit of time in SQLtopia and learned a couple of techniques that are quite different from what i was used to.

Let me explain with an example: The boss comes in and asks for a report that contains all the days on which i have worked this year. As a typical c# programmer i come up with the following solution: Iterate over all the days in the year and skip the days i was out on holidays:

IEnumerable<DateTime> FindWorkingDays()
{
 var begin = new DateTime(2010, 01, 01);
 var end = new DateTime(2010, 12, 31);

 var holidays = new[]
 {
  new Holiday { Begin = new DateTime(2010, 07, 01), End = new DateTime(2010, 07, 31) },
  new Holiday { Begin = new DateTime(2010, 09, 01), End = new DateTime(2010, 09, 15) }
 };             

 for (var date = begin; date <= end; date = date.AddDays(1))
 {
  if(holidays.Any(holiday => holiday.Begin <= date && date <= holiday.End)) continue;
  yield return date;
 }
}

In SQLtopia it is recommended to use set-based solutions instead of loops so we need a radically different solution.

Here is how we generate rows:

WITH
    [Nums1] AS ( SELECT 1 AS [Value] UNION SELECT 2 AS [Value] )
  , [Nums2] AS ( SELECT A.* FROM [Nums1] AS A, [Nums1] AS B, [Nums1] AS C)
  , [Nums3] AS ( SELECT A.* FROM [Nums2] AS A, [Nums2] AS B, [Nums2] AS C)
  , [Nums4] AS ( SELECT A.* FROM [Nums3] AS A, [Nums3] AS B )
SELECT * FROM [Nums4];

With rows we can generate numbers:

DECLARE @count INT = 1000;

WITH
    [Nums1] AS ( SELECT 1 AS [Value] UNION SELECT 2 AS [Value] )
  , [Nums2] AS ( SELECT A.* FROM [Nums1] AS A, [Nums1] AS B, [Nums1] AS C)
  , [Nums3] AS ( SELECT A.* FROM [Nums2] AS A, [Nums2] AS B, [Nums2] AS C)
  , [Nums4] AS ( SELECT A.* FROM [Nums3] AS A, [Nums3] AS B )
  , [Numbers] AS ( SELECT TOP(@count) -1 + ROW_NUMBER() OVER(ORDER BY [Value]) AS [Value] FROM[Nums4] )
SELECT * FROM [Numbers];

And with numbers we can generate dates:

DECLARE @begin DATETIME =  '2010-01-01';
DECLARE @end DATETIME =  '2010-12-31';
DECLARE @count INT = DATEDIFF(DAY, @begin, @end) + 1;

WITH
    [Nums1] AS ( SELECT 1 AS [Value] UNION SELECT 2 AS [Value] )
  , [Nums2] AS ( SELECT A.* FROM [Nums1] AS A, [Nums1] AS B, [Nums1] AS C)
  , [Nums3] AS ( SELECT A.* FROM [Nums2] AS A, [Nums2] AS B, [Nums2] AS C)
  , [Nums4] AS ( SELECT A.* FROM [Nums3] AS A, [Nums3] AS B )
  , [Numbers] AS ( SELECT TOP(@count) -1 + ROW_NUMBER() OVER(ORDER BY [Value]) AS [Value] FROM[Nums4] )
  , [Dates] AS ( SELECT DATEADD(DAY, [Value], @begin) AS [Date] FROM [Numbers] )
SELECT * FROM [Dates];

With dates we can build the dates on which we were out on holidays:

DECLARE @begin DATETIME =  '2010-01-01';
DECLARE @end DATETIME =  '2010-12-31';
DECLARE @count INT = DATEDIFF(DAY, @begin, @end) + 1;

WITH
    [Nums1] AS ( SELECT 1 AS [Value] UNION SELECT 2 AS [Value] )
  , [Nums2] AS ( SELECT A.* FROM [Nums1] AS A, [Nums1] AS B, [Nums1] AS C)
  , [Nums3] AS ( SELECT A.* FROM [Nums2] AS A, [Nums2] AS B, [Nums2] AS C)
  , [Nums4] AS ( SELECT A.* FROM [Nums3] AS A, [Nums3] AS B )
  , [Numbers] AS ( SELECT TOP(@count) -1 + ROW_NUMBER() OVER(ORDER BY [Value]) AS [Value] FROM[Nums4] )
  , [Dates] AS ( SELECT DATEADD(DAY, [Value], @begin) AS [Date] FROM [Numbers] )
  , [Holidays] AS ( SELECT '2010-07-01' AS [Begin], '2010-07-31' AS [End]
                    UNION
                    SELECT '2010-09-01' AS [Begin], '2010-09-15' AS [End]
                  )
SELECT [Date] FROM [Holidays],[Dates] WHERE [Date] BETWEEN [Begin] AND [End] ORDER BY [Date];

And now we can easily select the dates that are not holiday dates:

DECLARE @begin DATETIME =  '2010-01-01';
DECLARE @end DATETIME =  '2010-12-31';
DECLARE @count INT = DATEDIFF(DAY, @begin, @end) + 1;

WITH
	[Nums1] AS ( SELECT 1 AS [Value] UNION SELECT 2 AS [Value] )
  , [Nums2] AS ( SELECT A.* FROM [Nums1] AS A, [Nums1] AS B, [Nums1] AS C)
  , [Nums3] AS ( SELECT A.* FROM [Nums2] AS A, [Nums2] AS B, [Nums2] AS C)
  , [Nums4] AS ( SELECT A.* FROM [Nums3] AS A, [Nums3] AS B )
  , [Numbers] AS ( SELECT TOP(@count) -1 + ROW_NUMBER() OVER(ORDER BY [Value]) AS [Value] FROM[Nums4] )
  , [Dates] AS ( SELECT DATEADD(DAY, [Value], @begin) AS [Date] FROM [Numbers] WHERE [Value] <= DATEDIFF(DAY, @begin, @end))
  , [Holidays] AS ( SELECT '2010-07-01' AS [Begin], '2010-07-31' AS [End]
                    UNION
                    SELECT '2010-09-01' AS [Begin], '2010-09-15' AS [End]
                  )
  , [HolidayDates] AS (SELECT [Date] FROM [Holidays],[Dates] WHERE [Date] BETWEEN [Begin] AND [End] )

SELECT [Dates].[Date] FROM [Dates]
					  LEFT OUTER JOIN [HolidayDates] ON [Dates].[Date] = [HolidayDates].[Date]
					  WHERE [HolidayDates].[Date] IS NULL;

Look Ma, no loops!

Exploring window functions

Here is a sample query that allows you to explore the behavior of various aggregate window functions:

WITH
    [Nums1] AS ( SELECT 1 AS [Value] UNION SELECT 2 AS [Value] )
  , [Nums2] AS ( SELECT A.* FROM [Nums1] AS A, [Nums1] AS B, [Nums1] AS C)
  , [Nums3] AS ( SELECT A.* FROM [Nums2] AS A, [Nums2] AS B, [Nums2] AS C)
  , [Nums4] AS ( SELECT A.* FROM [Nums3] AS A, [Nums3] AS B )
  -- Build numbers from 1 to 1000
  , [Numbers] AS ( SELECT TOP(1000) ROW_NUMBER() OVER(ORDER BY [Value]) AS [Value] FROM[Nums4] )
SELECT
    [Value]
  , ROW_NUMBER() OVER(PARTITION BY [Value] / 100 ORDER BY [Value] % 10) AS [RowNumber]
  , RANK() OVER(PARTITION BY [Value] / 100 ORDER BY [Value] % 10) AS [Rank]
  , DENSE_RANK() OVER (PARTITION BY [Value] / 100 ORDER BY [Value] % 10) AS [DenseRank]
  , NTILE(4) OVER (PARTITION BY [Value] / 100 ORDER BY [Value] % 10) AS [Tile100]
FROM
    [Numbers]
ORDER BY
    [Value];

Convert AAC/M4A files to MP3 with VLC and PowerShell

Here is a way to convert your AAC/M4A files to MP3 using VLC media player:

vlc.exe -I dummy old.m4a :sout=#transcode{acodec=$codec,vcodec=dummy}:standard{access=file,mux=raw,dst=new.mp3} vlc://quit

Let’s wrap this command in a bit of PowerShell:

function ConvertToMp3([switch] $inputObject, [string] $vlc = 'C:\Program Files\VideoLAN\VLC\vlc.exe') {
    PROCESS {
        $codec = 'mp3';
        $oldFile = $_;
        $newFile = $oldFile.FullName.Replace($oldFile.Extension, ".$codec");
        &"$vlc" -I dummy "$oldFile" ":sout=#transcode{acodec=$codec,vcodec=dummy}:standard{access=file,mux=raw,dst=`'$newFile`'}" vlc://quit | out-null;
        #Only remove source files when you are sure that the conversion works as you want
        #Remove-Item $oldFile;
    }
}

And now we can use this function for *all* m4a files in a given folder:

function ConvertAllToMp3([string] $sourcePath) {
    Get-ChildItem "$sourcePath\*" -recurse -include *.m4a | ConvertToMp3;
}

Using the function is as easy as:

ConvertAllToMp3 'C:\Users\timvw\Music';

Load all script files at PowerShell startup

These days i have quite some scripts files that i want to be loaded each time i launch PowerShell. Windows PowerShell Profiles teaches me where i should store my $profile. Here is what it looks like:

# Lookup powershell scripts location
$UserProfile = (Get-ChildItem Env:UserProfile).Value;
$ScriptFolder = "$UserProfile\My documents\WindowsPowerShell";

# Source all .ps1 files in PowerShell profile folder
Get-ChildItem $ScriptFolder -name -include '*.ps1' -exclude 'profile.ps1'
 | foreach {  (. "$ScriptFolder\$_") };

# Configure environment for VS2010
SetVS2010;