http://www.microsoft.com/Downloads.

Then I need to parse this string and get the domain name “www.microsoft.com”.

Looking at the parallel libraries of .NET 4.0,  I thought to use them and increase our data crunching performance. I planned to use Parallel.ForEach in place of foreach  to harness all the CPU cores of my box.

However, to my amazement,  I realized that using parallel version of foreach is taking more time than the single thread version of foreach. After little head scratching, I realized the work in my foreach loop was so small that cost of creating thread and destroying them was more than the work they need to do them self. It is like if the work is small enough that explaining someone to do it will take more time than actually doing it, sometime parallel processing and delegating work is not good.

However, as soon as I put heavy processing in the loop, for e.g. making the thread sleep for 10 millisecond :), my parallel version started performing better than the single threaded version of foreach.

Here is the code I wrote to do the parallel parsing:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics;
using System.Threading.Tasks;
using System.Threading;

namespace DP_Parallel
{
class Program
{
static void Main(string[] args)
{
 List str = new List();
 using (AdventureWorksEntities AwContext = new AdventureWorksEntities())
 {
  IEnumerable uri = AwContext.ClickStreams
  .Where(u => u.ReferringURI.Length > 0)
  .Select(u => u.ReferringURI)
     .Take(1000000);

     //Parallel
     Parallel.ForEach
  (uri, (string s) =>
     {
      int Pos = s.IndexOf(@"/", 8) - 7;
      if (Pos > 5)
      {
        //Thread.Sleep(10);
       str.Add(s.Substring(7, Pos))  ;
       //Some other processing on the string - upper case etc
      }
     }
    );
   }
  }
 }
}

For a million rows C# non parallel code took 8.67 second, whereas Parallel version took 1 Minute and 1.51 second, but only 16 second in SQL with the following query

print cast ( getdate() as time )
SELECT  TOP 1000000
  case CHARINDEX ( '/', [referringURI], 8 )
  when 0 then ReferringURI
  else substring ( [ReferringURI] , 8, (CHARINDEX ( '/', [referringURI], 8)  ) - 8)
  end as DomainName
   into #T
  FROM [StagerDW].[dbo].[ClickStream] where len (ReferringURI) > 0
print cast ( getdate() as time )
 

So for just this task, C# code is twice as efficient over SQL and  many fold over the parallel code. But, as I said if you have to do a little heavy processing than parallel code may win over the single thread code.  So the best option to write a CLR Stored procedure in SQL, and from the CLR code enjoy the multi core parallel processing advantages.

yes, we don’t wait whole day to process the billion rows but crunch them as they come in …

using (AdventureWorksEntities AwContext = new AdventureWorksEntities())
 {
   var LoginId = AwContext.Employees
            .Where ( u => u.LoginID.Length > 0 )
            .Select(u => u.LoginID)
            .Take(10);

As you can see from the above code, I am trying to get the top 10 rows from Employee table and only one column LoginID, where loginID length is more than zero. If you fire this query you may get a SQL in SQL Profiler as follows:

SELECT TOP (10)
      [c].[LoginID] AS [LoginID]
        FROM [HumanResources].[Employee] AS [c]
            WHERE LEN( [c].[LoginID] ) > 0

<pre>using (AdventureWorksEntities AwContext = new AdventureWorksEntities())</pre>
 {
 var LoginId = AwContext.Employees.AsParallel
 .Where ( u => u.LoginID.Length > 0 )
 .Select(u => u.LoginID)
 .Take(10);

SELECT
   [Extent1].[EmployeeID] AS [EmployeeID],
   [Extent1].[NationalIDNumber] AS [NationalIDNumber],
   [Extent1].[ContactID] AS [ContactID],
   [Extent1].[LoginID] AS [LoginID],
..all other columns
     FROM [HumanResources].[Employee] AS [Extent1]
         WHERE LEN( [c].[LoginID] ) > 0

DownloadDataContext db = new DownloadDataContext();
IQueryable downloads =
      from f in db.Files
      join fr in db.FamilyReleases
          on f.ReleaseID equals fr.ReleaseID 
      join fdl in  db.FamilyDetailsByLocales
          on fr.FamilyID equals fdl.FamilyID
              select new DownloadDetails{
                FamilyID = fdl.FamilyID,
                FileName = f.FileName,
                FileSize = f.FileSize, URL = f.URL
)

Once I got the data in IQueryable downloads, Out of old habit, I proceeded to make the XML as follows:

MemoryStream ms = new MemoryStream();
using (XmlTextWriter w = new XmlTextWriter( ms, Encoding.UTF8 )) {
     w.Formatting = Formatting.Indented;
     w.WriteStartElement("Products");
           foreach (DownloadDetails d in downloads) { 
          w.WriteStartElement("FamilyID"); w.WriteValue(d.FamilyID.ToString());                                

w.WriteEndElement(); w.WriteStartElement("FileName"); w.WriteValue(d.FileName ); w.WriteEndElement(); w.WriteStartElement("FileSize"); w.WriteValue(d.FileSize); w.WriteEndElement(); w.WriteStartElement("URL"); w.WriteValue(d.URL); w.WriteEndElement(); }
w.WriteEndElement(); //products
}
StreamReader r = new StreamReader(ms,Encoding.UTF8);
ms.Seek(0, SeekOrigin.Begin);
return r.ReadToEnd();

Very simple, However when I check the returned XML string,  it is always cut in between and a proper XML is not formed. As the data is localized there may be a possibility that there is some funny character came in between and stream reader ReadToEnd function got confused. Debugging with 40+ languages in data was a nightmare, and I had no clue why streamreader is not working as it is suppose to work. There were other problems too, If any of the values are null, TextWriter.WriteValue complain very loudly. So every where I have to put a check to see if the value is null or not. However, the XElement came to my rescue. XElement does take all the possible data types, and does not complain if any of the data type is null.

XElement x = new XElement("Products");
foreach (DownloadDetails d in downloads) {
x.Add(new XElement("Product",
new XElement("FamilyID", d.FamilyID.ToString()),
new XElement("FileName", d.FileName),
new XElement("FileSize", d.FileSize),
new XElement("URL", d.URL + ""))); }

XElement not only handles NULL gracefully but does not complain about any of the unknown characters which threw StreamReader off. Clearly XElement is a better choice to make  XML out of data, here is another example of XElement use.

With this blog entry, I want to make sure that you will never ever forget their definition and never get mixed up between these two patterns.

Decorator pattern is a design pattern that allows new/additional behavior to be added to an existing class dynamically. Decorators provide a flexible alternative to sub classing for extending functionality.

The decorator pattern can be used to make it possible to extend (decorate) the functionality of a class at runtime. Taking the example from Allan Shalloway book, Design patterns explained. Suppose you have to print a SalesTicket, and depending up on sales, you may need to print multiple headers and multiple footers. Decorator pattern helps you control headers, by chaining together the headers desired in the correct order.

Header1->Header2->Header3->Concrete Component.

The Decorator pattern separates the dynamic building of this chain of functionality from the client. It will also separates the building of the chain from the chain components (Header and Footer).

Now, here is a way to remember it forever, a different way to think is, Decorator pattern help decorate. decorator pattern help us create  necklaces with different bead color combinations.  Decorator pattern is chain of objects. To achieve this you wrap the original bead class with a decorator class.

Suppose we need to write a program, which make different necklaces with different color combinations of beads. Because of decorator pattern it is fairly easy to build the different necklaces and the change in code can be limited to factory class.

Here is the code:

using System;

namespace Decorator
{

///
/// The Beads abstract class
///

abstract class Beads
{
        public abstract void MakeNecklace();
}

///
/// The 'ConcreteComponent' class
///

class ConcerteBead : Beads
{

public override void MakeNecklace()
{
         Console.WriteLine("ConcreteBead");
}
}

///
/// The 'Decorator' abstract class
///

abstract class Decorator : Beads
{
      protected Beads beads;
      public Decorator(Beads _beads)
     {
           this.beads = _beads;
     }
    public Decorator()
   {
    }

 public override void MakeNecklace()
 {
       if (beads != null)
     {
         beads.MakeNecklace();
      }
 }
}

class RedBeads : Decorator
{
       public RedBeads(Beads _beads)
      {
           beads = _beads;
        }
       public override void MakeNecklace()
      {
          base.MakeNecklace();
         Console.WriteLine("RED");
     }
}

class GreenBeads : Decorator
{
         public GreenBeads(Beads _beads)
        {
              beads = _beads;
        }
        public override void MakeNecklace()
       {
          base.MakeNecklace();
          Console.WriteLine("GREEN");
        }
}

class WhiteBeads : Decorator
{
        public WhiteBeads(Beads _beads)
       {
           beads = _beads;
        }
         public override void MakeNecklace()
         {
            base.MakeNecklace();
            Console.WriteLine("WHITE");
         }
}

class Program
{
        static void Main()
       {
           Beads beads = BeadFactory(true);
           beads.MakeNecklace();
           Console.ReadKey();
       }

private static Beads BeadFactory(bool Colorful)
{
           if (Colorful)
         {
                return new GreenBeads(new RedBeads(new WhiteBeads(new ConcerteBead())));
          }
else
        {
              return new WhiteBeads(new WhiteBeads(new WhiteBeads(new ConcerteBead())));

       }
  }
 }
}

The output of the above code will be:

Concrete Bead
WHITE
RED
GREEN

To decouple client from the chain of object,  a factory objects can instantiate the chains based upon some configuration information.  Instantiation of the chains of objects is completely decoupled from the Client objects that use it. Decorator Allows for extending the functionality of an object without resorting to subclassing.

May be tomorrow we will talk about Facade pattern.

The system ran fine for couple of days, then suddenly the ETL processed slowed and the ETL complete time increased by many folds, from 15 minutes it reached to four hours or so.

Our first attempt to find the cause of slow query performance was to check the Query plan. There I noticed that we are using an Index Lookup.

   SELECT @DLStartTime = MIN(DownloadTime)      FROM etl.VLSCRawDataFromStager(nolock)      WHERE (DownloadURI = @DownloadURI AND ASID = @ASID) OR TransactionID = @TransactionId</p>  <p>

We had couple of non-clustered indexes on DownloadURI, ASID and on TransactionId. And a clustered Index on RowId. Looking at the query, you may think the query must be using these simple non-crusted indexes to get the DownloadTime. However, the Query plan looks like as follows:

Query is using the key lookup operator. The use of a Key Lookup operator in a query plan indicates that the query might benefit from performance tuning. For example, query performance might be improved by adding a covering index. read more on http://msdn.microsoft.com/en-us/library/bb326635.aspx

Detail of two non-clustered indexes seeks, output shows as  ‘RowId’ and then this ‘RowId’ is used in turn to do a Key lookup on clustered Index for the output ‘DownloadTime’. Key lookup fetches the extra columns from the clustered index when the non clustered index that’s used to retrieve the rows doesn’t have all the columns required.

Bookmark lookup is not a cheap operation. The  plan can be improved by  adding one or more columns to an existing index so as to eliminate a bookmark lookup, in SQL Server 2005 onwards, you can add columns using the include clause of the create index statement.  Included columns are more efficient than key columns; they save disk space and make searching and updating the index more efficient.  SQL decide on the query plan depending on the number of rows returned. Read more on

http://sqlserverpedia.com/blog/sql-server-bloggers/key-lookup-threshold/comment-page-1/
http://blogs.msdn.com/craigfr/archive/2006/06/30/652639.aspx

We modified one of the index and included other columns.

 CREATE NONCLUSTERED INDEX [VLSCRawDataFromStager_ndx2] ON [etl].[VLSCRawDataFromStager]    (  [DownloadTime] ASC,  [DownloadURI] ASC,  [ASID] ASC,  [TransactionID] ASC   )     <br />GO</p>  <p>

This change removed the Key lookup from the query plan, and suddenly our performance increased by many fold.

It worked for few hours, but before our celebration beer can become warm, the performance again tanked. Now what?

What the hell happened?  For every run we are truncating the tables, and the system ran fine for weeks, now it is tanking, specially when we removed the nasty key lookup. We thought we improved the system but system performance is not improved.

Now, I used sys.dm_db_index_physical_stats check the indexes, this returns fragmentation information for the data and indexes of the specified table or view.

 select * from sys.dm_db_index_physical_stats (DB_ID(N'DataBaseName'), OBJECT_ID(N'TableName'), NULL, NULL , NULL); 

and of course we see the indexes are fragmented more then 95%. I  realized the truncate table statement does not get rid of Indexes, TRUNCATE TABLE removes all rows from a table, but the table structure and its columns, constraints, indexes and so on remain. read more http://msdn.microsoft.com/en-us/library/aa260621(SQL.80).aspx

Overlooking this fact threw us in wrong direction, else this is the first thing we should be checking, but we thought truncate table is cleaning the indexes too, which turn out not be the case. However, the simple fix was to get rid of indexes too, and recreate them after every truncate. This fix bought our ETL performance back to acceptable level.

P.S

However the result set from sys.dm_db_index_physical_stats has index_id and object_id as the column name. You may waste few minutes to write query to get the index name from these ids.  So here is the query:

 select i.*, stat.* from sys.dm_db_index_physical_stats (DB_ID(N'DataBaseName'), OBJECT_ID(N'TableName'), NULL, NULL , NULL) stat join sys.indexes i on i.object_id = stat.object_id and i.index_id = stat.index_id 

OBJECT_ID is evaluated in the current database context, not in the context of the database which you specified in for DB_ID. If the current database does not have the table, then Object_ID will return NULL and your result will be wrong

To get all the indexes in a database it is simply:

 select i.*, stat.* from sys.dm_db_index_physical_stats (DB_ID(N'AdventureWorks'), NULL, NULL, NULL , NULL) stat join sys.indexes i on i.object_id = stat.object_id and i.index_id = stat.index_id where stat.avg_fragmentation_in_percent &gt; 10   <p></p> 

Read more at http://msdn.microsoft.com/en-us/library/ms188917.aspx

 select top 1000  LimitedProgramTBNEventID into #Events from LimitedProgramTBNEvent(NOLOCK)  	where eventStatus = 7 

we changed it as follows:

 DECLARE @Events table (LimitedProgramTBNEventID int NOT NULL) insert into @Events select top 1000 LimitedProgramTBNEventID 	from LimitedProgramTBNEvent (NOLOCK)where eventStatus = 7  

Here is another reason for not to use the Temp table. If you have a temp table then you cannot use the ‘Display Estimated execution plan"’ option in the query analyzer. When a query is run using the "Display Estimated Execution Plan" option, it does not really run so no temp tables are created. Since they are not created, any references to them in the code will fail, which prevents an estimated execution plan from being created. 

select distinct t.[name], c.* from sys.tables t
    join sys.columns c on t.object_id = c.object_id
             and c.is_identity = 1 and c.is_replicated = 0

this query basically tells that you have tables which have identity columns but they are not created with ‘NOT FOR REPLICATION’ option.

If that is the case then your identity columns are out of synch between publisher and subscriber. You can fix it by setting the is_replicated column to 1.

EXEC %%
ColumnEx(
ObjectID = @
object_id,
Name = @
identity_column).
SetIdentityNotForRepl(VALUE = 1)

<Dictionary>
<Word Spelling="Munificent">
    <Meaning>very generous</Meaning>
    <Synonyms />
    <Antonyms />
    <Example />
  </Word>
  <Word Spelling="Anfractuosity">
    <Meaning>Twist or turn, A complicated or involved process. </Meaning>
    <Synonyms />
    <Antonyms />
    <Example> The anfractuosities of his intellect and temper. –Macaulay </Example>
  </Word>
</Dictionary>

LINQ made job to read the data and convert it to XML very easy, as you can see all this happen in one line.

using System;
using System.Linq;
using System.Text;
using System.Data.OleDb;
using System.Xml.Linq;
using System.Data.Linq;
using System.Data.Linq.Mapping;

namespace Convert2XML
{
    class Program
    {
static void Main(string[] args)
        {
                  string ConnectionString = @"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=Vocabulary.mdb;Persist Security Info=false";
                  DataContext db = new DataContext(new OleDbConnection(ConnectionString)) ;
                  Table <words> tblwords =   db.GetTable<words>();
                  System.IO.File.WriteAllText("Data.XML",
                            ( new XElement("Dictionary", from w in tblwords
                                                   select new XElement("Word" ,
                                                          new XAttribute("Spelling", w.Word),
                                                   new XElement("Meaning", w.Meaning),
                                                   new XElement("Synonyms", w.Synonyms),
                                                   new XElement("Antonyms", w.Antonyms),
                                                   new XElement("Example", w.Example)
                                                   )
                                           )
                             ).ToString()
                             , Encoding.UTF8);
        }

        [Table(Name = "words")]
        public class words
        {
            [Column]
            public string Word;
            [Column]
            public string Meaning;
            [Column]
            public string Synonyms;
            [Column]
            public string Antonyms;
            [Column]
            public int weight;
            [Column]
            public string Reference;
            [Column]
            public string Example;
            [Column]
            public string link;
        }
    }
}

If you don’t have OLEDB 12.0 then You can use the following connection string too

string ConnectionString = @”Provider=Microsoft.Jet.OLEDB.4.0;Data Source=Vocabulary.mdb;Persist Security Info=True” ;

If you don’t have Microsoft.ACE.OLEDB.12.0 then get it from here

http://www.microsoft.com/downloads/details.aspx?familyid=7554F536-8C28-4598-9B72-EF94E038C891&displaylang=en

These oledb providers are 32 bit, so make sure to compile your project for 32 bit on a 64 bit machine.
Error handling and comments are removed for the brevity.

Using  Microsoft.AnalysisServices  Object API, you can travel the whole Hierarchy of objects  and completely administer an Analysis Service Instance. From the connection Object you can get to the Server and database instances. And in database you can transverse the Cube and their different measures. Once you get hold of a measure then you can transverse the different partitions of measures and dig deep in to each partition query definitions etc.

To keep things simple, there is one prerequisite to name your measure partitions in the in following format: MeasureName_DateSKey. DateSKey is a long number showing the date from 1^st Jan 1990. To get the DateSKey for a date you can use the following formula:

 DateTime d;
d.Date.Subtract(new System.DateTime(1900, 1, 1, 0, 0, 0)).Days

So, if your measure name is ‘Distinct Downloaders Daily’ then it may have following partitions for every 7 days as follows: Distinct Downloaders Daily_39812, Distinct Downloaders Daily_398129… etc

Open a new DTS package, drag the script component and double click to start editing a C# script.

Include the following library:

using Microsoft.AnalysisServices;
using System.Collections.ObjectModel;

Define a new structure as measure, to hold the definition of your measures and there partition boundaries.

// Structure used to keep the Measure name and their partition durations
struct Measure
{
       public string MeasureName;
      // Partition duration in days
      public int iPartitionDays;
      public Measure(string measureName, int ipartitionDays)
        {
          MeasureName = measureName;
          iPartitionDays = ipartitionDays;
        }
}

Now define a collection of measures which you want to partition. Sure, this information you can get from a Config file too … It is hardcoded in code for the illustration purpose.

Collection Measures = new Collection();
Measures.Add(new Measure("DownLoad Summary", 7));
Measures.Add(new Measure("Distinct Downloaders Daily", 7));
Measures.Add(new Measure("Distinct Downloaders Weekly", 30));
Measures.Add(new Measure("Top Referer", 30));

Now you can use this collection to call a method called ProcessCube. Here is the code, removed all the error handling for the sake of brevity.

private void ProcessCube( Collection Measures, string cubeName)
{
Server oServer = new Server();
try
{
// Get Connection object and then get the Server and Database name from DTS connection object
ConnectionManager oConnection = null;
for (int i = 0; i < Dts.Connections.Count; i++)
{
oConnection = Dts.Connections[i];
if (oConnection.Name == "CubeConnectionName") //Connection name as defined in ETL package
{
break;
}
else
oConnection = null;
}

string sServer = (string)(oConnection.Properties["ServerName"].GetValue(oConnection));
string sDatabase = (string)(oConnection.Properties["InitialCatalog"].GetValue(oConnection));
oServer.Connect(sServer);// connect to the server and start scanning down the object hierarchy
Database oDB = oServer.Databases.FindByName(sDatabase);
Cube oCube = oDB.Cubes.FindByName(cubeName);
foreach (Measure m in Measures)
{
CreatePartition(oCube, m.MeasureName, m.iPartitionDays); //Create old paritions
DeletePartitions(oCube, m.MeasureName, m.iPartitionDays); // Delete old partitions

}
}
finally
{
if (oServer.Connected)
oServer.Disconnect();
}
}

///
/// Create a new partition based on the skipdays
///
/// Cube Name to be processed
/// MeasureName which need to be partitioned
/// Partition duration e.g. for daily=1, weekly=7, Monthly =30
/// true on success
///
///
/// This code Creates the new partition names, using the default Measure name as the partition name and the
/// DateSKey. for e.g. if the measure name is Browser and the DateSKey=39241
/// then the partition name will be 'Browser_39241'
///
///
///

bool CreatePartition(Cube oCube, string measureName, int iDaysToSkip)
{
Byte[] dataBytes = new Byte[5000];
QueryBinding oQueryBinding;
bool bflag = false;
int iCurrentDateSKey = (int)Dts.Variables["CubeCurrentDateSKey"].Value; //Get the currentDateSkey
MeasureGroup oMeasureGroup = oCube.MeasureGroups.FindByName(measureName); // Find all measure groups
if (oMeasureGroup == null) return false;
Partition oPartition = oMeasureGroup.Partitions[oMeasureGroup.Partitions.Count - 1]; //Get the Last partition
int LastCubeDateSkey = GetPartitionDateSKey(oPartition);//Get the lastkey DateSkey
string DefaultPartitionName = oMeasureGroup.Name; //Get the Partition name first part
string sNewPartitionName = string .Empty ;
if (LastCubeDateSkey == 1) //if it is first default partition
{
sNewPartitionName = DefaultPartitionName + "_" + iCurrentDateSKey; //Making the new partition Name
}
else
{
if (LastCubeDateSkey + iDaysToSkip <= iCurrentDateSKey)
sNewPartitionName = DefaultPartitionName + "_" + iCurrentDateSKey;
else
return true; //No need to create the partition yet. Every thing is fine, go back
}

//Just another check, if the partition already existing then skip
oPartition = oMeasureGroup.Partitions.FindByName(sNewPartitionName);
Partition oPartitionNew;
if (oPartition == null)
{
//Did not get the partition, lets create one,Get the last partition
oPartition = oMeasureGroup.Partitions[oMeasureGroup.Partitions.Count - 1];
//Clone the properties from the last partition to the new partition.
oPartitionNew = oPartition.Clone();
oPartitionNew.ID = sNewPartitionName;
oPartitionNew.Name = sNewPartitionName;
oQueryBinding = oPartitionNew.Source as QueryBinding;
if (oQueryBinding == null)
{
return true; // No Query, No Partition
}
oQueryBinding.QueryDefinition = GetNewQuery(oQueryBinding.QueryDefinition, iCurrentDateSKey, iDaysToSkip);
if (oQueryBinding.QueryDefinition == null || oQueryBinding.QueryDefinition == string.Empty)
{
Dts.Log("Partition : " + oPartitionNew.Name + " Empty Query returned", 0, dataBytes);
Dts.Events.FireInformation(0, "CreatePartition", "Partition : " + oPartitionNew.Name + " Empty Query returned", null, 0, ref bflag);
return false;
}

if (oMeasureGroup.Partitions.Contains(oPartitionNew))
{
Dts.Log("Partition : " + oPartitionNew.Name + " Already exists.", 0, dataBytes);
return true; //Oh! what a surprise, this partition already existing.
}
oMeasureGroup.Partitions.Add(oPartitionNew);
try
{
oPartitionNew.Update();
Dts.Log("Partition updated: " + oPartitionNew.Name, 0, dataBytes);
Dts.Events.FireInformation(0, "CreatePartition", "Partition updated: " + oPartitionNew.Name, null, 0, ref bflag);
oPartitionNew.Process(ProcessType.ProcessFull);
Dts.Log("Partition Processed: " + oPartitionNew.Name, 0, dataBytes);
Dts.Events.FireInformation(0, "CreatePartition", "Partition Processed: " + oPartitionNew.Name, null, 0, ref bflag);

}
catch (Exception e)
{
if (oPartitionNew != null)
Dts.Log("Error in Create partition: " + oPartitionNew.Name + "Exception: " + e.Message, 0, dataBytes);
else
Dts.Log("Error in Create partition: Unknown. Exception: " + e.Message, 0, dataBytes);
return false;
}
}
return true;
} //~Create partitio ends here

///
/// This function Gets the partition DateSKey from the partition name
///
/// On Success the right DateSKey, on error zero
///
private int GetPartitionDateSKey(Partition oPartition)
{
if (oPartition == null) return 0;
try
{
return Convert.ToInt32(oPartition.Name.Substring(oPartition.Name.LastIndexOf("_") + 1));
}
catch (Exception ex)
{
return 0;
}
}

///
/// Get the new Query for the partition.
///

string GetNewQuery(string sourceQuery, int iCurrentDateSKey, int iDaysToSkip)
{

string mainQuery = string.Empty;
string newQuery = string.Empty;
string newCondition;
//From the Query get the Where clause
int LastIndex = sourceQuery.ToUpper().LastIndexOf("WHERE") ;
if (LastIndex > 0)
{
mainQuery = sourceQuery.Substring(0, LastIndex);
newCondition = " [DateSKey] >= " + Convert.ToString(iCurrentDateSKey) + " AND [DateSKey] = " + Convert.ToString(iCurrentDateSKey) + " AND [DateSKey] < " + Convert.ToString(iCurrentDateSKey + iDaysToSkip ) + " ";
newQuery = sourceQuery + " WHERE " + newCondition;
}

return newQuery;
}

//Deletes old partitions based on the data retention period.
bool DeletePartitions(Cube oCube, string measureName, int iDaysToSkip)
{
Collection PartitionTobeDeleted = new Collection();
int iDateSkeyTobeDeleted ;
Byte [] dataBytes = new Byte [5000];
bool bflag = false;
// 13 * 30 = 390 days - 13 months
int iDaysTobeDeleted = 390 ; //TODO: Get from a variable ["DataRetentionPeriodInDays"]

//Getting the number for which partition needs to deleted. old partitions
iDateSkeyTobeDeleted = GetDateSKey (DateTime.Now) - iDaysTobeDeleted ;

MeasureGroup oMeasureGroup = oCube.MeasureGroups.FindByName (measureName);
if (oMeasureGroup == null)
{
Dts.Log("Measure : " + oMeasureGroup.Name + " not found.", 0, dataBytes);
return false;
}
Dts.Log("[DeletePartitions] Working on Measure: " + oMeasureGroup.Name, 0, dataBytes);
string sPartitionName = string.Empty; //used for logging the correct PartitionName
int MaxPartitions = oMeasureGroup.Partitions.Count;
foreach (Partition oPartition2 in oMeasureGroup.Partitions)
{
sPartitionName = oPartition2.Name;
if (!oPartition2.Name.Contains("_"))
continue; //_ is not there it means it is not the partition which we need to delete
try
{
// Get the boundary partition date from partition name
int spartitionBoundaryDateSkey = GetPartitionDateSKey(oPartition2);
if (spartitionBoundaryDateSkey == 0) continue;

//if partition DateSKey is smaller then the partition to be deleted then delete
//MaxPartition protects us from deleting all the partitions from the measures
if (spartitionBoundaryDateSkey 1)
{
MaxPartitions--;
//can not drop object here because of foreach.
PartitionTobeDeleted.Add(oPartition2);
#region Log
Dts.Log("Found partition that needs to be dropped: " + oPartition2.Name, 0, dataBytes);
Dts.Events.FireInformation(0, "DeletePartition", "Found partition that needs to be dropped: " + oPartition2.Name, null, 0, ref bflag);
#endregion
}
}
catch (Exception e)
{
Dts.Log("Error in Delete partition: " + sPartitionName + "Exception: " + e.Message, 0, dataBytes);
Dts.Events.FireInformation(0, "DeletePartition", "Error in Delete partition: " + sPartitionName + "Exception: " + e.Message, null, 0, ref bflag);
}
}
//Deleting partition from the measure group
foreach (Partition oPartition in PartitionTobeDeleted)
{
XmlaWarningCollection warningColln = new XmlaWarningCollection();
oPartition.Drop(DropOptions.Default, warningColln);
}
return true;
}

Push subscription replication of character data with SQL Server 2008 running on Windows Server 2008 yielded a 104 percent increase over SQL Server 2005 running on Windows Server 2003, and pull subscription replication of the same data yielded a 1,298 percent gain.

Lot of performance improvement is attributed to windows server TCP/IP Stack and SQL Native Client driver. Different reader and writer thread behavior resulted in push and pull performance difference.

Reader thread fills two buffers  of 40 KB  and signals the Writer thread to write the buffer on subscriber, so the 40 KB packet moves thru and fro over the wire, but in case of Pull model Writer thread pulls the data in one huge chunk as big as TCP/IP let it get.  One of the most significant improvements to Windows Server 2008 is the autotuning of the receive window size, designed for high-latency environments. In previous versions of the operating system, the maximum window size was limited by the 16-bit Window field in the TCP header amounting to a maximum window size of 64 KB. In Windows Server 2008, by combining the Window field together with a Scale Factor field of the TCP header, the window can be scaled or tuned, up to 16 megabytes (MB) in size.

Read the full study here http://msdn.microsoft.com/en-us/library/dd263442.aspx