Processing multiple rows and multiple columns in SSIS

I previously posted a problem with result set bindings but I have not been able to resolve my problem. I guess all this comes with being new to programming in this environment! Anyway, I am trying to figure out how to process from an ADO.NET connection multiple rows with multiple columns. I have to read and manipulate each row. I was originally looking at using a foreach loop but have not been able to get it to work. One reply to my previous thought I should be using a data task to accomplish this. Could someone tell me the best way to handle this situation? As a note, I am new to programming in SSIS and basically trying to learn it as I go so please bear with me! Thanks in advance!

My first suggestion would be to run through the BOL tutorial for SSIS. This will show you some information on the data flow task and some of the things it can help you do. I would agree with the previous post that you should be using a data flow task, with an ado.net connection source, the misc. transformations that will manipulate your data and then your destination.

You might consider picking up one of the various SSIS books if you are going to be doing any amount of work in this environment (either the wrox professional or Kirk Haselden's books are both good starts).

processing comma-delimited strings (redux)

Hello,

I have no trouble converting a comma-delimited string of values into multiple records, bur have recently encountered a problem that's giving me a headache - hopefully someone on the forum already has some experience doing this task:

create table tester (col1 int, col2 varchar(1000), col2 varchar(1000))
insert into tester values(1, '1,3,5,7', 'a,c,e,g,')
insert into tester values(2, '11,13,15,17', 'aa,ac,ae,ag,')

There is no correlation between rows, but between the 2 varchar columns is a positional relationship - in the first record, the '1' in col2 relates to the 'a' in col 3, same for the '3' and 'c', on and on. The values within each of the comma-delimited strings of the 2 columns are positionally related. Say they could be time and temperature values, with a string of time values in col1 and a string of related temps in col2. This is data from an external system that I have no control over, but must load the data into my system

I need to write a select statement that will return the contents like so:

1, 1, a
1, 3, c
1, 5, e
1, 7, g
2, 11, aa
2, 13, ac
2, 15, ae
2, 17, ag

Has anyone encountered such as this? Any clues or code snippets?

Thanks for any ideas,

saWhat split function do you use? If it uses a tally table you could return the numbers and link on those.
Not used but how about Kristen's here?
http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=50648&whichpage=2|||your "relationships" are based on offsets...so what value or meaning does that infer??

what the hell...use a cursor

actually a simple loop and udf will do

Process with status SLEEPING.

Hi all,

We have a web application with multiple users connecting to SQL Server 2005 database.

Once the users start using the system, I could see in the Process Info from the activity monitor with a several process with status as sleeping and command as Awaiting Command.

I have few questions regarding this for which simple answers would help me ( I searched and couldnt understand from the pages that explained the process information )

My questions are,

1. Does the processes with status sleeping and command as Awaiting Command are needed for the application even after the operations specific to them are complete?

2. Does these process affect the performance ?

3. Does killing these process would affect the application connectivity ?

4. Why do we get several process with status sleeping and command as Awaiting Command? Is this because of the front end programming errors ?

5. What should we do to avoid the increasing number of process with status sleeping and command as Awaiting Command?

Any help would be appreciated.

Thanks in advance,

DBLearner.

Hi DBLearner,

1. Does the processes with status sleeping and command as Awaiting Command are needed for the application even after the operations specific to them are complete?

-- No

2. Does these process affect the performance ?

-- It will if you have left thousand connections open. Connections need resources.

3. Does killing these process would affect the application connectivity ?

-- You shouldn't have to go around killing these processes. I don't think it would affect. More work for you.

4. Why do we get several process with status sleeping and command as Awaiting Command? Is this because of the front end programming errors ?

-- Web app doesnot close the connection after the work has been done.

5. What should we do to avoid the increasing number of process with status sleeping and command as Awaiting Command?

-- Your web application needs to be configured or modified so that whenever it needs to do any work on SQL Server it does the following:

1. Opens a connection.

2. Do the SQL work e.g. execute a stored procedure or select some data back.

3. Close the connection.

regards

Jag

|||Most web applications use a "connection pool" for SQL connections. This is a normal operation. Leave it alone.

Openning and closing connections is very time consuming. You do NOT want to open and close several thousand connections per min.

|||

Thanks Jag & Tom. But what should I do to optimise my connections and applications ?

DBLearner

|||It depends on your web application. You will need to look and see what mechanisim it is using to access the SQL server.

|||

Agree with Tom, if it is using Connection pooling then the connections should be left alone.

But if there is no connection pooling present, then the web application should be closing there connection after the work has been completed.

regards

Jag

|||

Hi,

Actually I am having a same problem where there are quite a number of process with sleeping status in the database. And I noticed that all these processes are called by the crystal report (using store procedures to get the data from db). FYI, I am using VS.Net 2003 and crystal report 9.0. Does anyone have idea on the solution for the this problem? Thanks.

|||

As Jag and Tom say - if this is connection pooling then you are best to leave it alone.

What you are seeing is one sleeping process for each active pool connection. They are waiting, ready to respond quickly when needed (without logging in again). The only time you have a problem is if the number of these sleeping connections does not hold steady but instead rises dramatically (implying that either it is not pooling somewhere or the pooling is failing).

The pool of connections means that the application can be responsive, and if you start killing them then there will be a reduction in that responsiveness (and servicing requests is the database's function). Even worse if you manage to kill a connection just as it is activated to service a request you could cause all kinds of problems (hopefully just longer delays).

|||This is not a "problem", this is how it WORKS. As said before, leave it alone unless you are seeing hundreds of sleeping processes overloading your SQL server.

Generally it opens new pipes when other pipes are busy. So if you have 10 sleeping processes, it is because you NEEDED THEM sometime before. It should timeout the pipe and close them after 60 mins or some time of inactivity.

Crystal does have some options to change the min/max connections, but I would leave them the default unless you are seriously having a problem.

|||

I have the same problem but i don't have hundread of conn just about 40.

These 40 sleeping connections thought eat me about 90% from the CPU and reset users Session and they loose their products before ariving to the payment.

I did close all the recordsets and connections and still appear new sleeping connections.

Any suggestions?

i used for any recordset opened

rs.close

set rs = nothing

still make me these problems i don't know what to do anymore

Any help will be appreciated

thank you

Process with status SLEEPING.

Hi all,

We have a web application with multiple users connecting to SQL Server 2005 database.

Once the users start using the system, I could see in the Process Info from the activity monitor with a several process with status as sleeping and command as Awaiting Command.

I have few questions regarding this for which simple answers would help me ( I searched and couldnt understand from the pages that explained the process information )

My questions are,

1. Does the processes with status sleeping and command as Awaiting Command are needed for the application even after the operations specific to them are complete?

2. Does these process affect the performance ?

3. Does killing these process would affect the application connectivity ?

4. Why do we get several process with status sleeping and command as Awaiting Command? Is this because of the front end programming errors ?

5. What should we do to avoid the increasing number of process with status sleeping and command as Awaiting Command?

Any help would be appreciated.

Thanks in advance,

DBLearner.

Hi DBLearner,

1. Does the processes with status sleeping and command as Awaiting Command are needed for the application even after the operations specific to them are complete?

-- No

2. Does these process affect the performance ?

-- It will if you have left thousand connections open. Connections need resources.

3. Does killing these process would affect the application connectivity ?

-- You shouldn't have to go around killing these processes. I don't think it would affect. More work for you.

4. Why do we get several process with status sleeping and command as Awaiting Command? Is this because of the front end programming errors ?

-- Web app doesnot close the connection after the work has been done.

5. What should we do to avoid the increasing number of process with status sleeping and command as Awaiting Command?

-- Your web application needs to be configured or modified so that whenever it needs to do any work on SQL Server it does the following:

1. Opens a connection.

2. Do the SQL work e.g. execute a stored procedure or select some data back.

3. Close the connection.

regards

Jag

|||Most web applications use a "connection pool" for SQL connections. This is a normal operation. Leave it alone.

Openning and closing connections is very time consuming. You do NOT want to open and close several thousand connections per min.

|||

Thanks Jag & Tom. But what should I do to optimise my connections and applications ?

DBLearner

|||It depends on your web application. You will need to look and see what mechanisim it is using to access the SQL server.

|||

Agree with Tom, if it is using Connection pooling then the connections should be left alone.

But if there is no connection pooling present, then the web application should be closing there connection after the work has been completed.

regards

Jag

|||

Hi,

Actually I am having a same problem where there are quite a number of process with sleeping status in the database. And I noticed that all these processes are called by the crystal report (using store procedures to get the data from db). FYI, I am using VS.Net 2003 and crystal report 9.0. Does anyone have idea on the solution for the this problem? Thanks.

|||

As Jag and Tom say - if this is connection pooling then you are best to leave it alone.

What you are seeing is one sleeping process for each active pool connection. They are waiting, ready to respond quickly when needed (without logging in again). The only time you have a problem is if the number of these sleeping connections does not hold steady but instead rises dramatically (implying that either it is not pooling somewhere or the pooling is failing).

The pool of connections means that the application can be responsive, and if you start killing them then there will be a reduction in that responsiveness (and servicing requests is the database's function). Even worse if you manage to kill a connection just as it is activated to service a request you could cause all kinds of problems (hopefully just longer delays).

|||This is not a "problem", this is how it WORKS. As said before, leave it alone unless you are seeing hundreds of sleeping processes overloading your SQL server.

Generally it opens new pipes when other pipes are busy. So if you have 10 sleeping processes, it is because you NEEDED THEM sometime before. It should timeout the pipe and close them after 60 mins or some time of inactivity.

Crystal does have some options to change the min/max connections, but I would leave them the default unless you are seriously having a problem.

|||

I have the same problem but i don't have hundread of conn just about 40.

These 40 sleeping connections thought eat me about 90% from the CPU and reset users Session and they loose their products before ariving to the payment.

I did close all the recordsets and connections and still appear new sleeping connections.

Any suggestions?

i used for any recordset opened

rs.close

set rs = nothing

still make me these problems i don't know what to do anymore

Any help will be appreciated

thank you

Process Multiple Reports at once?

I have a project that has 10 reports that I need to schedule in various
combinations with different parameters on 5 different schedules, excel output
to a file server. Possibly up to 170 different reports a day so regular
subscription set up would be a nightmare to maintain. I have looked at
Data-Driven subscriptions however what I need is different report
combinations generated not different variations of one report. Something like
this:
Schedule 1
Package 1
Reports A, B, C, D, G (All use same data parameters)
Output to Folder 1
Package 2
Reports B, C, D, E, F
Output to Folder 2
Schedule 2
Package 1...etc.
Is there a way to accomplish this with SSRS, SSIS or SOAP (which I know nada
about) - What is the best process? Subreports are not viable since the
worksheet names cannot be set. (Using 2005)
Any suggestions appreciated.
--
ABNo ideas anyone? Must be asking the impossible :)
--
Allison
"allison" wrote:
> I have a project that has 10 reports that I need to schedule in various
> combinations with different parameters on 5 different schedules, excel output
> to a file server. Possibly up to 170 different reports a day so regular
> subscription set up would be a nightmare to maintain. I have looked at
> Data-Driven subscriptions however what I need is different report
> combinations generated not different variations of one report. Something like
> this:
> Schedule 1
> Package 1
> Reports A, B, C, D, G (All use same data parameters)
> Output to Folder 1
> Package 2
> Reports B, C, D, E, F
> Output to Folder 2
> Schedule 2
> Package 1...etc.
> Is there a way to accomplish this with SSRS, SSIS or SOAP (which I know nada
> about) - What is the best process? Subreports are not viable since the
> worksheet names cannot be set. (Using 2005)
> Any suggestions appreciated.
> --
> AB|||You could do it from within a programming environment... but I don't
believe this is possible within the standard portal.|||I need to do something similar...
I know that I've done this sort of thing with Active Reports and Crystal
Reports. When you create 1 report, then a second report, and want to stitch
them together and render as 1 PDF... I haven't come across anything that
pops out in the documentation for RS 2000 that can allow me to do this so
far. Should I be looking at 2005 to accomplish this or is 2000 sufficient?
"toddnappi@.gmail.com" wrote:
> You could do it from within a programming environment... but I don't
> believe this is possible within the standard portal.
>