Footer rows to skip: import

Showing posts with label import. Show all posts

Thursday, March 29, 2012

field mapping

Hi all,
I have written many stored procedures in our database to import data from
another database. Now I need to create a report that lists the source table
and column names and their corresponding destination table and column names.
I think it will be difficult and time-consuming for me to open each
procedure and copy this information manually. Are there any procedures or
queries that can help me in generating this report.
Thanks in advance.Hi,
maybe this will help you ...
SELECT OBJECT_NAME(id) AS Source, OBJECT_NAME(depid) AS Uses
FROM dbo.sysDepends WHERE id = OBJECT_ID('proc_sys_InsertPropertyValue'
)
All dependencies of objects are stored in the relation dbo.sysdepends!
Gru, Uwe Ricken
MCP for SQL Server 2000 Database Implementation
GNS GmbH, Frankfurt am Main
http://www.gns-online.de
http://www.memberadmin.de
http://www.conferenceadmin.de
________________________________________
____________
dbdev: http://www.dbdev.org
APP: http://www.AccessProfiPool.de
FAQ: http://www.donkarl.com/AccessFAQ.htmsql

Wednesday, March 7, 2012

faster box but slower database

Hi,
Got two SQL servers (production,dev) and I noticed recently that my
production servers is much slower when running one import stored
procedure. The stored proc only uses tables variables a lot and only
performs selects on actual tables. The specs are:
Productions server:
SQL 2000 v. 8002039 (SP4)
2x Dual Core Intel Xeon 3,2GH
1GB Memory
RAID 1
Windows 2003 Server
Development server:
SQL 2000 v. 800194 (no service pack)
1x Dual Core Intel Pentium 4 3GHz
Windows XP Professional
The stored procedures takes 20 sec running on the dev box but 2.4 min
on the production box.
All indexes and tables are the same in both databases. I tried
updating the stats, rebuild indexes, recompile the stored proc, free
the proc cache but without any luck.
Any help would be very much appricated.
KristjanOne thing I forgot to mention. The execution tree found in the
profiler differ for the same query in diffrent databases.|||You didn't mention how much ram is on the dev box.
Also how many rows and what is the DB size? 1GB of ram on a production SQL
box with that kind of CPU seems way low to me.
Mike
"kgb" <kristjan.gudni.bjarnason@.gmail.com> wrote in message
news:1138125661.913476.224860@.g14g2000cwa.googlegroups.com...
> Hi,
> Got two SQL servers (production,dev) and I noticed recently that my
> production servers is much slower when running one import stored
> procedure. The stored proc only uses tables variables a lot and only
> performs selects on actual tables. The specs are:
> Productions server:
> SQL 2000 v. 8002039 (SP4)
> 2x Dual Core Intel Xeon 3,2GH
> 1GB Memory
> RAID 1
> Windows 2003 Server
> Development server:
> SQL 2000 v. 800194 (no service pack)
> 1x Dual Core Intel Pentium 4 3GHz
> Windows XP Professional
> The stored procedures takes 20 sec running on the dev box but 2.4 min
> on the production box.
> All indexes and tables are the same in both databases. I tried
> updating the stats, rebuild indexes, recompile the stored proc, free
> the proc cache but without any luck.
> Any help would be very much appricated.
> Kristjan
>|||The dev box also got 1 GB ram. The databases are almost the same size
because the app is not writing very much on runtime.
The stored proc spends most time on single table containing about
600,000 rows (both dev and production server). By the way, the
profiler tells me the production servers is doing ~10 times more reads
and spending ~10 more CPU than the dev server.
The total databsize is about 0,5 GB but maybe 1/4 of it is accessed
realtime.|||What different you found in Execution plan?
Try to run query on Prod. using option (maxdop 1).
If you have joins check that fields on both sides of the join have same
datatypes.
Regards
Amish Shah|||Tried the maxdop 1 option, no big difference.
The datatypes were ok.
Installed the database on my laptop to reproduce this. With no service
pack the stored proc ran fine (30 secs). With SP4 it ran for minutes.
Seems like updated server optimizer is not doing a good job with my
stored proc. Collecting some more data which I will post to this
thread soon.|||There have been a number of changes from GOLD and SP1 to SP4. A number of
these fixes result in different query plans being generated. It is not a far
comparison of performance between GOLD and SP4. Your GOLD version is also a
lot less secure in that it does not protect against SLAMMER.
Chris
"kgb" <kristjan.gudni.bjarnason@.gmail.com> wrote in message
news:1138274581.095820.120850@.o13g2000cwo.googlegroups.com...
> Tried the maxdop 1 option, no big difference.
> The datatypes were ok.
> Installed the database on my laptop to reproduce this. With no service
> pack the stored proc ran fine (30 secs). With SP4 it ran for minutes.
> Seems like updated server optimizer is not doing a good job with my
> stored proc. Collecting some more data which I will post to this
> thread soon.
>|||This may sound silly, but have you opened the sproc on the SP4 box and
recompiled it on SP4. I have seen a few odd things from time to time when
sprocs were compiled on a lower SPack DB then restored to a differnt SPack
server. Views too.
At least you would give the SP4 query optimizer a chance to take a look at
the sproc to build a new plan to store with the sproc.
Mike
"Chris Wood" <anonymous@.discussions.microsoft.com> wrote in message
news:uu3f3HpIGHA.532@.TK2MSFTNGP15.phx.gbl...
> There have been a number of changes from GOLD and SP1 to SP4. A number of
> these fixes result in different query plans being generated. It is not a
> far comparison of performance between GOLD and SP4. Your GOLD version is
> also a lot less secure in that it does not protect against SLAMMER.
> Chris
> "kgb" <kristjan.gudni.bjarnason@.gmail.com> wrote in message
> news:1138274581.095820.120850@.o13g2000cwo.googlegroups.com...
>> Tried the maxdop 1 option, no big difference.
>> The datatypes were ok.
>> Installed the database on my laptop to reproduce this. With no service
>> pack the stored proc ran fine (30 secs). With SP4 it ran for minutes.
>> Seems like updated server optimizer is not doing a good job with my
>> stored proc. Collecting some more data which I will post to this
>> thread soon.
>|||can you post the DDL and query you are running?|||Currently changing the query to "fit" the SP4 optimizer and think I
almost there, meaning the duration of the query is dropping to what I
expected. The changes sofar involved rewriting of several joins. I
get back to the thread in few hours or less - hopfully with result of
success. Thanks for all your replies. Its amazing to find the support
available on this group.|||Below you find the exec tree for the query, both versions. Being
novice mssql programmer I am not sure what part of the tree is taking
most time. The execution plan makes not much sense to me - the figures
are cost estimated rows seem all wrong.
the execution tree from the profilier is like this for the SP4 box.
Execution Tree
--
Table Insert(OBJECT:(@.f3), SET:(@.f3.[outbound]=RaiseIfNull(1),
@.f3.[groundDur_2]=[t2].[TransitTime],
@.f3.[groundDur_1]=[t1].[TransitTime], @.f3.[duration_3]=[f3].[Duration],
@.f3.[duration_2]=[f2].[Duration], @.f3.[duration_1]=[f1].[Duration],
@.f3.[FlightDate_3]=[f3].[FlightDate],
@.f3.[f1_FlightID]=[f2].[FlightID], @.f3.[flightset_3]=[f3].[flightset],
@.f3.[FlightID_3]=[f3].[FlightID],
@.f3.[fromairport_3]=[f3].[fromairport],
@.f3.[toairport_3]=[f3].[toairport], @.f3.[eta_3]=[f3].[eta],
@.f3.[std_3]=[f3].[std], @.f3.[FlightDate_2]=RaiseIfNull([@.out]),
@.f3.[f0_ID]=[f1].[ID], @.f3.[f0_FlightID]=[f1].[FlightID],
@.f3.[flightset_2]=[f2].[FlightSet], @.f3.[FlightID_2]=[f2].[FlightID],
@.f3.[fromairport_2]=[f2].[FromAirport],
@.f3.[toairport_2]=[f2].[ToAirport], @.f3.[eta_2]=[f2].[ETA],
@.f3.[std_2]=[f2].[STD], @.f3.[flightdate_1]=[f1].[FlightDate],
@.f3.[flightset_1]=[f1].[flightset], @.f3.[FlightID_1]=[f1].[FlightID],
@.f3.[fromairport_1]=[f1].[fromairport],
@.f3.[toairport_1]=[f1].[toairport], @.f3.[eta_1]=[f1].[eta],
@.f3.[std_1]=[f1].[std], @.f3.[ID]=RaiseIfNull([Expr1009]),
@.f3.[FlightNumber_1]=[f1].[FlightNumber],
@.f3.[AirlineCode_1]=[f1].[AirlineCode],
@.f3.[FlightNumber_2]=[f2].[FlightNumber],
@.f3.[AirlineCode_2]=[f2].[AirlineCode],
@.f3.[FlightNumber_3]=[f3].[FlightNumber],
@.f3.[AirlineCode_3]=[f3].[AirlineCode]))
|--Top(ROWCOUNT est 0)
|--Compute Scalar(DEFINE:([Expr1009]=getidentity(1295343679, 2,
'@.f3')))
|--Filter(WHERE:([f3].[std]>[f2].[ETA]+Convert([@.transitDuration])+isnull([t2].[TransitTime],
0)))
|--Nested Loops(Left Outer Join, OUTER REFERENCES:([f2].[ArrTerm],
[f2].[ToAirport], [f3].[depTerm], [f3].[fromairport]))
|--Filter(WHERE:([f2].[STD]>[f1].[eta]+Convert([@.transitDuration])+isnull([t1].[TransitTime],
0)))
| |--Nested Loops(Left Outer Join, OUTER REFERENCES:([f1].[arrTerm],
[f1].[toairport], [f2].[DepTerm], [f2].[FromAirport]))
| |--Nested Loops(Inner Join,
WHERE:([f1].[toairport]<>[a].[AirportCode]))
| | |--Nested Loops(Inner Join,
WHERE:([f2].[ToAirport]<>[b].[AirportCode]))
| | | |--Hash Match(Inner Join,
HASH:([f1].[toaptjoin])=([f2].[FromAptJoin]),
RESIDUAL:([f1].[fromairport]<>[f3].[fromairport] AND
((([f1].[eta]-[f1].[std])*[f3].[dayafter]>0 AND
([f1].[eta]-[f1].[std])*[d].[dayafter]>0) OR ([f1].[eta]>[f1].[std] AND
([f2].[ETA]-[f2].[STD])*[f3].[dayafter]>0))))
| | | | |--Table Scan(OBJECT:(@.f1 AS [f1]),
WHERE:([f1].[outbound]=1))
| | | | |--Bookmark Lookup(BOOKMARK:([Bmk1002]),
OBJECT:([dohop].[dbo].[Flight1] AS [f2]))
| | | | |--Nested Loops(Inner Join, OUTER
REFERENCES:([d].[date], [d].[weekday], [f3].[fromaptjoin]))
| | | | |--Nested Loops(Inner Join)
| | | | | |--Table Scan(OBJECT:(@.flast AS
[f3]), WHERE:([f3].[outbound]=1))
| | | | | |--Table
Scan(OBJECT:(@.weekdays_out AS [d]))
| | | | |--Index
Seek(OBJECT:([dohop].[dbo].[Flight1].[IX3_Flight1] AS [f2]),
SEEK:([f2].[ToAptJoin]=[f3].[fromaptjoin]),
WHERE:((([f2].[ValidFrom]<=[d].[date] AND [f2].[ValidTo]>=[d].[date])
AND [f2].[Status]=0) AND (Convert([f2].[Weekdays])&[d].[weekday])>0)
ORDERED FORWARD)
| | | |--Table Scan(OBJECT:(@.arrivalList AS [b]))
| | |--Table Scan(OBJECT:(@.arrivalList AS [a]))
| |--Clustered Index
Seek(OBJECT:([dohop].[dbo].[Transit].[PK_Transit] AS [t1]),
SEEK:([t1].[Apt1]=[f1].[toairport]),
WHERE:([t1].[Apt2]=[f2].[FromAirport] AND (([t1].[Term1]=' ' AND
[t1].[Term2]=' ') OR ([f1].[arrTerm]=[t1].[Term1] AND
[f2].[DepTerm]=[t1].[Term2]))) ORDERED FORWARD)
|--Clustered Index Seek(OBJECT:([dohop].[dbo].[Transit].[PK_Transit] AS
[t2]), SEEK:([t2].[Apt1]=[f2].[ToAirport]),
WHERE:([t2].[Apt2]=[f3].[fromairport] AND (([t2].[Term1]=' ' AND
[t2].[Term2]=' ') OR ([f2].[ArrTerm]=[t2].[Term1] AND
[f3].[depTerm]=[t2].[Term2]))) ORDERED FORWARD)
and for the GOLD box its like:
Execution Tree
--
Table Insert(OBJECT:(@.f3), SET:(@.f3.[outbound]=RaiseIfNull(1),
@.f3.[groundDur_2]=[t2].[TransitTime],
@.f3.[groundDur_1]=[t1].[TransitTime], @.f3.[duration_3]=[f3].[Duration],
@.f3.[duration_2]=[f2].[Duration], @.f3.[duration_1]=[f1].[Duration],
@.f3.[FlightDate_3]=[f3].[FlightDate],
@.f3.[f1_FlightID]=[f2].[FlightID], @.f3.[flightset_3]=[f3].[flightset],
@.f3.[FlightID_3]=[f3].[FlightID],
@.f3.[fromairport_3]=[f3].[fromairport],
@.f3.[toairport_3]=[f3].[toairport], @.f3.[eta_3]=[f3].[eta],
@.f3.[std_3]=[f3].[std], @.f3.[FlightDate_2]=RaiseIfNull([@.out]),
@.f3.[f0_ID]=[f1].[ID], @.f3.[f0_FlightID]=[f1].[FlightID],
@.f3.[flightset_2]=[f2].[FlightSet], @.f3.[FlightID_2]=[f2].[FlightID],
@.f3.[fromairport_2]=[f2].[FromAirport],
@.f3.[toairport_2]=[f2].[ToAirport], @.f3.[eta_2]=[f2].[ETA],
@.f3.[std_2]=[f2].[STD], @.f3.[flightdate_1]=[f1].[FlightDate],
@.f3.[flightset_1]=[f1].[flightset], @.f3.[FlightID_1]=[f1].[FlightID],
@.f3.[fromairport_1]=[f1].[fromairport],
@.f3.[toairport_1]=[f1].[toairport], @.f3.[eta_1]=[f1].[eta],
@.f3.[std_1]=[f1].[std], @.f3.[ID]=RaiseIfNull([Expr1011]),
@.f3.[FlightNumber_1]=[f1].[FlightNumber],
@.f3.[AirlineCode_1]=[f1].[AirlineCode],
@.f3.[FlightNumber_2]=[f2].[FlightNumber],
@.f3.[AirlineCode_2]=[f2].[AirlineCode],
@.f3.[FlightNumber_3]=[f3].[FlightNumber],
@.f3.[AirlineCode_3]=[f3].[AirlineCode]))
|--Top(ROWCOUNT est 0)
|--Compute Scalar(DEFINE:([Expr1011]=getidentity(1329804195, 2,
'@.f3')))
|--Filter(WHERE:([f3].[std]>[f2].[ETA]+Convert([@.transitDuration])+isnull([t2].[TransitTime],
0)))
|--Nested Loops(Left Outer Join, OUTER REFERENCES:([f2].[ArrTerm],
[f2].[ToAirport], [f3].[depTerm], [f3].[fromairport]))
|--Filter(WHERE:([f2].[STD]>[f1].[eta]+Convert([@.transitDuration])+isnull([t1].[TransitTime],
0)))
| |--Nested Loops(Left Outer Join, OUTER REFERENCES:([f1].[arrTerm],
[f1].[toairport], [f2].[DepTerm], [f2].[FromAirport]))
| |--Nested Loops(Left Anti Semi Join,
WHERE:(@.arrivalList.[AirportCode]=NULL OR
[f2].[ToAirport]=@.arrivalList.[AirportCode]))
| | |--Nested Loops(Left Anti Semi Join,
WHERE:(@.arrivalList.[AirportCode]=NULL OR
[f1].[toairport]=@.arrivalList.[AirportCode]))
| | | |--Nested Loops(Inner Join,
WHERE:(((((([f1].[eta]-[f1].[std])*[f3].[dayafter]>0 AND
([f1].[eta]-[f1].[std])*[d].[dayafter]>0) OR ([f1].[eta]>[f1].[std] AND
([f2].[ETA]-[f2].[STD])*[f3].[dayafter]>0)) AND
Convert([f2].[Weekdays])&[d].[weekday]>0) AND
[f2].[ValidFrom]<=[d].[date]) AND [f2].[ValidTo]>=[d].[date]))
| | | | |--Bookmark Lookup(BOOKMARK:([Bmk1001]),
OBJECT:([dohop].[dbo].[Flight1] AS [f2]))
| | | | | |--Nested Loops(Inner Join, OUTER
REFERENCES:([f1].[toaptjoin], [f3].[fromaptjoin]))
| | | | | |--Nested Loops(Inner Join,
WHERE:([f1].[fromairport]<>[f3].[fromairport]))
| | | | | | |--Table Scan(OBJECT:(@.f1 AS
[f1]), WHERE:([f1].[outbound]=1))
| | | | | | |--Table Scan(OBJECT:(@.flast AS
[f3]), WHERE:([f3].[outbound]=1))
| | | | | |--Index
Seek(OBJECT:([dohop].[dbo].[Flight1].[IX3_Flight1] AS [f2]),
SEEK:([f2].[ToAptJoin]=[f3].[fromaptjoin] AND
[f2].[FromAptJoin]=[f1].[toaptjoin] AND [f2].[Status]=0) ORDERED
FORWARD)
| | | | |--Table Scan(OBJECT:(@.weekdays_out AS [d]))
| | | |--Table Scan(OBJECT:(@.arrivalList))
| | |--Table Scan(OBJECT:(@.arrivalList))
| |--Clustered Index
Seek(OBJECT:([dohop].[dbo].[Transit].[PK_Transit] AS [t1]),
SEEK:([t1].[Apt1]=[f1].[toairport]),
WHERE:([t1].[Apt2]=[f2].[FromAirport] AND (([t1].[Term1]=' ' AND
[t1].[Term2]=' ') OR ([f1].[arrTerm]=[t1].[Term1] AND
[f2].[DepTerm]=[t1].[Term2]))) ORDERED FORWARD)
|--Clustered Index Seek(OBJECT:([dohop].[dbo].[Transit].[PK_Transit] AS
[t2]), SEEK:([t2].[Apt1]=[f2].[ToAirport]),
WHERE:([t2].[Apt2]=[f3].[fromairport] AND (([t2].[Term1]=' ' AND
[t2].[Term2]=' ') OR ([f2].[ArrTerm]=[t2].[Term1] AND
[f3].[depTerm]=[t2].[Term2]))) ORDERED FORWARD)|||after reading this
http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=58294&whichpage=1
thread I think I will move to SP3.

Friday, February 24, 2012

Failure to insert multiple rows into SQL database

I have an app that imports data from a csv file into a dataset. The user views the dataset and then decides to import the data into the database by clicking the code below.

I am getting the error message below when the app gets to the line 'objCommand.ExecuteNonQuery()'

"Message="The variable name '@.PartNumber' has already been declared. Variable names must be unique within a query batch or stored procedure.
Incorrect syntax near '?'."

My questions are:

1) Why is the code failing at this point ? ( i have hard coded the part number and part name to test my code)

2) How do i pass the actual value of PartNumber and PartName from each datarow into the parameter ?

Private Sub btnImport_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles btnImport.Click

Dim objCommand As SqlCommand = New SqlCommand

Dim datatable As DataTable

DataTable = dsimport.Tables(0)

Dim dataRow As DataRow

'Setup SQLcommand

objCommand.Connection = objConnection

objCommand.CommandText = "INSERT into Part (PartNumber,PartName) VALUES (?,?)"

objCommand.CommandType = CommandType.Text

For Each dataRow In dsimport.Tables(0).Rows

'Parameter for the PartNumber field...

objCommand.Parameters.AddWithValue("PartNumber", "2R8T-14A005-AA")

'Parameter for the PartName field...

objCommand.Parameters.AddWithValue("PartName", "Test3")

'Open the connection...

objConnection.Open()

'Execute the SqlCommand object to update the data...

objCommand.ExecuteNonQuery()

'Close the connection...

objConnection.Close()

objCommand = Nothing

objConnection = Nothing

End Sub

This article talks about where you can use (?) placeholder.

http://authors.aspalliance.com/aspxtreme/adonet/usingstoredprocedureswithcommand.aspx

In your case you can use OleDbCommand to accomplish the same thing. I have an example below in which I read couple of rows from an excel spreadsheet and insert them into SQL Database.

Dim cn As New OleDbConnection, cn1 As New OleDbConnection
Dim adapter As New OleDbDataAdapter
Dim dtset As New DataSet
Dim cmd As New OleDbCommand
Dim dr As DataRow

cn.ConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;" + "Data source= C:\Test.xls;" + "Extended Properties=""Excel 8.0;HDR=Yes;"""
cn1.ConnectionString = "Provider=SQLOLEDB;Data Source=.;Integrated Security=SSPI;"

cn.Open()
cn1.Open()

cmd.Connection = cn
cmd.CommandText = "Select * from TestTable"
adapter.SelectCommand = cmd
adapter.Fill(dtset)

cmd.Connection = cn1
cmd.CommandText = "Insert into Parts (PartNumber,PartName) Values(?,?)"
cmd.CommandType = CommandType.Text

cmd.Parameters.Add("PartNumber", OleDbType.VarChar, 20)
cmd.Parameters.Add("PartName", OleDbType.VarChar, 20)
For Each dr In dtset.Tables(0).Rows
cmd.Parameters("PartNumber").Value = dr(0).ToString()
cmd.Parameters("PartName").Value = dr(1).ToString()
cmd.ExecuteNonQuery()
Next

cn.Close()
cn1.Close()

Hope this helps

|||

Using SqlCommand you can use this example:

Dim cn As New OleDbConnection
Dim sqlcn As New SqlConnection
Dim adapter As New OleDbDataAdapter
Dim dtset As New DataSet
Dim cmd As New OleDbCommand
Dim sqlcmd As New SqlCommand
Dim dr As DataRow

cn.ConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;" + "Data source= C:\Test.xls;" + "Extended Properties=""Excel 8.0;HDR=Yes;"""
sqlcn.ConnectionString = "Data Source=.;Integrated Security=SSPI;"

cn.Open()
sqlcn.Open()

cmd.Connection = cn
cmd.CommandText = "Select * from TestTable"
adapter.SelectCommand = cmd
adapter.Fill(dtset)

sqlcmd.Connection = sqlcn
sqlcmd.CommandText = "Insert into Parts (PartNumber,PartName) Values(@.a,@.b)"
sqlcmd.CommandType = CommandType.Text

sqlcmd.Parameters.Add("@.a", SqlDbType.VarChar, 20)
sqlcmd.Parameters.Add("@.b", SqlDbType.VarChar, 20)

For Each dr In dtset.Tables(0).Rows
sqlcmd.Parameters("@.a").Value = dr(0).ToString()
sqlcmd.Parameters("@.b").Value = dr(1).ToString()
sqlcmd.ExecuteNonQuery()
Next

cn.Close()
sqlcn.Close()

Hope this helps

Thursday, March 29, 2012

field mapping

Wednesday, March 7, 2012

faster box but slower database

faster box but slower database

faster box but slower database

Friday, February 24, 2012

Failure to insert multiple rows into SQL database

Footer rows to skip

Blog Archive

About Me