Showing posts with label storing. Show all posts
Showing posts with label storing. Show all posts

Wednesday, March 21, 2012

Feasability of storing PDFs and WORD.DOC files in MSSQL?

I'm building a system when one can upload a document to the website.

I will be storing the document on the hard-drive for quick/easy access,
but I was also thinking of storing it in an existing database since most
of the sites information is all stored there.

As well there would be only one place to worry about backing up. And if
the file on the hard-drive was ever missing or became corrupted, I could
restore it form tha database. Is this feasable? Has anyone ever done this?

--
* Don Vaillancourt
Director of Software Development
*
*WEB IMPACT INC.*
phone: 416-815-2000 ext. 245
fax: 416-815-2001
email: donv@.web-impact.com <mailto:donv@.webimpact.com>
web: http://www.web-impact.com

/ This email message is intended only for the addressee(s)
and contains information that may be confidential and/or
copyright. If you are not the intended recipient please
notify the sender by reply email and immediately delete
this email. Use, disclosure or reproduction of this email
by anyone other than the intended recipient(s) is strictly
prohibited. No representation is made that this email or
any attachments are free of viruses. Virus scanning is
recommended and is the responsibility of the recipient.
/
Don Vaillancourt wrote:

> I'm building a system when one can upload a document to the website.
> I will be storing the document on the hard-drive for quick/easy access,
> but I was also thinking of storing it in an existing database since most
> of the sites information is all stored there.
> As well there would be only one place to worry about backing up. And if
> the file on the hard-drive was ever missing or became corrupted, I could
> restore it form tha database. Is this feasable? Has anyone ever done
> this?

Sure it's possible to save any size binary object in the DBMS, but it is
not usually a performant use of a DBMS as a simple file store. I would
first consider a hardware solution for backups of read-only files. Either
just store copies on separate disks, or get a fancier RAID setup where
any one disk failure won't lose any data.
Joe Weinstein at BEA|||Don Vaillancourt (donv@.webimpact.com) writes:
> I'm building a system when one can upload a document to the website.
> I will be storing the document on the hard-drive for quick/easy access,
> but I was also thinking of storing it in an existing database since most
> of the sites information is all stored there.
> As well there would be only one place to worry about backing up. And if
> the file on the hard-drive was ever missing or became corrupted, I could
> restore it form tha database. Is this feasable? Has anyone ever done
> this?

I have not done this myself, but certainly it is a common scenario.

Storing the documents on disk makes for a simple implementation.
Storing them in the database requires a battle with the text/image
data types, which are somewhat difficult to use.

But as you note, storing in the database is safer. Backup is simpler,
and a file on directory can easily "disappear". And if all you store
is the file path, another problem is that you don't have two-phase
commit with the file system.

--
Erland Sommarskog, SQL Server MVP, esquel@.sommarskog.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techin.../2000/books.asp|||I know that technically I can store this info in the database and
retrieving it is not an issue.

My issue really is how manageable will the database be with tons of
PDFs, Word, etc. documents in it. Or how willing people would be
willing to manage such a beast.

Right now our databases are about 300MB each. I may decide to create a
second database, not sure, but the total number of documents may be at
the very least in the 1000's which would most probably create a database
about 5GB + overhead.

To some this may be an intimidating size to manage. It would certainly
fit on a tape backup.

I'm just wondering about other issues that may come up.

Erland Sommarskog wrote:
> Don Vaillancourt (donv@.webimpact.com) writes:
>>I'm building a system when one can upload a document to the website.
>>
>>I will be storing the document on the hard-drive for quick/easy access,
>>but I was also thinking of storing it in an existing database since most
>>of the sites information is all stored there.
>>
>>As well there would be only one place to worry about backing up. And if
>>the file on the hard-drive was ever missing or became corrupted, I could
>>restore it form tha database. Is this feasable? Has anyone ever done
>>this?
>
> I have not done this myself, but certainly it is a common scenario.
> Storing the documents on disk makes for a simple implementation.
> Storing them in the database requires a battle with the text/image
> data types, which are somewhat difficult to use.
> But as you note, storing in the database is safer. Backup is simpler,
> and a file on directory can easily "disappear". And if all you store
> is the file path, another problem is that you don't have two-phase
> commit with the file system.
>

--
* Don Vaillancourt
Director of Software Development
*
*WEB IMPACT INC.*
phone: 416-815-2000 ext. 245
fax: 416-815-2001
email: donv@.web-impact.com <mailto:donv@.webimpact.com>
web: http://www.web-impact.com

/ This email message is intended only for the addressee(s)
and contains information that may be confidential and/or
copyright. If you are not the intended recipient please
notify the sender by reply email and immediately delete
this email. Use, disclosure or reproduction of this email
by anyone other than the intended recipient(s) is strictly
prohibited. No representation is made that this email or
any attachments are free of viruses. Virus scanning is
recommended and is the responsibility of the recipient.
/|||Don Vaillancourt (donv@.webimpact.com) writes:
> I know that technically I can store this info in the database and
> retrieving it is not an issue.
> My issue really is how manageable will the database be with tons of
> PDFs, Word, etc. documents in it. Or how willing people would be
> willing to manage such a beast.
> Right now our databases are about 300MB each. I may decide to create a
> second database, not sure, but the total number of documents may be at
> the very least in the 1000's which would most probably create a database
> about 5GB + overhead.
> To some this may be an intimidating size to manage. It would certainly
> fit on a tape backup.

Few of our customers that run our system have databases that small. My test
database alone is 600 MB.

And 5 GB in one database is easier to manage, than 300 MB in a database + a
lot of loose files.

However, if you are using MSDE the size could be a concern, since MSDE
has a size limit of 2 GB.

--
Erland Sommarskog, SQL Server MVP, esquel@.sommarskog.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techin.../2000/books.asp|||No MSDE for us. We tell how clients which version of whatever database
they must purchase in order to use our software.

Otherwise, from your experiences, my issue is really a non-issue.

Thanks for your input.

Erland Sommarskog wrote:
> Don Vaillancourt (donv@.webimpact.com) writes:
>>I know that technically I can store this info in the database and
>>retrieving it is not an issue.
>>
>>My issue really is how manageable will the database be with tons of
>>PDFs, Word, etc. documents in it. Or how willing people would be
>>willing to manage such a beast.
>>
>>Right now our databases are about 300MB each. I may decide to create a
>>second database, not sure, but the total number of documents may be at
>>the very least in the 1000's which would most probably create a database
>> about 5GB + overhead.
>>
>>To some this may be an intimidating size to manage. It would certainly
>>fit on a tape backup.
>
> Few of our customers that run our system have databases that small. My test
> database alone is 600 MB.
> And 5 GB in one database is easier to manage, than 300 MB in a database + a
> lot of loose files.
> However, if you are using MSDE the size could be a concern, since MSDE
> has a size limit of 2 GB.

--
* Don Vaillancourt
Director of Software Development
*
*WEB IMPACT INC.*
phone: 416-815-2000 ext. 245
fax: 416-815-2001
email: donv@.web-impact.com <mailto:donv@.webimpact.com>
web: http://www.web-impact.com

/ This email message is intended only for the addressee(s)
and contains information that may be confidential and/or
copyright. If you are not the intended recipient please
notify the sender by reply email and immediately delete
this email. Use, disclosure or reproduction of this email
by anyone other than the intended recipient(s) is strictly
prohibited. No representation is made that this email or
any attachments are free of viruses. Virus scanning is
recommended and is the responsibility of the recipient.
/

Monday, March 12, 2012

Fastest way to delete to huge datarow?

Hi all,
I've got a table which is storing Images with a size up to 1GB. When
I'm going to delete one of this rows the DELETE Statement takes up to 5
minutes. Which is the best way to delete the row more faster?
Has anyone an idea? Thanks in advance!
Gerrit Horeis
Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.deOn 04.12.2006 15:29, Gerrit Horeis wrote:

> I've got a table which is storing Images with a size up to 1GB. When
> I'm going to delete one of this rows the DELETE Statement takes up to 5
> minutes. Which is the best way to delete the row more faster?
> Has anyone an idea? Thanks in advance!
Did you measure what is slow there? Do you have an index on that table?
Is it used for deletion? ... With the little information you disclose
it's hard to come up with any reasonable advice.
Regards
robert|||Hi
Thanks for your answer. In the table im storing several image formats.
The delete Statement is
"Delete from Formats where ID = @.FormatID"
"ID" is the primary key of the table. Do I have to create an index as
well? I ever thought Sql just removes the reference to the data and so
it must be very fast.
Please help!
Robert Klemme schrieb:

> On 04.12.2006 15:29, Gerrit Horeis wrote:
>
> Did you measure what is slow there? Do you have an index on that table?
> Is it used for deletion? ... With the little information you disclose
> it's hard to come up with any reasonable advice.
> Regards
> robert|||On 05.12.2006 14:22, Gerrit Horeis wrote:
> Thanks for your answer. In the table im storing several image formats.
> The delete Statement is
> "Delete from Formats where ID = @.FormatID"
> "ID" is the primary key of the table. Do I have to create an index as
> well? I ever thought Sql just removes the reference to the data and so
> it must be very fast.
It seems you got the basics right. But storage of BLOBS is usually more
complex and less efficient than, say, VARCHAR columns. I guess you have
to run the Profiler to find out why it is actually slow.
Kind regards
robert

Fastest way to delete to huge datarow?

Hi all,
I've got a table which is storing Images with a size up to 1GB. When
I'm going to delete one of this rows the DELETE Statement takes up to 5
minutes. Which is the best way to delete the row more faster?
Has anyone an idea? Thanks in advance!
Gerrit Horeis
Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.de
Hi
Thanks for your answer. In the table im storing several image formats.
The delete Statement is
"Delete from Formats where ID = @.FormatID"
"ID" is the primary key of the table. Do I have to create an index as
well? I ever thought Sql just removes the reference to the data and so
it must be very fast.
Please help!
Robert Klemme schrieb:

> On 04.12.2006 15:29, Gerrit Horeis wrote:
>
> Did you measure what is slow there? Do you have an index on that table?
> Is it used for deletion? ... With the little information you disclose
> it's hard to come up with any reasonable advice.
> Regards
> robert

Fastest way to delete to huge datarow?

Hi all,
I've got a table which is storing Images with a size up to 1GB. When
I'm going to delete one of this rows the DELETE Statement takes up to 5
minutes. Which is the best way to delete the row more faster?
Has anyone an idea? Thanks in advance!
--
Gerrit Horeis
Software Developer
CI-Gate Development & Consulting GmbH
http://www.ci-gate.de
http://www.xira.de
http://www.bitbauer.deOn 04.12.2006 15:29, Gerrit Horeis wrote:
> I've got a table which is storing Images with a size up to 1GB. When
> I'm going to delete one of this rows the DELETE Statement takes up to 5
> minutes. Which is the best way to delete the row more faster?
> Has anyone an idea? Thanks in advance!
Did you measure what is slow there? Do you have an index on that table?
Is it used for deletion? ... With the little information you disclose
it's hard to come up with any reasonable advice.
Regards
robert|||Hi
Thanks for your answer. In the table im storing several image formats.
The delete Statement is
"Delete from Formats where ID = @.FormatID"
"ID" is the primary key of the table. Do I have to create an index as
well? I ever thought Sql just removes the reference to the data and so
it must be very fast.
Please help!
Robert Klemme schrieb:
> On 04.12.2006 15:29, Gerrit Horeis wrote:
> > I've got a table which is storing Images with a size up to 1GB. When
> > I'm going to delete one of this rows the DELETE Statement takes up to 5
> >
> > minutes. Which is the best way to delete the row more faster?
> >
> > Has anyone an idea? Thanks in advance!
> Did you measure what is slow there? Do you have an index on that table?
> Is it used for deletion? ... With the little information you disclose
> it's hard to come up with any reasonable advice.
> Regards
> robert|||On 05.12.2006 14:22, Gerrit Horeis wrote:
> Thanks for your answer. In the table im storing several image formats.
> The delete Statement is
> "Delete from Formats where ID = @.FormatID"
> "ID" is the primary key of the table. Do I have to create an index as
> well? I ever thought Sql just removes the reference to the data and so
> it must be very fast.
It seems you got the basics right. But storage of BLOBS is usually more
complex and less efficient than, say, VARCHAR columns. I guess you have
to run the Profiler to find out why it is actually slow.
Kind regards
robert