A Valuable Lesson Learned

I really need to start actively blogging
again,, I’ve been doing so many other things it kinda took a back seat.
To start the process I’m going to admit to something that probably belongs
alongside all the stories from Bill and Pauls Worst Practices session at
Lotusphere…

One of the new features in BlogSphere
V3 Beta 9 was the integration of OpenLog to catch any errors that might
occur when the rendering agents ran. I also included the ability to output
different levels of debugging information so if anybody contacted me with
a problem I could get them to turn on the debug and they would be able
to send me logs that would help me trace the problem quickly. There are
a couple of different levels of debugging ranging from minimal to insane.
When set to ‘insane’ every stage of the agent produces a new debug document
in the database so just viewing your blogs homepage could result in about
200 new debugging documents.

While testing it out I turned on debugging
to the ‘insane’ level on this blog and then promptly forgot about it. I
went on vacation to Disney World for a week and came back to find my server
starting to run very low on disk space. In over just one week I had managed
to add over 7 million debug documents to my blog database increasing it’s
size to just over 26 Gb.

The Domino server was still ok and serving
pages from my blog but obviously I had to delete all these documents from
the database. I had a number of options, I could go into the view where
they were all stored and try select them all and delete them manually or
I could write an agent to delete all the documents.

Have you ever tried a ‘Select All’ on
over 7 million documents, it was still trying to select them after one
day. I had to drop my Domino server just to get the notes client to timeout
as even ctrl-break wasn’t working on the client.

So I decided to try the agent route,
it was working very slowly and would delete about 75,000 documents an hour.
My guess is that it had to keep recalculating the view’s index and it would
have taken about 4 days to get rid of the documents and I’d still have
to deal with the deletion stubs taking up space.

In the end I copied the .nsf from the
server to my client and then deleted the file from the server. I then created
a new replica of the database back onto the server and wrote in a replication
formula to exclude the debug document. It only took about 18 hours to complete
with most of the time taken up by the Notes client reading in the local
database to figure out what to replicate and then about 40 minutes for
it to actually create the new replica.

In the end I ended up with a new replica
of my blog minus the 7 million debug documents and no deletion stubs to
worry about.

And the lesson… ‘Insane’ means insane,
if you do turn on debugging in Blogsphere for any reason make sure you
remember to turn it off when finished and delete the debug documents.