Bill and Paul’s session gets off to a good
start with a mexican wave from the top of the room to the end.
On with the show with the 12 worst practices
they could find.
Saving Disk Space : single server site,
started running low on diskspace. hadn;t compacted for 30 days.Tried loading
compact and fixup but ‘File Not Found’.
Admin tried to free up diskspace and
saw notes directory on c drive. deleted it. Server stayed up for
30 days. Fixed by reinstalling domino over what was left after the deletion.
Notes data directory was on the D drive.
the admin didn;t know some files were in use on the c drive.
ECL Hell : all users locked out of EVERYTHING.
no dialogs or warnings. Investigation showed that ecl was 100% locked down.
A junior admin back from training removed everything and added himself.
Resolution : shoot the admin TWICE.
remove policy restrictions and reconstruct the ECL. Lessons, a little knowledge
is dangerous.
The magical mailfile. small site, 5
server 200 users. every week the directors mailfile was missing. checked
adminp and logs. nothing. Looked at AV logs, and it was set to scan
the notesdata directory and found a false positive in the directors mailfile
and deleted it. AV should not run on notesdata directory as it effcts
server performance. Have let a file system AV san nsf’s
Now you see it… Mission critical
database, all data disappears.talked to dev, he ran an agent. corrupted
all readers fields, it replicated. After shootin the developer use full
access admin to override the reader fields or restore from backup. Never
run test code in production environ.
Son of email from hell. large domino
environ. One server grinds to a halt, spreads to other server. Checked
the console and see a ‘dispatching messages’ on the router info. nrouter
is using up 99% cpu. Opened mail.box515593223 bytes. sent to 428 people
: 220 Gb total.. after deleting the mail from mail.box the cause was a
health and safety officer mailing a large mpeg video to all staff. Then
of course shoot the user. Limit attachment sizes
The DeCrappy Code. Large govt organisation
used web based form to collect info. It took 20 minutes to load first webpage.
cpu running at 100% and used agent to construct xml via xls and passed
to javascript to display on screen. Investigation showed the agent constructed
a 4Mb datastream for each page.. Rewrote code to use a view and then mounted
dev’s head on a wall as a warning to all other dev’s.. Yes, you can use
sexy new web2.0 stuff but still write crap applications.
Identity Crisis. Large environment,
hub spoke setup. Major hub stops working. Cannot see anything else on network.
Admin said he hadn’t changed anything. Hahahaha. Server logs show that
the server had no access. admin had wanted to access database on server.
Admin created user with same name as hub. Then afterwards deleted
the user account. Adminp kicked in and removed ALL entries from groups,
reader fields etc. Again shoot the admin.
Missing you already. An exec gets email
and phone calles from all people he communicated with for past three years.
OOO had been setup with incorrect dates on a blackberry. It miscommunicated
with Notes caused the OOO agent to send loats of OOO messages.
Security For Beginners. Large financial
org. lots of usage, new admin, seasoned dev.. Put laptop on network, was
handed an IP address by DHCP. outside users ID file could open the domino
server. Admin had set the server document to allow anonymous notes
connections. OOps.Paul just described himself as a small server and Bill
as a large partitioned server. Dev was sick of switching ID’s and had asked
new admin to change the settings in the nab.
Agent Ageism. Govt agency sends letter
to people over a certain age. Doing a hardware refresh. Customer
said they had backups. something went wrong. Asked for backups but they
were empty for the past 7 years. lgs had never been checked. ALWAYS check
the tapes before doing work, even on as a consultant. It’s a lesson
you never forget.
Oh is that important. Over 90 servers,
65k users around the world. One friday all replication stops.checked logs,
consoles etc. The LocalDomainServer group had been deleted. Adminp removed
it from all fields and groups and ACL’s.. As the deletion replicated out
to all server all servers stopped. It’s not an easy fix. all servers
to be shut down. admminp needs to be flused, backups of nab to be restored
loads of work, in this case took 3 days
The Leak. Political part used a domino
based site to allow people to donate money to the party, They were told
there was a security leak, so a pen test was performed. default views
and forms had not been secured, no reader fields. Resolved by shooting
dev and test person. Take it offline and inform anybody who’s data may
have been seen. Use the domino security model to it’s fullest, especially
with dealing with person data on the web.
Dec's Dom Blog 