No matter how much I stress it, I cannot overemphasize the importance of this tip. In fact, it is not even tip; it is well documented in Oracle RAC manuals. However, if you are using RAC, chances are that you have not read the Oracle RAC documentation. The non-RAC documentation covers this point in a rather sketchy fashion, in bits and pieces. If you yawn as often as I do while reading manuals, you will certainly miss it. And none of the documentation emphasizes and illustrates this fact adequately. If I had any say in the matter, this piece of advice would be printed in huge 36-point bold type. It is a wicked show-stopper.
A while ago, total lack of knowledge in this area made me stay up for more than 56 hours straight in order to finish an Oracle installation for a client. Since the client was running in 24*7 mode, we had scheduled a planned outage to migrate to a newer version of Oracle. We also planned to migrate simultaneously to a different more powerful system. And this time we decided to go raw partition. We were expected to finish most of the upgrade tasks, such as creating the raw devices and setting up the new database on the new machine prior to the data migration, during the outage, which was supposed to be relatively short.
Well, the actual outage was hort. But the amount of pre-work stretched into continuous day and nights. It turned out that the system administrator had inadvertently created all of the raw partitions on cylinder 0 of each disk. Having done that, he handed everything over to us and asked us to do our DBA tasks. I went ahead and created database. We tried some practice loads and everything went smoothly. Then we bounced the system. But when we came back up, the disks were completely useless. All of the data we had loaded was gone. After few more iterations, we finally uncovered the elusive problem; creating the raw disk on cylinder 0 had overwritten valuable disk partition information.
Cylinder 0 is used to maintain key disk specific information, such as partition and logical volume control table maps. When you are using a file system, cylinder 0 is automatically protected by Unix or Linux operating system. However, when you use raw devices, Unix leave the application totally in charge. As such, you have to manually ensure that you do not create raw partitions on cylinder 0. Newer version of Unix are said to prevent this problem.