After the problem cited below, there is another pesty little problem to resolve. If you're reading this before it is visible on this blog, it means you have subscribed to the status blog and you've got this in your digest. Lucky you.
The blog will also be promoted to the latest updated list on the home page, but if you click to read it... lo and behold -- you will only see the next to last post.
Your new posts are/were in limbo -- visible in the edit view and in the digest, but not on the public view of your blog. Once again, sorry 'bout that. It should be fixed rather soon.
Sorry sorry sorry. (Scusateci!)
Last night, at around 7:00 p.m. here in the boot-shaped country (GMT +1), Mo'time went down down down.
We had a major crash -- a kernel panic (gotta love the name). The server froze, some files were corrupted, we did some reconstruction and restoring and you are reading this as a result of a successful first aid operation.
Please accept my humble apologies.
I hope that most of you were at the beach, lazing off in bed, busy at work, having coffee or cocktails while Mo'time was down for many hours today. For those who were trying to blog --- it did not work out great for you.
The Apache servers went down down down earlier and were simply unresponsive in the restart department. After a lot of late summer sweat, things are back and running.
Please accept our humble apologies for the downtime.
Love note from our service provider. We still have some essential services running in Canada.
"In order to complete our move down to the Terminal Level of the CN Tower, we are scheduling a power outage which will affect you.
Date: Saturday, January 27, 2007
Maintenance window: 02:00 - 05:00 MDT (09:00 - 12:00 UTC/GMT)
Expected duration: 3 hours
During this time, your service will be unavailable. If you have any questions regarding this, please feel free to email us at support@tera-byte.com. We appreciate your patience during this transition period and thank you for your continued support.
Yours Sincerely,
Steve Keyser
CEO, Tera-byte Dot Com Inc."
Patience is a virtue, they say. But I wouldn't know about that. :(
**POSTPONED** to Thursday, the 21st.
We're conducting the final part of the move from our old server farm, which was housed by another company, to our very own servers!
Today, on Saturday, December 16, 2006, from 10:00 p.m. to midnight GMT/UTC (22:00-00:00), we will be moving the file storage system for photos, videos, audio files and any other stuff that you can upload to your archives to the new servers.
As a result, we will have to shutdown the upload function during that time. Everything else should function well!
The other service that will be down is your "syndication" -- the RSS and Atom feeds will not be updated during this time frame either. In order for them to snap back into place, you will need to write a new post. That's all.
• What time is it now in GMT/UTC?
• Use this time converter to calculate the times above for your time zone.
Read the post below! And go get some coffee or something...
The main router at tera-byte, the company that houses some key pieces of our server farm, will be physically moved on Saturday, December 9, 2006 at 09:00-10:00 GMT/UTC.
Blogs will be visible, but not available for posting or commenting and www.motime.com will simply be invisible -- and no courtesy page will appear, not because we are no longer courteous, but because the time required to set this up will exceed the downtime! :)
• What time is it now in GMT?
• Use this time converter to calculate the times above for your time zone.
After putting a complex system back online, there may be some bugs, glitches or malfunctions. The only one that I am aware of right now is that the number of members displayed on the online users page is not correct. It seems to only show the people who are online with the instant messenger, and not the people simply logged in to Mo'time. Let me know if you find other problems and we'll take care of them one by one as always. Thanks for understanding that these things usually take a little time.
We will be physically moving the server farm from our historical plot of cyberspace in the Rocky mountains to a new undisclosed location. :-)
During that time, your blogs will be down for a few hours beginning at about 18:30 GMT. I'm estimating about 4 hours of downtime:
• What time is it now in GMT?
• Use this time converter to calculate the times above for your time zone
We seem to be back at full speed. Let me know if you experience bugs, buglets or monster bugs!
We will be improving and upgrading the electrical power system that powers our server farm in order to offer even more reliable service. This will require us to be offline on Saturday, October 14th from 7:00 a.m. to about Noon GMT. All Pro accounts will be extended by 2 days of free service to make up for this downtime.
During that time, your blogs will not be visible, you will not be able to post and Mo'time will be offline. Get out and take a walk! ;-)
Since we have bloggers from all around the world, here are a few links to help you figure out when the downtime will happen in your time zone:
• What time is it now in GMT?
• Use this time converter to calculate the times above for your time zone
I was just informed that Recent Comments (in most of your side columns in the more recent templates) have been turned off because they seemed to be the cause of a mysteriously high load on the database. When that happens, there is the risk of downtime, so we are investigating what happened.
I am hoping that they will back online tomorrow. As usual, apologies all around for the problem...
The multimedia files are still giving us trouble tonight. We're working on it but have determined that the problem is actually rather serious -- due (again) to a faulty power supply (UPS) issue that the company that houses our servers can't seem to get right. It is frustrating because it should be on of the simpler pieces of hardware to maintain. We are mad and working to fix something that should never have broken. *clink* *hicc*
It probably will not be fixed until tomorrow afternoon, the earliest. I am really very sorry about this. There's nothing more I can say, so back to work. :-(
One of the servers down on the farm is having problems doing its job and shuts down whenever it feels like it. The result is that the multimedia files do not show up as they should. We hope to have this solved during the course of the day...
There is a momentary problem with the multimedia file storage system, and so those files (photos, avatars and such) are currently invisible. We are working to bring them back from "the other side".
Recipe: elbow grease, patience, magic dust and (of course) spumante.
It seems that sometimes comments are disappearing, reappearing and disappearing again. Don't worry about them being gone forever -- they have been saved but there is some little (and hard to find) bug that sends them to a dark and distant land. We are sending in the helicopters to bring them back.
OK, well, that was no fun.
But we are back to normal service now. Please let me know if you notice any post-disaster bugs.
Thanks again for your patience and understanding (directed to those who were patient and understanding). ;-)
At 10:00 a.m. (GMT) we will have to go offline for about one (1) hour to complete the emergency repairs! Mo'time is almost back to full strength! (**hic**hic** :-P )
We have moved many of the blogging functions to the one server that seems to have survived the troubles. it appears that blogging and commenting are back. We are still working on bringing back photos, videos, avatars and a series of other services back to normal.
What happened? We are not 100% sure but it appears to have been a faulty "Uninterruptible Power Supply", aka UPS, down on the farm. This is the system designed to prevent power surges and outtages from damaging the servers. As a result, ever since Sunday we have been experiencing a series of servers going offline intermittently and coming back online damaged. Since we have redundant servers, the service kept on ticking through it all until it collapsed yesterday. :-(
Keep tuned!
Update: Certain features like photos, videos, audios and the IM/chat will not be available for about 24 hours, maybe less (not more!)...
I would not be surprised if there is other strange behavior as well. We need to physically replace some hardware that appears to be damaged and that is what will take some time. Believe me, we hate when this happens even more than you do! :-(
today
December 2007
August 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
November 2005
October 2005
September 2005
August 2005
July 2005
May 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004
September 2004
August 2004
July 2004
June 2004
May 2004
April 2004
March 2004
February 2004
January 2004
December 2003
November 2003
October 2003
September 2003
August 2003
July 2003