FYI IMPORTANT NOTICE -- SITE DOWNTIME MONDAY, 8-March

[xfadmin]

A note to let everyone know that the site will be off-line during the afternoon of Monday, March 8, for some significant behind-the-scenes site work.

We will be doing the following during this time:

1. Fixing the recurring Elasticsearch (search engine) failure issue once and for all -- re-installing and revamping the search capability and fixing the issue with the random search engine disconnection with the web server
2. Migrating the site's database from MySQL 5.7 to MySQL 8.0. This future-proofs the database for the next 5+ years, as version 8 of MySQL is the newest version and will be the standard for years to come.
3. Performing general site cleanup and maintenance on the back-end -- removing a lot of old and unused software, older versions of software that are no longer used, removing a log files that are old and no longer needed, and making some tweaks as far as retention of log files that are wasting storage space.

Getting all of this work done is a very high priority and is somewhat akin to a "C" service at 30,000 miles on the E500E -- everything is checked over and a lot of things are maintained. The site has been up for two years, so this is an "every couple of years" type of site maintenance.

Expect the site to be "down" from around noon EST on Monday, to around 6 PM EST. I expect that the actual "downtime" of the site will be much less than this, but I am giving a "maximum" window of downtime as a "worst case" scenario. It is almost 100% likely that the site will be up and visible during this time, and you will be able to see it running, but not be able to log in, except for the short (10-15 minute) period while the database is being upgraded. Otherwise you will likely see the "site closed ... do not wig out, go postal, etc." message displayed.

And a message to all of you -- DO NOT blow up @gerryvz's phone with text messages and e-mails breathlessly telling him that the site is down, you are planning to commit suicide because you can't access the forum, etc. I'm sure he will already know the site is down, and there really won't be anything he can do while it is down. So please don't bother him, or anyone else, with the fact that the site is down for up to 6 hours. If you feel the need to scratch your E500E forum itch during this time, there is a V-8 W124-specific forum over at Benzworld, located at the link below, that you can post to. Seeing as @gsxr pretty much runs it these days, you should be more or less right at home !!

Thanks in advance for your patience and understanding. These things happen from time to time.

— @xfadmin

 

[nocfn]

I will get my covid shot then!

 

[alabbasi]

I'll be hitting refresh every 3 seconds while it's down.

 

[gerryvz]

I'll be hitting refresh every 3 seconds while it's down.

That’s what 281LXM does. Is it a Texas thing?

 

[alabbasi]

That’s what 281LXM does. Is it a Texas thing?

OK he can refresh, I'll email you.

 

[xfadmin]

This has been postponed to the same time, but on March 15 (one week later).

Sorry for the false alarm.

 

[xfadmin]

All:

I am happy to report that everything was successful, and the forum came out of some major behind-the-scenes surgery yesterday (Tuesday) afternoon.

The migration of the forum's database to MySQL 8.0 (the current 8.0.23 version) was 100% successful, which was the primary goal of the work. Other things that were successfully accomplished:

1. Re-vamped the Elasticsearch search engine, along with a clean-up of the Java environment that it runs on top of. The search engine reliability should now be rock solid going forward, and you should not see errors from time to time, as you have in the past, when trying to use the search function.
2. The "typeahead" functionality for the search engine has been restored. This means when you type keywords into the search engine, it immediately suggests threads/topics in a drop-down box based on your input. The more letters/words you enter, the more accurate the search results become in the drop-down. This function was disabled since last October due to an issue with the add-on, that was just fixed by the developer.
3. The size of the Elasticsearch heap was set to 512MB. The auto sizing it normally uses was very likely causing issues with available memory, which then caused the search engine server to stop communicating with the forum's primary web server. The Elasticsearch index was also rebuilt.
4. Older, and un-necessary versions of the php language (starting with 5.6, which was originally installed two years ago), were removed. The forum has been running recently on php 7.4, which is the current production version.
5. Updated all of the Ubuntu system files and packages with the latest versions (something I pretty much do every day anyway).
6. Ran the rdfind command. This creates hard links between identical attachments, and saves a lot of storage space. Doing this rectifies multiple links that point to different versions of the same image that have been stored on the server. By reconciling and combining links, it allows files that have been "deleted" to actually now be deleted. Doing this reduced physical disk storage for attachments by 5GB, from 75GB down to 70GB.

So, this was more or less a "once every two years" house-cleaning effort to remove a lot of old stuff on the server, and clean and neaten things up, get all of the cob-webs out of the system, and bring all of the system software up to the current version. And we did it with basically ZERO forum downtime, except while the database was being upgraded.

One task remains to be done, which can now, easily, be accomplished. This is the upgrading of the Ubuntu system software from version 18.04.5, which is going to actively be supported through March 2023, to the current standard Ubuntu 20.04.2, which is supported through March 2025. I expect that this will require about 10-15 minutes of forum downtime to do, and I expect to do it on Sunday morning (March 21) about 7:00 AM EDT.

I believe that from a technical and foundational point of view, that the forum infrastructure has never been in a better condition, and all of the forum software running will be more stable than it has been over the past two years.

If you were impacted yesterday by any forum downtime, my apologies for this, and I'm hoping it didn't inconvenience you too much.

Thanks,
xfadmin