Site Killing Spam Bot

mail-311519_1280You might have noticed, although there aren’t too many visitors to ChaosManor Reviews (CMR), some problems accessing the site this month (Sept 2016). The site would not load at all, usually due to a timeout of the request to load the desired page. Then yesterday (23 Sep 2016), the site was not available at all. And neither was the Chaos Manor (CM) site.

That prompted a concerned email from Jerry to me about site access problems. So I hobbled over to my computer (dealing with a sciatica problem) and started troubleshooting.

The cause of both sites not being available was the hosting company (BlueHost – BH) suspending the hosting account due to the load on the shared servers. (It is common for web sites to ‘share’ a single server and its resources; that’s how a hosting place keeps the hosting costs low.)

Calling Support

I got onto the BlueHost support line via a chat (after a long delay; they were a bit overloaded, so help chats responses were slow). I got the account un-suspended, and both the CM and CMR sites were again available.

Some background on how the hosting is set up here. Jerry uses BH as his hosting place. They are a very large hosting place, probably in the top 5, I’d guess. I’ve used their hosting (and their sister’s JustHost hosting service) for many years for my personal web sites and those of clients. The up time is good, and support is good also.

A hosting account can support more than one domain. And, to keep costs down, hosting companies usually put more than one hosting account (with its associated domains) on a single server. So multiple domains – and multiple hosting accounts – are using the resources of a single computer.

This usually works well, especially for smaller domains – those that have small numbers of daily visitors. With small loading on a server by each domain on that server, the process works well.

Until one domain starts hogging the resources of a shared server. That results in the other domains on that server to start slowing down. It’s like a big truck (a domain with lots of concurrent visitors) on a one-lane road going up a hill. The truck is loaded down with visitors, and slows down all the other vehicles (other domains on that shared server). The guys in charge of the highway (the hosting place) don’t want the big truck to take up all the resources.

So when a domain starts taking up a higher percentage of user ‘load’, the hosting place has a few options. They can increase the resources of that server, or they can move the high-load domain to a different server – maybe one with less domains sharing, or more powerful hardware.

But that is not cost-efficient for the hosting place. If a domain needs higher capabilities, then the customer (the owner of the domain) should pay more for a better/faster/less-shared server.

The other alternative that the hosting place has is to suspend the high-load domain, which is what BH did to all of Jerry’s domains. That reduces the load on the shared server, and the other domains on that server get their normal resources.

BlueHost will send an email to the domain owner telling them that their accounts (and sites) are suspended. Which they did – Jerry got the email alert. The problem with that procedure is that there is no ‘warning’ of the impending ‘doom’ of domains going off-line. It’s like the traffic cop always giving you a ticket – no warnings.

Jerry gets lots of email, and he happened to notice that email from BH about his sites being suspended (off-line). He forwarded it to me (I do all the technical stuff for his sites), and I started working on the problem via an on-line chat.

BH wouldn’t (couldn’t) tell me what domain was causing the overload problem. At least during the first chat. All they would say is that a domain on Jerry’s account was overloading the shared server’s resources. And that the best solution they could come up with was to upgrade the hosting account to get a more powerful server, or a server that is dedicated to his accounts – either of which would cost more.

Jerry was OK with spending the extra money, but I really wanted to figure out the root of the problem. Just throwing more server resources at the problem wouldn’t fixed the problem.

I did get them to un-suspend the sites, so they came back on line. And a second support chat with BH got me the information that the CMR domain was the source of the overload. With that issue resolved – Jerry’s domains back online – it was time to figure out the cause of the excess load on the CMR domain.

The Investigation Starts

Most hosting places have visitor analysis logs that you can look at via a log analyzing programs, most commonly the AWSTATS and Webalizer programs. You can also download the ‘raw’ access logs, which you can analyze off-line via a program like Log Parser Lizard. I used the on-line Webalizer via the Control Panel (cPanel) on the BH account administration screens.

It seemed to me that I could look at the most-accessed pages to see if there was one page being accessed by an automated process (bot). Follow along as I wander through the Webalizer stats for the current month as I figure out the cause of the CMR site overload.

I first looked at the daily usage of the site. It’s easy to see that there was a big spike in usage on two particular days. (Click on any image to see a larger version in a new tab.)

clip_image002

Now, the CMR site doesn’t get a whole lot of visitors, because there’s nothing new there. Jerry is the main author of posts for that site, and since the stroke he has less time/energy to devote to writing CMR articles. (It does take a bit of time to write posts – this one took me a couple of hours.)

So a big spike in traffic is not real people accessing the site, it’s probably a bot of some sort. Those spikes on the 18th and 22nd were not a result of real people accessing this site. Those spikes are out of the normal range of visitors to CMR. I know this because I keep track of site statistics for all of the sites that I manage – my sites, Jerry’s sites, and my client sites.

This chart verifies the numbers of the first chart.

clip_image004

The conclusion of looking at the above two charts – an indication of an automated (bot) process accessing the CMR site.

Based on that theory, this next chart shows the most-accessed pages.

clip_image006

Usually there will a good spread of use on a site’s pages. The home page will be the most accessed, and other pages will show less activity that the access to the home page.

The above chart shows that one particular page (the “Tell a Friend” page – “TAF”) was getting tons of visits, out of proportion to the other pages (which show in the rest of the chart; not shown here).

This was verified by looking at the “Total Entry Pages” and the “Total Exit Pages” – the next two charts.

clip_image008

clip_image010

This tells me that the TAF page is the entry and exit point of the visitors – they are only looking at that page, and not elsewhere in the site.

I know what the TAF page is – I know what all the pages are, which is a good thing for a web site admin to know. And those two charts are verifying the hypothesis – a bot is accessing the TAF page. And the bot is only accessing that page, not other sites. You can see from the two charts above that a disproportionate number of visitors are accessing that TAF page.

So, who is the culprit? Is it a single user, or a bunch of visitors? This next chart, which shows the Top 30 IP addresses accessing the entire site, tells us that one IP address is the top visitor to the site. Note that the percentages are an indicator – but of the entire month.

clip_image012

And who is this visitor? Let’s ask the googles. I put that IP address into the search bar, and come up with this:

clip_image014

Aha! A computer somewhere in Shandong, China is the high-volume visitor to CMR. The googles give me more verification that the visitor is a bot:

clip_image016

So we have a good indicator that the excessive traffic to the CMR site is a bot:

· They are only accessing one page, not any other part of the site

· The bot attacker is from a known hacker location.

We could verify this by deeper analysis of the ‘raw logs’, but I didn’t think it was necessary. The above charts show that the TAF page is getting way too much traffic than would be normal.

The Form

Since I built the CMR site, I know what the TAF page is. It’s a simple way to let people send an email to someone they know. It is a simple form:

· The ‘from’ email (the visitor)

· The ‘to’ email (who the visitor wants to send a message to)

· A short message

When a visitor fills out the form, the site sends an email to the ‘to’ (recipient) with a short message of “check out this site; I think you’ll like it” plus any additional message that is entered into the form. It’s a simple way for a visitor to recommend the CMR site.

The form is set up via the “Contact Form 7” plugin for WordPress. A great plugin; it allows the site web guy to easily create contact forms. I had that particular form to send me a BCC, so I could monitor the use of the form. It worked well, although not too many people used it. I was getting under 10 emails a week from this form.

Contact forms are often a target of email spammers. They fill in the form, and put some links in the message area, hoping that someone will click on their spammy link, which is how the spammer collects revenue – by people clicking the links in the spam email. (Don’t do that, even if you are curious. It’s a risky business – a great way to get your computer hacked.)

We Digress

A digression about comment forms and spammers.

“Comment Spam” is a problem for web site owners. They clutter up valid comment areas, and they cause excess load on the site’s server resources (our problem), slowing down valid access to the site.

There are some ways to try to limit form access only to ‘human’ (non-bot) visitors. Captchas, hidden fields, a math problem (what is 2+3?), are among the possible ways to block comment bots.

What is needed – and more effective – is a way for the form to ‘sense’ a human user.

I’ve done a bit of research into this, and did come up with a solution that works. The basic premise of my solution is that bots cannot emulate a click on a form field. And you also don’t want the bot to grab the ‘form process’ page – for you geeks, it’s a parameter in the ‘form’ tag – that processes the form submission.

My solution for that, which I use on high-volume sites, is to include code in the form that hides the ‘form action’ page, and also senses a user clicking on a field. The implementation is technical, but I’ve made it available for free on my “Form Spammer Trap” (FST) site at http://bit.ly/2cvhFi6 site. With this additional code in the form, a non-human (the bot) will be ignored (and sent to the Form Spammer Trap site). The process works great; I’ve never gotten any bot-submitted spam on a form that uses that process.

Since the CMR site is not a high-volume site, I didn’t bother with implementing the FST code for the TAF page, although it is implemented on the Comment forms. It is a bit of work to implement on a Contact Form 7 – type form, and that form wasn’t being used much. I’d get an occasional TAF BCC (I get a copy of that form access just to monitor things), but not too many of them to worry about.

Until earlier this month.

The Initial Response

I started getting TAF BCC messages, several thousand at a time throughout a day. It was clear that the TAF form was being attacked by a form-bot. Since that form wasn’t being used much, my initial response was to remove the form from the site.  I removed the TAF link from the menu; the page was still there.

That initial step did not work, the form spam was still being sent by the bot, because it already knew about the page. And there was another result – since the form-bot was sending this to random email addresses, those emails were rejected by the recipients’ mail system. So I was getting bounce messages – tons of them.

I use Gmail, and Gmail was putting all of those bounce messages in the Spam folder. Tons of them each day for a couple of days. I then removed the page entirely from the site, which eventually (after a few days) resulted in stopping all of those bounce messages being processed by Gmail into my spam folder.

That turned out to be a temporary reprieve, as you can see by the above charts. Even though the page no longer existed, the spam-bot was still trying to access the page. So the spam-bot was still causing problem. Removing the page will stop ‘future’ spam-bots, but this guy was still trying to load the page.

The Big Stop

Back to yesterday. I identified the spam-bot by IP address. So I needed to block those IP addresses before the request got to the site. This was done by using IP Address blocking feature in the BH Control Panel.

I knew that the CMR (and Chaos Manor) sites probably did not need the visitors from the Shandong Hacking School. And that there would be multiple IP addresses being used by the hacking school. So I decided to block the range of addresses from there. Here’s that screenshot:

clip_image018

There are a couple extra ones in there, but it looks like anyone from the hacker school will be blocked from accessing any of the CM/CMR pages.

Closing Thoughts

This may not be a final solution. I’ll need to keep monitoring the site stats to see if other IP addresses are being used by the spam-bots. And we may need to use some ‘cloud’ caching to reduce the load on the shared server.

It may also be advisable to move to a more expensive hosting plan that allows for more powerful resources on the server. That’s what the BH support guys said, but I don’t think the extra cost is justified at the moment. If the CM/CMR sites get a lot more visitors (and it would have to be a significant increase than now), moving to a more expensive plan with more resources might be advised.

But I thought that is was an interesting process to determine why a site is getting hammered. Perhaps it will help you CMR readers.

Another Backup Strategy

[In a previous article, Drake Christensen described his backup strategy using a network attached storage (NAS) and software to immediately backup changed documents. Your intrepid editor has a different strategy, since he doesn’t need immediate backups with version control.]

pc-bruciato-fireIt is a Good Thing to have a way to back up your important files. And since it is also Emergency Preparedness Month, having a good backup strategy is a good subject to visit.

There are tons of ways to backup your data, and many reasons to actually Do It. You’ve probably heard about all the reasons ‘why’. My main focus on my backup strategy is three-fold:

  • keep backup copies in several physical locations; not just at home
  • make the backup process easy and mostly automatic
  • allow backups of multiple devices (there are four computer systems in our house) while reducing costs

Backing My Laptops

The first process is to copy important documents/files (pictures, project code, document, etc) to a central location in my home. That is a desktop computer sitting upstairs, connected to my LAN. It’s an older system, not used as much anymore, but there is a big hard drive on the system.

I use Microsoft’s SyncToy, a free program that synchronizes files between two locations. In my case, I use it to one-way sync between laptop (source) and desktop (target). SyncToy has the advantage of only copying or updating files that have changed or been deleted. Only those files that meet those criteria are copied from the laptop to the desktop. With thousands of files on my laptop, that saves a bunch of time.

Since SyncToy is free (thanks, Microsoft!), I have it installed and configured on all three of our laptops. Right now, it is a manual process to do the ‘sync’, but there are ways to set up a ‘batch job’ and schedule the SyncToy task on a regular basis. My practice is to do a sync every couple of days. This is OK, since most of my work is with web sites, and the web site code files are also available on the external web sites in case of disaster.

ScreenShot394SyncToy works fairly fast; once you set it up, just run the sync task and let it do it’s thing. Here’s the results screen of the last time I ran SyncToy. You can see that a bunch of files didn’t need to be copied, because they hadn’t changed. That saves a bunch of time on the backup.

Now, I could back up to an external hard disk, either manually or with SyncToy. But the desktop is available on my LAN, and backing up via wireless is Fast Enough for my purposes.

Backing Up the Desktop

At this point, all my important files are in two places: my laptop(s), and the desktop. But the files are in the same physical location. Any problems with that physical location (theft, fire, earthquake, zombies; take your pick) would result in loss of files – especially all the family pictures, many of them irreplaceable.

So the second part of my backup strategy is to copy important files to the ‘cloud’. For that, I have chosen the Carbonite backup service (http://goo.gl/45wv ). For a flat fee, all my important files are automatically copied to their encrypted servers. There are similar services available from other vendors.

The best part is that ‘automatic’ part. Any file that changes on the desktop is automatically backed up to the Carbonite servers. It happens in the background, so when I use that computer, the backup process doesn’t interfere with my use of the computer.

Carbonite stores multiple copies of my files, so there are some ‘history’ versions of files that are available. You can also access any of your backed up files on other devices or computers – phones, tablets, whatever. This is a great advantage if you travel a lot, since you can access that important file you left on your home system while on the road.

There is another advantage to using a ‘cloud backup’ service like Carbonite. That relates to ‘ransomware’.

I P0wn All Your Filez

Any backup strategy needs to account for damaged files. Files can be damaged by hardware problems, physical damage (fire, etc.) or theft. And then there is ‘logical’ damage – damage done by malware.

There is malware that encrypts your files, requiring a payment to recover your data. This ‘ransomware’ can be a big moneymaker – reports are that up to US$18 million has been paid to recover encrypted files. While there are things you can do to block ransomware – or any malware – think ‘safe computing’ practices – your backup strategy can also help prevent file loss from ransomware.

Ransomware can damage files from any infected computer on your network – even your little home network. If a file is available on the network from an ransomware-infected computer, then that file can be encrypted, even if it is on another computer.

So your backup strategy needs to take possibility into account.

There are a couple of ways that you can enhance your backup strategy to protect from ransomware:

  • Copy files to an external drive (or even DVDs), then physically disconnect that external drive from the network.
  • Use a ‘cloud backup’ service.

As you may have guessed, my backup strategy to prevent possible ransomware problems is using a cloud backup service from Carbonite.

gTWVIWhat about copying files to a Linux-based Network Attached Storage (NAS) system? It is likely that configuration includes access to the NAS by Windows-based systems. So there is no protection there. Of course, you could run all-Linux systems, but that is less likely for most people.

So the Carbonite-based cloud service is my solution. They keep multiple backup copies of my backups, so even if the ransomware gets to my desktop system, and Carbonite backs up those encrypted files, I can work with Carbonite to get a prior, non-encrypted versions of my files. I might lose a few recent files, but the majority of my important files would be available to get back – after I rid my systems of the ransomware.

Wrapping Up

So there you have it. My own personal backup strategy. It works pretty well for me. I haven’t had to recover files – mainly because I practice ‘safe computing’. And have been lucky enough to not have any disasters.

But I am prepared. My important pictures, documents, web site files, etc., are available-  Just In Case.

What is your backup strategy? What do you do different? Or do you just trust in your ‘karma’ to keep away the possibility of file loss? Let us know in your comments – or write up your own backup strategy for an article here on Chaos Manor Reviews.

After the Storm

 

[A bit different, but there is a “I did this so you don’t have to” angle on this story from our editor about emergency preparedness. It is, after all Emergency Preparedness month. Originally published on his blog. – Editor]

storm-treeOur house is on the northeast corner  of the Olympic Peninsula in Washington state, not far from the Hood Canal Bridge. Although we get about half the rain of Seattle, there are the occasional windy storms that come through here. This weekend was one of those; a extra-windy affair with rain, that usually happens once or twice a year around here.

The weather forecasters predicted very windy conditions, with gusts up to 55mph. Since there are a lot of trees in Washington, there was the real possibility of falling trees causing damage to the electrical lines, resulting in power outages.

There was several days of warnings about the storm, so plenty of time to lay in supplies. If there were electrical outages, they might last 12 to 48 hours or more.

We had a similar storm last year, with a power outage of about 8 hours, starting in the evening. But this storm was supposed to hit mid-day. I figured that would be the case again with this storm.

So I did some minor preparation at home. I knew that I had plenty of flashlights – and batteries. We had some canned food, plus fresh fruit, some energy bars, and four cases of bottled water. We have a small chest freezer with some meat in there, and a good propane BBQ grill with an extra fuel canister. I figured we could handle a short power outage, even if it did happen overnight.

Storm On The Road

The storm came as scheduled on Saturday. We had family in town, so had planned a trip across Hood Canal bridge to Silverdale to visit the local marine museum with my daughter’s’ family (husband, wife, two and four year old). We went across the bridge, and it was a little windy, but not too bad. Enough wind that there were 2-3 foot ‘rollers’ and a bit of whitecaps. But winds gusts under 20mph. The bridge will close if the winds get over 45mph.

We made it to SIlverdale OK, had a great visit in the museum (a hands-on place with lots of touching of the sea critters, to the delight of the grandchildren). It started to rain a bit more when we got there, but not really a downpour.

After the marine museum, a trip to a hamburger restaurant. Lots of people there; it was lunch time, but service was good. It was a bit windy outside, maybe 15-20mph, and some rain. There was a bit of light flickering due to power issues while we were inside, and one 10-second outage, but all was well.

During lunch, I was watching the roads (via Waze and Google Maps), and notices some slowdowns on the usual route home that appeared to be just traffic-related. The drive back home is about 35 miles; some four-lane divided highway, some two-way undivided before the bridge.

After lunch, we went over to the local Costco to look for a replacement laptop (didn’t find the right one). But I thought it would be a good idea to get a LED lantern and some extra batteries – extra batteries are usually a good idea. (the Costco Kirkland brand is a good value).

The Costco was the usual Saturday-busy, but we got out OK. And back into the car for the trip home. The traffic on the four-lane highway wasn’t too bad. But then we got to the about 10 mile two-lane highway part. That was backed up solid and stopped. It looked like traffic was coming from the other direction, so figured there was just more traffic than usual.

While stopped, I was checking out the traffic, seeing if there was another way that might be better. But there are really only two ways to the Hood Canal Bridge. Our usual route was jam-packed.

And, my cellphone was no help. No bars, so no traffic help from Waze or Google Maps. The wind had knocked out the power (trees into power lines), so no idea which way was the best way home. After sitting in nearly one spot for about 30 minutes (the “this should start moving in a few minutes” kind of wait), I decided to turn around and try the other direction home. That turned out to have less traffic to the bridge, although the ‘long way’ around.

We crossed the bridge (more rollers and white caps on the water) with some crosswinds. The bridge had been closed for a couple of hours due to the wind, which caused the big backup on the main route to the bridge. Our alternate route wasn’t as busy, and the bridge was open by the time we go there. Then on to the two lane road to our small town. And on that road, you could see several power lines that have been downed (but off the road) by trees. It didn’t look good for power when we got home.

Back Home and It Is Dark

And, that was correct. No automatic garage door when we pushed the button. In to the front door to a dark and power-free house. It was about 430pm, so plenty of light from the big windows in the main room. But it was time to prepare for darkness – find the flashlights (where were they?) check the batteries (several flashlights were dead, but I did have replacement batteries), and set up the LED camp light. The water was still running, though.

A reminder to everyone to stay out of the refrigerator and freezer (the ice cream cake we brought home to celebrate a birthday was a bit soggy due to the long ride home, but still good). There were hard-back books for a some, ebooks for others, and a movie on an iPad for the kids. When it got dark outside (and inside), we turned on the LED camp light (a nice amount of light) until it was bedtime for the kids, with flashlights issued as needed.

With that over, some quiet time for the adults, then off to bed about 10pm. I was able to keep up with the local power company’s efforts via social media on my phone; the cell towers were still working.

There were many power lines down in the area; the Olympic Peninsula around our home had about 12,000 customers in the dark, with much larger impacts throughout the region. Crews were (and still are) working on things, but big trees falling on power lines does cause some damage that takes a while to repair.

I use a CPAP machine for my sleep apnea. That didn’t work, of course, and sleep was difficult for me because of that. Power was finally restored around 3:30am for us. Up early for church, where everyone swapped power outage and storm stories. Some people in more rural areas were still powerless that morning, and throughout the day. Some still are, as I write this on Sunday night.

Reviewing Things

Now all of that is a rather long preface to ‘I did this so you don’t have to’. I read a few blog sites that talk about ‘prepping’. After thinking about my preparations for the wind storm, how did I do?

Well, I did have some flashlights, although it took a bit to find them all, and get them working. The food in the freezer and refrigerator stayed cold, because the power outage wasn’t very long (and it was a good excuse to eat extra ice cream cake). I didn’t have to worry about a cold night; I do have a propane fireplace, and the propane tank is full, but the nights are mild (around 55-65F) this time of year.

There was food that could be used for an extensive power outage, although not that much. I did have water (the municipal water supply was working through the outage). Lots of toilet paper, so that is covered. There would have been cold showers in the morning, though, since I have an electric water heater.

But my flashlight supply wasn’t really ready; I did have to do some digging around in the garage a bit to find working ones. The new LED lantern was a good purchase; we’ll get another the next trip to Costco. And I have lots of spare batteries, along with two crank-type LED flashlights, one with a radio.

My cell phone was mostly charged, but my backup cell phone battery pack was not (I had used it the weekend before, and hadn’t thought to charge it yet). My CPAP machine only runs on house power, so I didn’t sleep that well – getting one that runs on 12v might be a good idea.

Food supplies were passable, but an extended outage might result in a not-healthy diet. Our personal medicine supply was good. My first aid supply is very basic – bandages and antiseptic cream. I have some antiseptic hand wash stuff, but not enough for an extended period of time.

There were lots of trees down in my area. I had an alder tree, about six inches in circumference, that split and fell, luckily not on my house. A neighbor helped cut the damaged branch – he can use it for his wood stove, but I’ll need to cut the rest of the tree down – so where is my bow saw?

Lessons Learned

Looking back, I probably could have prepared better. There was several days warning of the impending wind storm, and I knew that the area is prone to power outages during wind storms. More and varied food might be better. I may need to consider a small generator to keep the refrigerator and freezer cold.

Perhaps heading out on the road just before a storm hits is something that is less than ideal.

When the power came back on, I didn’t think to check the status of frozen meats in the freezer; since the outage wasn’t that long, and we kept the doors shut, I think the frozen food is OK.

I was prepared to cook on the propane grill; I had an extra propane tank. But it might be a good idea to get a small two-burner propane stove, which would be more efficient than the propane grill for some meals. Both cars were full of gas, so I could have charged my cell phone batteries there, but I need to ensure my cell phone ‘battery-brick’ is kept charged, and maybe buy an extra one. II

I could use more LED flashlights, and batteries. Maybe even a solar battery charger.(I did order a couple of solar-powered flashlights to try out.) And another LED camp light or two. And I need to organize the emergency supplies to have them in a central space, so I can find them. (I still haven’t found my LED head lamp.)

I need to be aware of alternate routes in the area. Perhaps a paper map would be better for when the cell phone towers are dead because of power outages, or at least an on-line map study before the next emergency.

Perhaps an alternate power supply for my CPAP. Getting enough rest during an emergency is a Good Thing.

So, maybe an overall grade of C+? Good enough for this short outage, but I need to think (and act on) additional things to get ready for the next one. Whatever the emergency is.

What about you? Have you thought about your emergency preparation status? Are you ready for a short-term power outage? Could you survive on what you have in your house right now? Let us know in the comments.

Windows Live Writer – Almost Good Enough

Our intrepid editor maintains Chaos Manor and this site. Dr. Jerry Pournelle mainly writes for Chaos Manor. And he uses Microsoft Live Writer to do that. A new install of that resulted in a problem with entering the title of a post. And that resulted in a call to the Chaos Manor Advisors for help. Which, in turn, resulted in an interesting (well, to the editor, since he did it) troubleshooting process to figure out why.

Windows Live Writer main function is an easy way to write blog entries, and easily publish them to your blog site. It is a stripped-down version of Word, with basic HTML page editing functions. Once you set it up for your blog, you can write something, insert pictures, format text, add links, spell check, and the other usual things. Then there is a one-button ‘publish’ to your blogging site. The advantage to using it is that it is easy for anyone to easily write and publish blog entries.

Microsoft Word has some blog publishing capabilities also. So you could write a blog post in Word, then use the File, Save and Send to publish. One advantage to Live Writer over using Word is that you can see what your post will look like on your blogging site. The Live Writer editor screen will show your post with all of your blogs styling/look.

Sort of.

Windows Live Writer

Windows Live Writer (LW) is like a  step-child of Microsoft. It’s not very well maintained. It doesn’t work well as a full  WYSIWYG editor. Not to mention that if you want to install it, the proper download location (from Microsoft – you don’t want to get it from  non-Microsoft site) is not easy to find.

We use LW to initially create draft posts on this site, and other WordPress sites we have. We paste potential articles from Word (or email) into LW. Then we do the final editing within the WordPress editing screen.

Dr. Pournelle uses it for his posts to the Chaos Manor site. The process works fairly well for us and him. The LW editing screen is clean, with a ribbon bar to do basic formatting. Pictures pasted into LW will get uploaded to the site. The editing screen looks close enough to the final posted page on the web site.

Over in Chaos Manor, Dr. Pournelle has several computers that he uses to write his posts. He’s been rearranging his work areas lately, so he needed to get LW installed on a new system. That was the first problem.

Eric Pobirs, one of the Chaos Manor Advisors, helped get LW installed on the system at Chaos Manor. Dr. Pournelle was having difficulties getting the install process to complete. Eric said:

Essentially, it came down to downloading the correct file to start the install. For reasons that defy my understanding, Microsoft has never done a good job on how they manage the Live suite of apps. My impression is they regarded it more as something for OEMs to bundle with new PCs, like the MS Works suite of yore, and didn’t put the proper effort into presenting it to individuals downloading the product.

There were three major generations, 2009, 2011, and 2012. The earliest does not like post-XP versions of Windows. The middle version was intended for Vista, and the last version for 7 and 8.x. It was odd for a Microsoft program to display such compatibility issues but there it is. The 2011 version never gave me problems on Windows 7 but the only portion I used extensively is the Mail app, which has a long history as Outlook Express.

http://windows.microsoft.com/en-us/windows-live/download-windows-essentials#wetabs=we2012

Microsoft pulled the earlier versions from download availability but they are still offered on numerous sites that are likely to show up in search. They’re hard to distinguish because they always have the same wlsetup.exe file name, rather than carrying some clue to their version up front. Some people are still obsessively attached to the 8.3 file naming convention.

So, I made sure I was downloading the 2012 version and it simply worked. Notably, it showed a different icon than the one downloaded to Swan previously. The .NET 3.5 runtime must have been installed on Swan at some point because it didn’t ask for it as it did on my Windows 10 test machine a few days earlier.

So Eric was successful in getting LW installed on the “Swan” system, making it available to Dr. Pournelle on that system, after he set up the Blog Account in LW for the Chaos Manor site.


The LW Editing Screen

A bit about that. You can have multiple Blog Accounts set up in LW. Each account will ‘connect’ to the appropriate site. You enter the user credentials and the site URL, and LW does some trundling to get things set up. Part of that ‘trundling’ is to download the sites theme (‘look’), which results in templates that are used by the LW editing screen. That template includes the various HTML and CSS for the site’s theme, and is used to present the theme’s look in the LW editing screen. So the HTML/CSS of the site’s theme is an important part of the template used by LW to display content on the LW editing screen.

WordPress themes get updated all of the time with additional features, and probably new and changed CSS styles. LW has a button to update the theme, so it’s editing screen will ‘look’ like a published post on the live site.

The LW editing screen looks like this (a partial screenshot of the LW editing area.

imageYou can see the ribbon bar (similar to the one in Word) across the top for basic formatting (there is more that is not shown on this screen shot). There is an area to enter the post’s title, and the area underneath that is the content area. You click on the Post Title area, type in the title, then move to the content area and type in the content. When all is done, you hit the Publish button, and the post is published on your web site.

This first screenshot shows the LW editing screen when we are using the Chaos Manor Reviews blog account. The CMR site uses a theme called “Voyage”. We’ve done some modifications of it, adding some CSS and other changes that we wanted to have.

The Chaos Manor site uses a different theme called ‘Mantra’. I’ve modified it with additional CSS and code. If you look at the two sites, you can see the difference in how they ‘look’. That is because they use different themes, each having it’s own ‘look and feel’.

Now, let’s take a look at the LW editing screen for the Chaos Manor site:

imageSee the difference? No title area. Just the entry area for the post’s content, and the gray area of the site’s background. (LW doesn’t show the sidebar area, nor the heading/menu area.)

Both screenshots are in the LW “WYSIWYG” mode. On the Chaos Manor site, because it uses a different theme, you can’t enter the title of the post on this LW editing screen. You can get the title area if you toggle off the WYSIWYG mode (with Ctrl + F11). Here’s what the Chaos Manor site looks like in LW with the WYSIWYG mode turned off:

imageThe Post Title area is back, but the WYSIWYG (the look of the post with the site’s theme) is gone.

The missing post title area caused a problem for Dr. Pournelle on the new install of LW. The title area was OK on the other systems he uses, since they were working off of the older version of the Mantra theme.

So it appeared that additional CSS with the latest version of the Mantra theme was causing the Post Title area to disappear in WYSIWYG mode in LW.

Digging into the Problem

imageThat took a bit of digging around to figure out. This next part is a bit more technical, with HTML and CSS code references. But it is interesting, even to the non-web page designer.

LW stores the site’s theme templates in the AppData folder on the computer. Each site is stored in a folder with a GUID-type name. Inside that folder area is the template file for the site. Here’s the file list for the Chaos Manor site; other sites that I have installed on my system have a similar file structure, shown on the right.

The LW editing page uses the index.htm template. The older versions of the index.html file are prior template ‘syncs’, as are, I think, the other GUID-named folders.

If we look at the code inside the index.htm file, we see standard HTML code with CSS styles, etc. Here’s the BODY area of the template code in that file

image

Note the highlighted code at line 172:

<DIV class=”comments-link”><SPAN><SPAN class=”screen-reader-text”>{post-title}</SPAN></SPAN></DIV>

Again, this code is ‘built’ by LW from the theme’s generated code for a page. The {post-title} is used by LW for the input area for the post’s title. Note that it is surrounded by the CSS Class called ‘screen-reader-text’. That’s an indication of where our problem of not seeing the title area on the LW screen. Compare that to the code in the index.htm file for the Chaos Manor Reviews site, which uses a different theme:

<H2 class=”entry-title”><A href=”http://www.jerrypournelle.com/chaosmanor/”>{post-title}</A></H2>

That gives us a clue as to the problem with the disappearing Post Title area on the Chaos Manor LW screen. The {post-title} is surrounded by the ‘screen-reader-text’ class. In the CMR code, there is a different class. So looking at the ‘screen-reader-text’ class is our next step. Here’s the CSS code for that CSS class.

.screen-reader-text {

position: absolute;

left: -9000px;

}

Digging into our knowledge of CSS stuff, we see that any HTML code using that class will have the text positioned 9000 px (pixels) to the left of the current position. That will position the visual text off of the LW screen (and off of the browser screen when the page is viewed). Screen reader applications (for the visually impaired) will be able to read the text, but a ‘normal’ view of the page in a browser will not show any content surrounded by that CSS class ‘div’.

Since LW editing screen uses a browser-representation (based on the LW template created from the site’s theme) of the post, that CSS class was the cause of the disappearing post title area on the new (with the current updated theme) installation of LW on Dr. Pournelle’s ‘Swan’ system. That particular bit of code is not in the LW installs for the old version of the Mantra theme, which is why Dr. Pournelle was able to see the Post Title when using LW on those systems.

The Fix is In – Sort Of

So, how to fix that? The quick way is to use the Ctrl+F11 toggle to get out of WYSIWYG mode on the LW editing screen. The disadvantage of that is that you can’t see what the post will look like when published. For instance, the ‘block quotes’ we used above to show the code contents will look similar to the web site version, with indentation, a lighter gray background, and a white border around the box. If you toggle off the WYSIWYG mode in LW, that area is just shown as indented text. But that is a Good Enough solution for Dr. Pournelle.

You could modify the Mantra Theme to not put in that CSS code. That takes away some of the ‘accessability’ of the site to visually impaired visitors.

And there is a risk in modifying theme code, unless you use a ‘child theme’ (as we do on both sites, and on all WordPress sites we make). If you don’t use a child theme, any changes you make to the theme’s code or CSS will get destroyed with a theme update. And if you do use a child theme, you may have to duplicate a lot of the theme code – depending on how the theme is ‘built’. Either way, some PHP skills are needed (among other skills). We’ve done child themes, and recommend them, but there is some effort involved.

You could change themes, of course (and we may do that at some point on the Chaos Manor site), but that requires a lot of testing and tweaking the new theme; things that you don’t want to do on a live site. And finding just the right theme with all of the features you want to have can be quite a ‘time sink’. You can spend hours finding the right theme (I’ve done it). And still things aren’t quite what you want.

Or you could build your own theme. There are theme building templates to help out, but that is still a bunch of effort requiring PHP, HTML, and CSS skills. (Again, I’ve done it – or at least, started on the process. Many hours/days/weeks of coding and testing are required to build a theme that works well.)

Now, it may be, as the executive editor and web guy of the Chaos Manor and Chaos Manor Review sites, that I’ll change the theme of Chaos Manor to be closer to Chaos Manor Reviews. Again, time is involved in that.

But in the meantime using the Ctrl+F11 key to toggle in and out of WYSIWYG mode just so Dr. Pournelle can type in the post title is the best short-term solution.

Whither Live Writer?

There aren’t many good alternatives to LW. You could use the native editor in WordPress, but that requires learning a bit about the WP Admin area. LW is great, since it doesn’t require any access to the WP Admin area.

You could use Word and publish there. That would work with simple blogs, but once you get into more than the basic formatting, Word is not the best solution either; it creates a lot of HTML ‘gunk’ in the page code.

Will Live Writer ever be more than ‘good enough’? Microsoft has announced that they are planning on taking it open-source, which might fix all of the little problems it has (including, hopefully, this one). No announcement of when that will happen, though. One can hope that it will be Real Soon Now.

So, we’re stuck with Live Writer. It is, overall, a great way to easily ‘blog’.  You do have to work around some issues. But it is almost ‘good enough’.

What do you think? What is your favorite blog editor? Let us know in the comments. And if you have a story you’d like to share on Chaos Manor Reviews, let us know here.

A Raspberry Pi Media Server

clip_image001If you haven’t heard, the Raspberry Pi is a card-deck size small computer, great for tinkering around with for many projects. There is tons of information and projects that are available about the Pi; just ask your local search engine. (This article is not meant to be a full technical review of the Pi, there are many other sites that have done that. This page from the Raspberry Pi Foundation has the Raspberry Pi specs and capabilities; lots of other info on that site to understand the power and capabilities of the Pi.)

I was intrigued with the concept of using the Pi as a little computer, thinking back to the days when I would build my own PC. My first personal computer was the original IBM PC model 5150. It had 16K of memory, the 8088 processor, and the operating system was Cassette Basic. (Yes, I am that old.)

I spent about $5K on the whole setup, including an RGB monitor and some accounting software for my wife to start an accounting business with (that was the excuse for the computer). I added 256K more memory with an expansion card, added a couple of 360K floppy drives, and later a 10MB hard disk. It was very powerful.

Then. It was very powerful back then.

More Power!

The Raspberry Pi is more powerful than my first computer. And it costs $35 for the computer itself. I decided that I wanted to get one and try to build a media server for my large collection of DVDs.

clip_image002You do have to spend a bit more than $35 to make it a usable system. I found a kit from Canakit on Amazon that included the Pi Model 2 (Quad-Core 900 MHz 1GB RAM), a 2.5 amp power supply, Wi-Fi dongle (Ralink RT5370 chipset), the 8GB MicroSD memory card (which included the NOOBS operating system), a heat sink, and a case. All of that for around $70. I got mine from Amazon here, there are several kits available; you can get only the parts you need for your project. (That link has all the details and pictures of the parts in the kit.)

I then got a 1TB USB hard drive ($70, such as this one), along with an powered USB hub ($18 here), since the USB port on the Pi doesn’t have quite enough power to run the hard drive. So my total expenditure was around $160. A bit more than the $35 cost for the Pi itself, but much less than my original computer; and more powerful.

SWMBO allowed the purchase, because her hobby (scrapbooking) results in almost daily delivery of supplies for that. So my expenditure, in the grand scheme of ‘things we do around here as a hobby’, was acceptable (she also is the CFO here).

All the pieces arrived in a couple of days.

A Great Tutorial

In the meantime, I poked around the Interwebs for a good tutorial on how to set up the Raspberry as a media server. And I found an excellent tutorial from a guy named Mel Grubb ).

Back in the old days (pre-Windows, DOS 1.0 days), I got pretty good at doing computer things with the command line. The Raspberry Pi OS is “Raspbian” (aka ‘”NOOBS”), which is a distribution of Debian. That’s Linux stuff, which I have played around with some over the years, but never on a daily basis. But I knew some of the concepts, so it wasn’t totally unfamiliar.

Using Mel Grubb’s excellent tutorials, I was able to enter the necessary incantations to install and configure all of the software needed to set up the MiniDLNA media server software. I won’t repeat all of that here, but Mr. Grubb’s tutorials are the place to go for a very easy and clear introduction to getting things going. In addition to the MiniDLNA instructions, he tells you how to set up a NAS (Network Attached Storage), BitTorrent, a VPN, and more. All very readable, with clear instructions.

The result was a full configuration of the Pi as a Media Server, with remote access to the Pi (via Secure Shell – SSH and control via the Webmin interface) so it doesn’t really need a keyboard or monitor for things to work. There were a few false steps along the way, so I needed a bit more googles (plus some questions answered by Mr. Grubb)  to figure things out until I got things working as I wished.

Ripping and Copying

Now that the Pi Media Server was configured, and visible on my home network, it was time to figure out how to ‘rip’ the DVDs into media files that could be stored on the USB hard drive.

After a few false starts, I settled on the WinxDVD software at about $35. Installing the software is the usual install wizard process. Operation is just a basic three-step process; insert the DVD, click a few buttons, and the software rips the DVD into your desired format (I chose MP4). The ripping process takes about 45 minutes per DVD, depending on the DVD length, and the hardware capabilities of your computer. I ran it on my HP Pavillion DV7 laptop under Windows 7, all was well after the Windows 10 upgrade. The WinxDVD software runs nicely in the background, so I could use my laptop for other purposes while the ripping was being done.

The Pi is on my local network via it’s wired and wireless Ethernet connections, although video playback via wireless is just fine without any ‘stuttering pixels’.  I could transfer the files from my laptop’s wireless connection via the LAN. I found it a bit faster to shut down the Pi and connect the USB hard drive to my laptop and transfer the media files that way. Once the USB hard drive is connected back to the Pi Media Server, and the Pi is restarted, all of the DVDs I have ripped are available on my networked media Server.

Viewing the Movies

We have a Roku box here connected to the main TV (well, there’s another one for another TV). The Roku is connected to our LAN via a Wi-Fi connection. It comes with an application that will connect to the Pi media server. So all access to the movies on the Pi media server can be done with the Roku remote. The quality of the movies is just as good as the DVD player. (The WinxDVD software will not handle BluRay DVDs, but they have a BluRay version available.) We’re not that picky on movie quality, DVD 1080i is just fine. And we have some old DVDs that are only 720p resolution, but those play just fine through the Pi Media Server. I also copied some digitized home movies files, and they viewed fine. (Now I have even more things to embarrass my children when they visit.)

There are ways to convert your old video tapes to MP4 files; a quick search came up with this article; the process would require a RCA to USB cable and some software, plus your VCR. I haven’t done that, but it would be a way to get more old home movies onto the media server. There are also services that will do that conversion for you.

As I was reviewing this article with the Chaos Manor Advisors, Eric Pobirs chimed in with this observation on VHS conversion:

    I did that quite a lot for my sister when she had a  VHS-C camcorder. The product we used was the Pinnacle Dazzle, which came with their Studio software. (They’ve since been acquired by Avid, so they have heavy duty connections.) That product is now a lot less, listing at $70. It should work with anything that outputs to composite or S-Video.

http://www.pinnaclesys.com/publicsite/us/products/dazzle/dvd-recorder-hd/

There are a lot of much cheaper products out there. A few months ago I got an SIIG branded device that Frys was selling for under $10 on a promo. Just in case I ever needed such a thing again as the Dazzle went missing at some point after my sister switch to a digital video camera.

The Pi MiniDLNA Media Server will also handle pictures and music, so you can access those via your LAN, just copy the files to the appropriate folder on the USB hard drive. The 1TB drive will hold a lot of movies and music. The MiniDLNA software on the Pi will handle multiple hard drives if needed.

Wrapping Up

clip_image003The whole project was fun to do. So much so, that I made two more (one for each daughter and their family). I got a simple wooden case from the local craft store, and a can of brown spray paint. I cut a notch in the back of the wooden box for cables, and used some Velcro squares to mount the three pieces in the box. The picture with this article shows the finished project. (It is convenient to have a little surge protector to power up everything, since the Pi doesn’t have a power switch, just an external power supply.)

The Raspberry Pi is an interesting platform for anyone to try out. The Pi’s Raspbian NOOBS OS comes with MineCraft, plus the Python programming language, and a simple programming language for kids to try out.

There are a ton of projects out there for the Pi;. I’ve seen home security, robotics, motion-sensing cameras that send texts alerts, flashing lights, wireless phones, mini-tablets, and more – any web search will find ideas for your own projects. Home schoolers will be able to find projects that will help any kid ‘get geeky’. If you can think of a project for it, chances are that someone has already done one that you can try out.

Since the Pi’s operating systems is on the memory card, you can swap it out for your different projects. The ‘HAT’ interface (Hardware Attached Things) can be used to control just about anything. The possibilities are many and varied.

And it’s much cheaper than my original $$$ computer.

What do you think? Have you made something with the Raspberry Pi? Use the comments below to add your thoughts. Or write up your own story and submit it to Chaos Manor Reviews for publication consideration – details are here.