Bay 2 Predicted Failure

As reported previously, there was some excitement two weeks ago when a disk drive that’s part of the raid cluster in my main computer just vanished under load. The raid card did it’s thing, and the computer continued to function. I rebooted, and the disk showed up again, I made it a spare, the raid card grabbed it for the raid set, and started rebuilding.

All good.

But then the following Monday, the power blipped. The computer is on a UPS and was still on when I sat down at my desk, but the MacPro had a blank, black, screen, and a mouse cursor, and nothing else. Nothing worked to break it loose of that, so I power cycled it.

Aaaaaaaaaaand, it wouldn’t boot. It would give the startup chord, and then just a white screen.

If I pulled the raid card, it would boot up on a USB stick. I could then see all 4 disk drives, but they weren’t usable because they were raid disks.

After buying a replacement disk drive for bay 2 that showed up DOA, and another raid card that is still in the machine as I’m typing this, I determined it wasn’t the raid card.

Somehow, the raid set was borked. The Teeming Millions general opinion of the Apple Raid Card is pretty low, but I figured it’s whole reason to exist was to not let the whole raid set get borked, no matter what happened. Naive, I know.

What fixed it was this:

• Pull raid card again
• Boot from USB stick with MacOS Installer on it
• Start disk util
• Erase all 4 now non-raid disks
• Put raid card back in
• Boot from USB stick with MacOS Installer on it
• Use disk util to get to raid util, and re-create the raid set
• Install MacOS Sierra on raid set
• Restore from Time Machine backup

This all worked. Restoring from a Time Machine backup is about as painless as a backup could ever be. I lost all my virtual machines, because they’re just too large and too active to back up to TM, but that’s okay. The Ansible setup I use to create them was backed up.

But of course, bay 2 dropped out again during the rebuild three days ago. This time, the raid card notified me, and the machine is still running. I have 2 more Hitachi disks coming, hopefully one of them will be good, and I’ll replace bay 2. I’m going to run on the 3 remaining good disks until they arrive.

But today I had to reboot it. The disk drive showed up again, as “roaming”, which just means it isn’t a part of the raid set. I expected this. What I didn’t expect was the computer informing me that the disk in Bay 2 is about to fail.

I spent a solid two minutes looking at that thinking “No shit, you couldn’t come up with this two weeks ago? What the hell!”

Raid Set 1 Degraded

My MacPro has an Apple raid card in it, because I hate having 4 separate disk drives in a computer. The raid card makes 4 drives look like one big one. The card also protects when one drive fails. Which seems to be what happened just now:

The drive in Bay 2 just vanished while in use. Because I have raid 5, the computer is still functional, and I have access to all my data. But access will be slow, as the raid card has to calculate what was on the missing drive from info on the 3 still working. Also, if one more drive fails, I lose the whole machine.

The problem might just have been utilization. I had a Win10 VM cleaning up its disk space, Lightroom loading a large, 17,000+ photo catalog, iTunes playing, iDrive downloading a few gigabytes of photos. Disturbingly, the VM and Lightroom seemed to hang, which is not supposed to happen on a raid protected system when a disk fails.

This computer is a “Mid 2010” MacPro, purchased in early 2011. It’s kind of amazing that this 6-year-old computer and its equally old spinning hard drives haven’t thrown so much as a hiccup before now. In my experience, hard drives last 3 years.

There are 3 possibilities, as I see it:

1. The bay 2 drive is really failing.
2. Drives are getting old, just too much work for too long, could have happened to any of the 4 drives in the machine.
3. Power supply weakness, could have been kernel panic just as much as hard drive failure.

Powering down and powering up the machine brought the drive back. I made it a spare, and the raid card slurped it into the raid set and is now rebuilding.

I expect that to take a day or two, if I remember correctly from when I first set it up 6 years ago.

I have a plan to replace all the spinning drives with SSDs, but that plan really can’t happen now. I can buy a replacement hard drive of the same model number for about $60, and that’ll be an option if the rebuild doesn’t finish. Might need to do it anyway, since drives in a raid array like this tend to crap out all at the same time.

I might order one Hitachi HDS722020ALA330, and if it works with the raid card (it might not, this raid card is the worse one ever made and might be really picky) I’ll get 3 more and replace them all one at a time. Maybe.

St. Paul and the Broken Bones

If you miss ‘70’s soul you can’t go wrong with St. Paul and the Broken Bones. I found them thru an iTunes alternative rock playlist. They have horns. I thought they were Chicago (way way back when Chicago was good) until I looked at what was playing.

Very tasty. Given them a listen.

Wook Leaves Twitter

I deleted my @wookman Twitter account.

I wasn’t using it. I mostly followed local law enforcement, so I could see when they were doing roadblocks and speed traps. That didn’t happen often enough when I drive to make it worth it.

The account I’ll miss is InternetOfShit. You don’t realize just how stupid most “Internet Of Things” really are until you see it all en-mass. Tho Bloomberg got into it with the Juicero, which was hilarious.

The other reason is one I’ve been talking about for years: I just don’t get Twitter as anything other than a marketing platform. Want to crow about your product, but don’t want to spend money on press releases? Tweet. Want to publicly complain about something? Tweet. (Something I’ve been guilty of doing, and it didn’t help.)

But for keeping in touch with people? Impossible, unless you can like scrolling thru 8000 tweets to find them all.

There’s also the fact that bullies and trolls that make Twitter money, will be allowed to bully and troll while the rest of us get slapped down for lesser abuses of the Twitter terms of service.

So I’m out.

Fifteen Years Later

I hope everyone takes a moment today to remember that something horrible happened 15 years ago. People died in two cities and a remote farm field.

But do not dwell on it. Look to the future. Know that even tho the news stations will tell you all about how you aren’t safe, they are lying to you to to generate ratings. You are more likely to die by being hit by lightning than in a terrorist attack or in a school shooting.

I don’t say this lightly. I lived in New York the day the Twin Towers fell. It changed my life, and my city. But I am resilient, and so is New York. I don’t know if we are better now, but I do know we aren’t worse.

So remember, but look forward.

systemd

I’ve been working with CentOS and RHEL 7 lately, and had to get into using systemd and systemctl instead of the “service” command and the init.d system.

I gotta say, systemd is a cool piece of work…

…writen by a Windows user, for Windows users.

And no, I don’t mean that in a good way. This thing is fantastic for the desktop, but hardly applicable on servers. Linux lost on the desktop and owns servers.

The Mac has the launchctl system, which is an earlier re-do of the unix init system, and systemd looks a lot like it. Systemd is more capable in many ways, but none of those ways are going to make it easier for humans to administrate servers. Binary log files are hostile to server admins for starters.

Leaves me wondering why we need this on Linux servers.

MyFitnessPal App

I installed the MyFitnessPal app on my phone on Saturday. I signed up on the website for it too. It is a food slash fitness tracker, much like any other. It integrates with Health on the iPhone.

You put in some current information, and goals. I put in my height and weight, and gave it the goal of losing weight, and what my target is. It then suggests a calorie count to aim for each day, as well as the types of each nutrient that you should have, carbs, fat, etc. You enter what you eat each day, and it has a huge list of foods defined that you can pick from, and it then uses that info to track you against your goals.

I am not interested in the suggestions for how to reach my goals. I have a plan, and it has been working so far. I was more interested in finding out what my calorie counts are, and the other nutrition information.

Just three days in to putting my meals in this thing have already been surprising. I had sometimes said it was hard, eating like I’m eating, to actually have 3000 or 4000 calories of food in a day. But I wasn’t totally sure that was true. Now I am.

Friday night, I took Ally out for her birthday, and had a steak dinner, along with a couple of drinks, and desert. The total was a little over 3000 calories, including the drinks. This I expected.

What I didn’t expect was that it looks like my normal daily meals add up to only about 1300 calories total. This is not much. The app suggests 2500, and that’s in line with what my research said I should be doing.

In addition to not eating stuff that isn’t healthy, I’m also doing the traditional starving myself to lose weight, thing. But I’m not worried so much as I’m fascinated. I’m not hungry most of the time, and if I am, I eat something.

I doubt I’ll keep this up. I suck at doing daily things like this. But for a couple of days, this is enlightening.

First Hint of Google Fiber

Saw a car with a sticker on the side that said “Google Fiber Contractor”. Hopefully, they make this happen fast.

What Does Al Jazeera Have Against Trump

What does Al Jazeera have against Trump? Who picked that moment in time to immortalize him for this article?

Pic from Al Jazeera’s english website, but only on the front page. It wasn’t reproduced in the actual article, so there’s nothing to link to directly.

Deadpool

Saw Deadpool tonight. Huge fun. Most fun I’ve had watching a movie in a long time. Not a kids movie. Not even sorta a kids movie. That was part of the fun.

The Rest of My Life

I’ve been “on a diet” now for about 8 years. I’ve lost zero pounds in that time.

In fact, I’ve gained maybe 70 pounds since around the turn of the century, and I was already 60 pounds overweight then. In fact, with one exception, I have always been the heaviest I’ve ever been. Gaining weight year after year.

The exception: I lost 25 pounds in 4 months in 2011. I was walking 4 miles across the Brooklyn Bridge and back five nights a week. It had little to do with the food I was eating. I was eating nachos and loaded tater tots for lunch and 3 scoops of ice cream afterwards most weekdays, and sometimes twice on Saturdays and Sundays. But I was walking 15 to 25 miles a week, and losing weight.

Then I broke my foot, and the walking ended. About the time I could walk on it again I moved away from the bridge. I gained back the 25 pounds over the next year, and more since.

I did learn tho, that you don’t have to diet to lose weight. Walking is often called the best exercise you can get, and I’d have to agree. Four months of walking across the bridge also improved my core strength. I was able to do pilates moves that my trainer never would have thought to ask me to try prior to that.

I moved away from the Brooklyn bridge at the end of 2011, and it was probably the worse thing I had ever done for my health, ever. While walking across the Bridge was annoying, the annoyance factor kept me engaged and it was rarely boring. I was engaged in the process of getting across the bridge without getting killed by bicyclists or trapped by tourists asking me to take 1000 photos of them in front of every conceivable part of the skyline.

I’ve tried to get in the walking habit again here. It’s boring. City streets just don’t have any stimulation, no engagement. Interesting places to walk are a drive away, and if I can’t just go out and do it, I typically don’t. I’ve tried walking on treadmills, but couldn’t keep at that either. I started walking across the bridge when I realized I’d given up on the treadmill twice, soon to be three times.

So here, where I live now, with no Brooklyn Bridge to make the walk happen, weight loss has to be via diet.

I usually described the diet I claimed I was on as “like the paleo diet, but not exactly the paleo diet”. But I wasn’t really doing the paleo diet. Or any diet at all. I ate what I wanted, when I wanted it, as much as I wanted. Ben & Jerry’s. Haagen Dazs. Cheese on everything. Sandwiches a staple of almost everything I ate. Huge portions. I never felt hungry, never stopped.

My weight continued to increase. Last year, I swore to myself that I would not let my weight go above 350 pounds. I made no changes. Monday morning after Christmas last year the number on the scale was 354.

I might have freaked out. Just a little bit.

I had to change. I had to loose 150 pounds or so. The morbid joke in my head was “…or die trying.” I realized that I had to reduce my weight or I’d die early. Everyone dies, there is no cheating death, it comes for everyone. I’ve always been dismissive of people that were hyper-obsessed with everything healthy to the point of being almost OCD, as if somehow, you could get out of life alive. But I decided I wanted to fit in a narrow casket.

I made a Facebook post to my friends on January 1st, 2016:

Okay. Time to look forward now. A new year, a new plan, a new goal. My goal is 150 pounds gone. This is not a resolution, because this isn’t for 2016. This is my wanting to lose 150 pounds for the rest of my life. Might take longer than just 2016, but that’s okay. I expect the rest of my life to take longer than 2016 too.

My plan is simple: No eating crap.

Humans are omnivores, designed to eat meat and vegetables, and fruits when they were available. So that’s my diet now. No sugar, grains, dairy, legumes, and only limited fruit. After almost two months of this, I can say it isn’t an easy diet. There are only so many types of meat, and only so many types of vegetables. It requires imagination. Fortunately I have simple tastes in food, so the imagination I’ve needed has mostly been something I’ve been capable of. It helped that Ally has taken this cause up for herself.

So far tho, it’s been working. Quite nicely too.

I do “fall off the wagon” at times. I still love lasagne and cheeseburgers. Eating out is really hard because even simple dishes often come with proscribed sides, or are cooked in ways that include proscribed ingredients. I eat what I want, tho I try to stay towards things that are simple.

Not Fixing The Problem But Still Paying Less

The problem I have with network packet loss and resets didn’t in fact get fixed by the TWC internet speed reduction I talked about previously. However, the real opportunity of way faster speeds from our ISPs, is being able to drop down from the top speed tiers and save some money.

Anyway…

I’m still having the network packet loss, retransmits, and general network protocol ugliness I’ve had since upgrading to El Capitan a while back.

I found some talk about MacOS doing ARP cache validation rather aggressively, in a way that doesn’t jive with how router and switch manufacturers think it should be done. Windows doesn’t do ARP cache validation, as far as I can tell. Linux does it, but not until the ARP cache tables in the kernel get full.

There is a post at MacStadium about the issue. That post references an Apple discussion thread. And from there you can get a “patch”, which isn’t an actual patch, but is just a configuration change made sorta permanent. Notably, it didn’t fix the problem.

I’ve done all that, and while it might have made things better, it didn’t fix them. At least not completely:

--- 166.84.1.1 ping statistics ---
2025 packets transmitted, 2015 packets received, 0.5% packet loss
round-trip min/avg/max/stddev = 31.179/44.917/66.868/2.885 ms

A half-percent packet loss isn’t bad, but until El Capitan, my packet loss was zero.

I’ve replaced everything except the computer at this point. I replaced the old SB6141 cable modem with a SB6183, capable of 300Mbps throughput. I have switched to a Cisco RV325 router. I justified that one because it has 14 GigE switch ports on it, so I could lose the 10/100 switch I was using because I have too much crap on my desk that I want to plug in. With that setup, the Airport Extreme is no longer in the network path from internet to this computer, it is no longer routing or doing NAT, it is in bridge mode. All the cables have replaced with Cat6 cables. (More on this hardware in another post.)

I still have the problem.

I’ve re-loaded MacOS. I had to do that anyway. This machine has been upgraded thru every release since Lion, which it came with in 2011, all the way thru El Capitan, and it had some strange behavior. I also wanted to change the partitioning of the raid array, which can’t be done with Apple hardware without reloading. So I reloaded.

I’ve also checked as best I can, and none of the other computers have this issue, so it kind of has to be the MacPro or Time Warner. I can’t check the other two Macs, because the old MacPro died recently, and the MacBookPro doesn’t have a ethernet port on it, and I didn’t have a Thunderbolt to GigE adapter until very recently. And I keep forgetting to take it out and check. I didn’t see it happening to my work laptop tho, for what that’s worth.