random complexity

with a side of crazy

Posts Tagged 'rant'

The bitcoin distraction

Ok so with the surge in price of bitcoins in june-july spiking to USD30 each and then crashing back to about USD12 it got some decent media coverage.

Perhaps I was in a vunerable situation, looking for something hobby like and a decent distraction. See I needed a distraction from storage as that's a problem still sort of wrecking on the rocks waiting to sink. Mix into this the lack of a vmware box and I've sort of lost the plot again.

So I sank some cash into a box to mine bitcoins, which was fine, or would have been fine if the hardware didn't have issues. Basically I rushed into it and opted for something available right now vs shopping around for a quality product. So the Gigabyte board I have has a fault which took about 2 weeks to isolate, and one of 2 dimms is also faulty. Now I really hate returning stuff, and after dealing with gigabytes useless and slow support decided to never buy a product from them ever again, and I still have to convince the shop that there is a fault. So that will be downtime, and in bitcoin mining - all downtime costs money. Real money.

A further complication came from a software hang issue caused by ATI Catalyst drivers, OpenCL code and an accelerated video layer (or GPU call) on Windows. Basically a major bug in the ATI drivers causes a hang of the machine if a video plays while OpenCL code is running. This was after realising the driver bundle with control center caused crashes itself, so switching to the driver only pack at least let me isolate the problem. Oh and the fix has been committed for the driver release due out AFTER the next one. So October apparently. Thank's AMD/ATI. I'll never buy your products ever again too. For reference, this card was released in NOVEMBER last year.

The memory fault isn't such a problem now, but once this fad wears off (or becomes unprofitable due to the price of power going up due to failure of the government (to cater for population growth or restrict immigration)) I was hoping to use this box as a vmware host. But alas, it's again apparent that I'm not allowed to have things go well or to plan.

Further proof of this was the build up to the final Harry Potter film. My glasses broke, and not having spares anymore (long story) left me without sight. Skip forward a week and I'm still without, and now without more money too. Health insurance is such a scam. It barely covered 25% of the cost of replacement glasses. Sigh.

I'm not allowed to win. Someone please remind me why is it I try at all.

Perhaps I should move to some far away land which is cheaper than here. Perth's now in the top 10 most expensive cities to live, and it's higher up the list than New York. What the hell am I staying here for (apart from America being filled with americans).

Hmmm

HALP

How I became a hypocrite without noticing

Last week I realised I'm a hypocrite. I've skipped on the whole facebook fad, the twitter fad and so on primarily because I didn't trust the owners to do the right thing with whatever data of mine ends up on their service. I'd convinced myself this was the case and from my initial scan of the facebook policies wrote off the whole service. Of course since then there have been many privacy violations/leaks to add fuel to the fire and the privacy policy grew to larger than the US constitution (currently it's split into multiple pages so it doesn't look so bad). Twitter just fell into the chasm of general social networking stuff. I say fad because I'd ignored previous social networking sites just fine, and right when I decided to go sign up, a new one showed up and everyone jumped at once to the new one, so I held off to see if it'd take off and kept waiting again. Well perhaps facebook was third times the charm, it hasn't passed yet, and I'm right at that point again. Expect the next big thing any day now.

Well last week I realised I've got this all wrong. I've already sold my soul to the cloud and have ignored it like a blind spot. Gmail.

I started using gmail when spam became intolerable and spam assassin stopped working enough for me. Maintaining my own email server became too greater maintenance for not enough gain. Gmail's spam filter is excellent and just works. I like things that just work. A decent web mail client wasn't around at that stage or not for free. The ajax'y ones weren't that good yet and gmail's was again, excellent. So I did it, I switched. I even put up with having to forward all my mail to a single email account and handling it all through there. I also put up with the from header being set to it, exposing the actual account name which I never liked. Not a perfect solution but it worked. 80/20 rule be gone, this was more 95/5. Initially it was a tactical solution which just never got changed.

Then when google apps came along, even better, my whole email solution could be google hosted, however I still used forwards to the master gmail account (because the google apps account wasn't a full featured google account at the time) and I used other services. Since then google apps accounts are now 99% full featured (95/5 again) so I moved all my data from my gmail google account to my apps google account, and all was happy again. My from headers are right now, still no spam, it just works.

However by this stage I had already been sucked into the cloud. As much as I don't trust google (or trust them only slightly more than facebook/twitter) I do use it for important stuff - my email. The contents of my account is important, and due to this I was backing it up for a long time. Though not for a while now. When I moved between accounts I realised the backup tools all suck - hint: they don't backup everything and they battle to restore everything if restore is even an option.

From the privacy perspective; Email can be faked easily enough, or lost in spam, so the level of trust/denial is quite low anyway, just like sms. Being one in billions on the service helps you to blend in.

I believe eventually everything posted online will become public. Through poor information security, coding mistakes, disgruntled/curious employees, cracking and so on. If you're hiding information behind a sites privacy settings to keep something private, it's just a time delay lock, not a safe. It's happened before and it'll happen again, information leaks. Don't trust anyone.

So the answer is to either to play with fake information even if it limits the usefulness of it in the first place, or not play the game at all.

Right now I'm toying with two courses of corrective action:

  1. Wipe the slate clean and start a new fully embracing the cloud and what it provides. AKA the whole hog.

  2. Pull back the error of my ways and figure out a new way forward/around. AKA the long way.

I've ignored the middle ground; Combine existing with the rest of the cloud but not embracing by using fake details; because that dilutes the potential and raises more questions than it answers.

Or simply continue on as I have been and ignore my internal monologue's screaming. That's the do nothing option.

Mental note: must not forget to include a random image.

memes - Philosiraptor - Private Investigators

<3 the philosiraptor. As much as I like things that just work. It's getting harder. And no I haven't bought a mac, yet.

Zfs experiment continued

So the zfs experiment continues. Upon the release of b129 I set off into the unknown on a voyage of dedupe. Which at first had the promise of lower disk usage, faster IO speeds and a warm fuzzy feeling deep down that you only get from awesome ideas becoming reality. ahem

Most sources say you need more ram, and that is true, what they don't say is how much ram for what size data set, which might be more useful to home users like me. My boxes have 2gb of ram each, and that is not enough for dedupe, no way near. Not if you have a 6 TB of randomish data. I might retry when I get to 8gb ram but not before. You see, if it can't keep the whole of the dedupe table in ram ALL the time, any write to a dedupe enabled volume will result in reads for the rest of the table, or at least seeks. So what I saw was a gradual slowdown while writing to the volume, I was determined to let it finish, to see what savings I would make, and then scrap it due to performance, but after waiting 16 days for the copy, I cancelled it.

The only way I found to even see the contents/size of the dudupe table (DDT) is: zdb -DD which results in an output like this

DDT-sha256-zap-duplicate: 416471 entries, size 402 on disk, 160 in core
DDT-sha256-zap-unique: 47986855 entries, size 388 on disk, 170 in core

DDT histogram (aggregated over all DDTs):

bucket              allocated                       referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1    45.8M   5.69T   5.66T   5.66T    45.8M   5.69T   5.66T   5.66T
     2     394K   43.0G   40.3G   40.3G     821K   89.0G   83.0G   83.1G
     4    9.90K    527M    397M    402M    47.0K   2.35G   1.76G   1.79G
     8    2.06K    125M   82.4M   83.4M    21.1K   1.20G    795M    806M
    16      391   13.7M   8.54M   8.76M    7.26K    272M    162M    166M
    32       69   1.17M    776K    822K    3.08K   51.3M   32.7M   34.8M
    64       17    522K    355K    368K    1.43K   36.9M   25.1M   26.2M
   128        6    130K      7K   11.2K    1.07K   31.3M   1.50M   2.23M
   256        2      1K      1K   2.48K      833    416K    416K   1.01M
   512        4      2K      2K   4.47K    2.88K   1.44M   1.44M   3.32M
    2K        1     512     512   1.24K    2.79K   1.39M   1.39M   3.46M
 Total    46.2M   5.73T   5.70T   5.70T    46.7M   5.78T   5.74T   5.74T

dedup = 1.01, compress = 1.01, copies = 1.00, dedup * compress / copies = 1.01

Saving's of around 80gb with dedupe and compression (backup box so no real world performance requirement) is just not worth the need for 3-n times the ram and possibly an ssd for the l2arc cache to speed things up. Yep, the suggestion and observed behaviour was to hook up a cheap small (30gb) SSD for cache to accelerate it. I don't mind that so much for a primary but this is my backup/2nd copy box so it's not really ideal. Certainly not for 80gb of savings, or at current prices around $5 of disk.

My second attempt is now underway, this time I've sliced up my data sets into more volumes, and by more that means smaller average size, so this time around 2TB max per volume, which from experience at work I've learned is a good rule of thumb. So now I can enable compress+dedupe on only specific bits, hopefully where the most savings is to be made, and then the rest is just stored raw. This way the savings might be similar, but without the major write speed penalty. I've also realised for the production box if I want screaming performance, I'll throw an ssd on there, but that means more sata ports, which means a major change. I also need to work on power management too.

One thing that has gone right this time, is I'm now using CF->IDE adaptors and booting off that. This way the OS think's it's on a 2gb hdd, so booting doesn't have the complexity of usb boot and also uses less power and doesn't take up a sata port. Of course new boards don't have pata anymore so I might need to get a CF->sata one in future.

Another thing that must be said, Solaris's CIFS server is fast.

Copyright © 2001-2017 Robert Harrison. Powered by hampsters on a wheel. RSS.