Ok, so this was a bit of a work in progress. Basically a collection of screwups lead to an entertaining side effect, and then as these things do, swings and round-a-abouts later bit me in the ass.
So after the great Perth storm, I had no power for 2 days but when it came back I basically had free quota (after a power cycle to reconnect at a decent sync speed.) Not being one to jump to conclusions I initially just assumed it was temporary and would correct itself with the whole backlog of usage.
After a few days and finding no one else had the issue I decided to hammer it. And hammer it I did, to the tune of at least 1.25TB over 10 days. I say at least because since I stopped bothering with traffic accounting at home it's really hard to say. My Usenet account shows some pretty amazing stats, but that wasn't the only source. Roughly 120GB or so per day for about 10 days later, and including lots of PSN content and other stuff. All my Linux and Solaris iso's are up to date and so on.
Due to this, my forecasted storage upgrade had to be brought forward a month or two, and with my 1.1TB snapshot which I still couldn't delete (zfs dedupe related bug) I had to build a whole new pool. Sigh. So after some browsing all the local sites for the cheapest 2TB options, and emailing my usual drive provider who didn't have the model I wanted (unfortunately) I bought some from a store I don't usually frequent. I had to pay cash as the reason I don't frequent them is a stupid 5% surcharge for using common modern technology (Credit cards). Also I didn't really want such a large purchase attributed to me by name. I know the types of people who work in these places, and I'd rather they don't spread that information around. Also funny was the receipt had no serial numbers listed. Also adding to my dodgy opinion of such store.
So after mounting the new disks in my backup filer I did some initial disk testing to make sure the disks were ok (dodgy store etc) and then commenced the snapshot sync. 2 days later it was done, and it had 250gb more to sync. Whee. Once it was all up to date I had to swap the disks between the 2 cases. Only then was I able to create the new backup pool (bigger disks now) and sync back the other way. Fortunately ZFS makes it very simple to move stuff around with snapshots and send/recv. There is no other way to handle datasets of this size as nicely. Though when btrfs matures, it could become my choice due to being on Linux (only time will tell there, ZFS still has a pile of odd quirks). So I've now got an upgraded storage subsystem, still on Solaris/ZFS and still with a full mirror on the backup box.
Now for the first karma bit.
I knew this error couldn't continue for ever, and after a while you do run out of things to download. There were day long gaps in there before I thought, "why don't I get that". Ohh 250gb thats too big. Click. It was actually quite draining (emotionally more than sleep time). The thought of unlimited and then not being able to use it. Of course this only comes from living in a quota'd world, if you never had quotas and could get anything when you wanted it, you wouldn't store anything either, where as here you store everything and max out your quota every month because you use it or lose it. Ok so when my connection dropped and it reconnected, usage started counting... obviously that would be in the middle of the night, so right around 1am. By the time I woke up at 6am to stop it, I'd done 24gb of my 30gb quota. Whoops. Adding to that, of the 250gb queued, I had just over half, and being mildly obsessive compulsive, incomplete things are not good. So now I had enough to not want to chuck it away, but not enough to call it quits. Alas a dilemma.
Interesting sidebar: The connection history log showed "Admin Reset" rather than "Normal Disconnect" so I know it was knocked off deliberately.
I basically did nothing for the next 2 weeks until nearly the end of month. So having stuff queued up and a need to complete that, I proceeded to follow my usual trick. (Details not posted laa laa laa). So when I upgraded my plan, I clicked submit and the progress thing came up, and then nothing. Basically what happened is they had a major fault and the upgrade didn't work (no details posted on the fault beyond a thread on whingepuddle "Major service disruption", but it took out all their corporate sites, mail, streaming radio and lots more). The following morning the fault was fixed (according to the post on whingepuddle and the outage notification site) yet my plan change was not effective, so I did it again. Ah ha, now it all goes pear shaped.
Karma bite number 2.
Plan change claimed to take effect, but by 8am I was shaped. How could that be, not only should my off peak now end at 9am, I had plenty of quota unused. Disconnect, reconnect, fixed. Whatever, go to work. Few hours later I check on my queue. Hmmm, shaped again. Reconnecting didn't help this time, member tools of course show heaps of quota free and of course no shaping. Not being one to call the call centre (see rant below) I asked around why this might be and asked for my connection to be booted from their end to see if that fixed it (since the tool to do this in member tools was not loading). It did for a bit, until I got home at least.
When I got home, being the constant tinkerer I am, I noticed my transfer speed was going slower than it should, so I figured now was as good a time as any to swap modems (from a loaner to my new one). So I swapped them over, bit of down time aside for obvious reasons, and it reconnects probably 10 mins later. Shaped again. Assholes.
In many ways dealing with ISPs are like dealing with the government or simply the man. You can't talk to anyone who can really change anything, but you can talk to someone who might be able to pass a message along to someone who can influence something. That something might be related or not, but it usually can't make things any worse. So being generally annoyed and feeling that bite digging in I was all meh and decided to play some Rock Band and then watch some stuff before doing anything. Eventually I emailed in the fault, under the assumption (rightly or wrongly) that it might be fixed by a) Midnight, b) 1am, c) 2am without any involvement anyway. I prefer to think of things like my problems as non critical, so not to bother the call centre with it. But it's worth getting it logged because if it escalates you at least have that to fall back on when you've had no net for 2 weeks and they claim to just hear about it.
So I snoozed and at midnight it was still shaped, at 1am it was still shaped and sometime between then and 4am when I woke up it was unshaped. Kicked things off again, more snoozing, shaped again. Reconnect, all ok again. Hmmm. I noticed my email had been logged at 3am as an issue, so maybe it was "fixed" then. After talking myself into calling the call centre (quite a feat) to add to the issue that it was up and down again I called up. Informed of a 6 minute wait, logged a callback. While waiting for the callback I got shaped again.
Call centre rant.
I was prepared for having to follow the script, because to get anywhere you need to fit into a pre-defined customer mould otherwise the person on the other end, lets call him/her Sam (not the real name in this case) would not be able to help you one bit and could even get timid or scared from the unfamiliar situation. Sam was very caring and read the notes from the email the night before, and then I explained that it was pretty much the same thing and I wanted to add that it had continued on. Of course that’s not one of Sam's scripts, so we went down the power cycle modem, what modem, check cabling, username, password path, swapped back to the ISP provided (crap) modem. He/She couldn't quite get the modem-router two device idea, so I quickly glossed over that. I knew the problem was not my end - after all I could still drag data down off the ISP's ftp at full speed. Sam did port resets and so on, eventually they wanted to try my account on their test bench, which they did and reported (after 20mins on hold) that yes it appeared that I was shaped. Great, 50mins on the phone after a 10min wait to be told what I'd emailed in the night before. Excited voice: It was even ok to plug my modem-router back in ... they got me to unplug that while they tested my account.
A few hours later I got a call saying it was fixed and to power cycle my modem (grin). Of course being at work I was unable to do that, but was able to reboot the connection remotely, which I did and things were ok again. Fired up the queue again. By this stage my second chance at off peak was long gone and my 50gb of planned usage was wasted. Sigh. Also of interest; the issue was closed before confirmation was received. That was the end of Sam, bless his/her heart. Of course my stories never end that well. Sam's caring voice managed to reassure me that not only do they do an excellent job and offer great products at competitive prices; they just aren't always able to get things going smoothly without some further tweaking. Perhaps in this case treating the symptoms rather than the cause. About 20 minutes later another new problem started.
More karma?
Now the connection was dieing every 20mins or so. The logs from my router suggested the connection was just falling over and then a reconnection was failing causing further delay before it reconnected successfully. A friend of mine who I was chatting to on msn thought this was quite funny after a few hours but also realised that even though the NOC had been involved, they wouldn't know what to do to fix it and realised that the issue would continue beyond the quota month until fixed. He managed to pass a message of this situation to someone still at ii (clearly they don’t let all their good staff leave) and still in a position to be able to look at and fix the issue. From what we've been able to figure out, the traffic accounting system was not able to identify which account to assign the traffic counts emitted from my connection to, so it killed the connection. This process happens periodically so that explains the delay. The duplicate account in LDAP was probably caused by the failed plan change when their "major service disruption" occurred. Which is funny because databases have had ACID commits for years, perhaps the plan change developers were on acid while working. As soon as this error was fixed, the connection stayed up without issue, no power cycle needed.
The to add salt to the wound, while shopping around for prices on some other hardware, I spot the 2TB disks of the model I want are now available from my preferred supplier, for $50 cheaper (per drive) than I paid 3 weeks prior. That wouldn't be such a problem if I'd paid with credit card, but alas I did not so no opportunity to claim the price drop back. The irony here is when I upgraded to 1TB disks nearly 3 years ago they too dropped 20% within 2 weeks of my purchase. So to recap, if it wasn't for the start of month freebie, I might have saved myself nearly $500. Karma baby.
So what did I learn?
Against my best judgement I gave the call centre a go, only to be let down again for multiple reasons. Pity I didn't get sent the customer feedback survey thing, because the guy/girl did everything expected of them. No more, no less and would get top marks from me in nearly every category. Unfortunately they were not able to identify let alone fix the root cause due to it not being in their script book.
It's never totally safe to change your plan.
Taking advantage of free quota is perfectly fine and encouraged, because you'll get screwed over in the long run anyway, may as well lift the average.
Next time I invest in new drives, wait 4 weeks no matter what. (Though that could just infuriate me more if it then drops within 3 weeks of that again.)
Next post coming alot sooner than this one did, it's all about TV and stuff.