It's been a couple months since our last MD development update. We've achieved some progress during this time!

Active Cooling Package / Module

We made first active cooling module for a cluster. This has not been needed prior, but in anticipation of higher power consumptions we wanted to prototype one already.
Found immediately ways to make it better, but already performs fully as expected. No long term data yet, we'll need a little bit more time for that.

Space restrictions makes it tough to build mechanically strong, but we've achieved that. Some fine tuning and building assembly tools remains to be done for that. In the new datacenter these will be completely obsolete, but in traditional datacenters these are a good extra safety factor.

Dynamic Pricing

We've made enhancements to that. Pricing pressure is now being applied by individual components and capacities as well.
Fine tuning for more stable, yet allowing faster flings in period of high demand. It also collects and displays some extra pricing data for admin observations, this data allows us to better choose what kind of nodes to build.

Dual NVMe Models

Dual NVMe models were introduced. These are important for those looking a little bit of added redundancy or added performance.

Power Meter Data

This has been enlightening and showed some future potential development paths to go even further with efficiency, there's a low hanging fruit of few % additional efficiency to be gained here. Every % at these scales become important, just 1% better efficiency can mean additional 100 servers.

We estimate there could be potentially as much as 8% additional efficiency to be gained here working on more efficient power delivery. Experiments are being planned, but development will take time -- the planned development path is rack scale changes. Ironically, this is because our system is already so efficient, which leads to inefficiency in power delivery. This wouldn't have been easy to notice without the power meters.

We also need more data before ensuing in that development path.

Better Networking, Density and Cost improvements via Networking

We started exploring one of the development paths for networking, and found out we can both increase density, lower TCO significantly directly and indirectly. We've done some mockups, and checking out vendors. We expect first units to enter production during summer.

This path also opens the door for 2.5Gbps units.

Testing units for 6x2.5" in-hand; Looks like a good match

We have some testing units for 6x2.5" setups in hand, and at quick mockup seems to be rather trivial integration. We might have some units in production by end of Q3/24.

Automation, control panel and distros

Internally we have a level of automation already working, but it needs a lot of polish and only works for single drive setups so far. It's mostly just software and hours upon hours to be spent in testing and qualification. The good news is that all it seems to require is just hard work and we can start rolling out the capabilities.

It also completely changes the processes of upbringing nodes into production, which causes it's own issues, delays.

If you want to beta test and have single drive model; Contact support. There will be caveats and will require you to have prepaid your server for at least 6 months due to the manual work entailed. For now for single unit nodes we can change distro for you (completely unoptimized!), but later (few weeks in future) on we can start providing the self service control panel as well. We'll only accept a few beta testers, and feedback is to be expected.

Standard Rollout will start by moving a certain single MD model at a time to the automation system.

Tooling

Tooling updates and enhancements found again, making things faster and easier to build. Manufacturing is difficult, and we expect finding ways to be more efficient to just continue the more we build these. On small scale you'd never find these efficiencies, neither would probably be worth the effort.

Every tooling update and enhancement has it's own ROI, in terms of money and time spent.

Revolutionized VM Seedbox Performance: Comprehensive Stability and Speed Enhancements Achieved

We are excited to announce a major breakthrough in our new VM seedbox services. After extensive testing, we've successfully eliminated all instances of crashes. This achievement marks a significant improvement in the stability and performance of our VM-based seedboxes. Initially targeting the most problematic nodes, we've implemented systematic changes across all hosts, enhancing stability under high I/O loads. Our automated script streamlines this process, ensuring all participating guests benefit without the need for manual intervention.

Detailed Issue Analysis and Effective Solutions

The root of the stability issues traced back to an old problem with pre-emptive kernels interrupting I/O processes, exacerbated by default Proxmox kernel settings not optimized for heavy I/O loads. This problem became apparent only under extreme conditions, involving high I/O demands across multiple VMs, compounded by shared SSD caching. With targeted I/O setting adjustments, we've eradicated these stability issues.

Some I/O related setting changes later; These issues are gone.

Performance Positively Impacted

Subsequent performance analysis revealed opportunities for further enhancements. Preliminary tests on select nodes have shown promising results, with some instances experiencing up to 3x I/O performance improvements. While these gains vary, they represent a significant step forward in our ongoing quest for excellence in Seedbox services.

Buried in statistics, however, some other related host I/O settings has also been changed which have shown some performance improvement. When these were implemented in a hurry during the worst of energy price crisis schedulers and read aheads were heavily stacked on top of each other. Each scheduler adds latency. These has been fixed now on host level, and slowly rolling out to the guest level too.

Understanding the Initial Oversight

Our initial reliance on synthetic testing, while thorough, failed to replicate the unpredictable nature of real-world usage. Despite positive aggregate performance data, the nuanced complexities of simultaneous VM operations and diverse request patterns eluded our tests. Synthetic testing can only do so much. Real world seedbox usage is chaotic, very chaotic. The natural chaos of real world was missing, ie. fluctuating queue depths, request sizes, all guests heavily active etc etc.

Further, upon inspecting total statistics, everything looked not just fine, but great. Total absolute performance increased by a statistically significant margin, so much infact that it was obvious to human eyes from plain bandwidth utilization graphs. So aggregate data showed performance improvement.

Instability was impossible to test for, it's like a loose electrical connection in your car; Constantly annoying and causing you issues, but you don't know what it is. The moment you take it to a garage, the issue goes away. It was impossible to reproduce, and there was absolutely no data what-so-ever. All you had was instinct/gut-feeling to go with for diagnosis. Therefore, it takes months and months of time to hunt something like this down.

We're now more equipped than ever to deliver an unparalleled VM seedbox experience, characterized by unwavering stability and enhanced performance. This milestone is a testament to our commitment to continuous improvement and customer satisfaction.

This took multiple days to notice because no errors were not generated, some tickets still got imported etc. It was only noticed after longer time completely no tickets imported and error message was generated that import had not finished fully in a while.

We are still investigating and working on this, it's rather slow because loading a inbox of that size is ... Well slow.

If you opened a support ticket via e-mail in the past few days, please login to client portal and ensure you have that ticket or that you got a reply of ticket being opened.

UPDATE: RESOLVED

If you tried to open ticket OR reply to a ticket by emailing support, please ensure you actually got the ticket opening. No email client could function with the inbox anymore so it had to be 100% cleared sadly.
Some still got imported, some did not get imported. It was impossible to tell if all legit replies / ticket opens were imported.

This email was chosen to be sent to that specific user group.

We use a 3rd party billing system, probably the most common one among hosting companies. We recently updated from a very old version to the newest version. Turns out there's a plethora of new bugs // usability gotchas on old features; This time it was sending an e-mail meant for couple dozen users to everyone.

So this defaulted to just send everyone and anybody the email, which was meant for specific product, only active services, specific domain -> Send it to everyone.
You have to go look for it to see the email sending progress, and to how many. Future we will recheck the recipient numbers on each emailing. Slows significantly down every day work or less emails sent to users of specific servers as a group. We will see.

We are really sorry for having pestered so many users with thing that does not concern them the least.

2xNVMe models by default are RAID1 but can be changed to RAID0.
All of them have i5-8500t CPU, 32 to 64GB of RAM and NVMe drives are 2x1TB or 2x2TB for now.

We are now at 24 different models.

Check the new models out at https://pulsedmedia.com/minidedi-dedicated-servers-finland.php

We've again made some progress! Below a photo on the latest prototype reaching a production rack.
This is hardware rich development, iterate iterate and iterate in-production, each time making something a little bit better. There's an unbelievable number of little nuances in a system like this.

This just entered production last week, yet isn't even our latest model. That's still on the "healing" bench, waiting for some parts. We got the wrong parts accidentally in stock.

New Challenges Await

One of the models refuses to network boot without display present. Easy enough fix, just took a while to get the "dummy plugs".

Some models which have Realtek NIC has random seeming network performance issues with some targets, upload performance on average limited to about 60% what it should be. Easy enough, just need addon NICs when the onboard one is not usable. It seems a bit random which unit has which. Mounts still need to be designed and made before moving into production testing.

Bios Woes, once again. Bios configs for these can be very nuanced. Issues will be fixed once they arise, no sense to take hundreds of nodes out of the rack each time we find something which should be better or fixed, unless that issue rises for the particular node.

Approximately 30% of the MD models produced to date are currently down due to various little nuanced issues, some are having actual hardware issues tho; Namely we received a really bad batch of RAM and NVMe drives, all of these were brand new but with sky high failure rates. Most are config errors and such however, small human errors. Plan is to fix these during February.

We planned to use PC-ABS from PM for electronics enclosures etc. but unless you make solid part this material is pretty much unprintable, it cannot be bridged at all even within 65C chamber. Either you have bridging with no layer adhesion, or you have no bridging with layer adhesion. The sweet spot is way too narrow to make parts which are not mostly solid. We are probably being over cautious with material choices however. Since these do not need high continuous loads or high stiffness, we will be trying another material.

Burn In Testing; We need to automate this and make a process out of it. Some 7-8" displays are now on order to make many tiny burn in + bios config stations, something to work on after February. Does anyone know of a minikeyboard which simply has arrow keys + esc + F5 through F12 keys, and works universally? Let us know!

Progress Made Over The Past Couple Months

1) New units will have new power cabling, which is faster and therefore cheaper to produce AND more flexible with no specific model tie-in anymore. This power cabling takes couple minutes to assemble per unit, instead of the old one averaging to very laborous 20minutes or so per node. Old power cabling cost was about 3.50€ per node, new one costs about 5€ per node BUT saves about 15minutes of labor while providing much neater and nicer end product. This is a huge win!

2) Power Meter + RJ45 panel and also is better mounted now, earlier affixation was a little bit too weak (photo above)

3) Dual NVMe Drive Models will shortly be available

4) Bios updates (fixes some non-bootable units)

5) Automation is starting to get a long, some further tests has been made and first unit installed through new system done. Working on API for power management.

6) Some parts removed from the platform which were deemed unnecessary and potentially even detrimental to final product.

7) TOOLING! Can't stress enough about tooling. New tooling is pretty much operational and in use now. It makes some annoying portions now outright satisfactory.

8) New rack mounts are in stress testing right now. Just proof of concept, but we want to first see how well they hold weight before committing more resources into design.

9) MD Product Page; The little chart next to price shows if the price is going up or down, but if you hover over it you can see last 30D and 7D average for sale prices too.

10) Dynamic pricing is working excellent, EXACTLY as we hoped and planned for.

11) 19th model was just added, with at least 3 new models inbound during February if there are no surprises.

Tooling in action examples

When the marketplace doesn't have what you need? You make it. This time a HVAC ducting adapter which is eccentric and conical, this exact size was not immediately available and conical eccentric sometimes go for as much as 500€ a piece (seriously!) when they should be 50€ a piece. This cost 20€ in materials, 30minutes in design, less than 1 day in print time each. Immediate need was for 2, but later on more as fans tend to die eventually. Especially 315mm sized ducted fans, they have very weak bearings to their size, no matter of manufacturer. Couple of years per fan. Our first 315mm sized ones cost nearly 1000€ a pop, but now we adopted older tech and they are more like 250€ a piece. They last about 2 years each. The bearings can be replaced however if you have hydraulic press, and spend the time to arduously open the stamped shut casing etc. This job is better left for someone who specializes in HVAC fans tho, so we've just been accumulating failed fans over the years.

It is now quick and easy to make all kinds of custom tooling as well to make production faster. Here we see an very weird network cable organizer. Assembling an MD set takes multiple lengths, so varying width slots required.

Jigs and such has saved tremendous amount of work too.

We've got an 800x800x1000mm printer en route to make HVAC intake/exhaust grilles too. While those are very basic in their construction, the large sizes tend to be exorbitantly high price. We calculated a minimum of 8000€ just for a few grilles if bought from the market. CAD Design takes less than 1hour, materials for all those will probably be around the 150-200€ mark, and printing time in the 1 week range. Another ~7000-7500€ saved in costs with just a little bit of effort.

Material woes? PLA is UV safe with just a shallow paint coat, that stops UV rays. Also even if you don't paint, only the surface layer is affected, UV cannot penetrate the whole material. Temperature range is more than sufficient, intake never gets to  65C, and on exhaust side even with our 8M high DC and taking from the ceiling, we won't probably see 65C exceeded, even tho we are hoping the heat stacks up nicely. What if 65C is exceeded? The PLA anneals, and then it can take 150C, however shape will change slightly as the material crystallizes. Surface annealing is both easy and preferrable in post production, this takes mere minutes and is much simpler than most expect; Just take a blowtorch on the darned thing! ;D The extreme temperature (~1300C) not only eliminates the "whiskers" but actually heats up the very surface to beyond 65C causing some crystallization of the polymer structures, therefore surface annealing the part. Once the surface cools down, it can withstand 150C, and once temperatures reach over 65C the surface can keep the internal structure shape better, therefore more thorough annealing without loose of dimensional accuracy / material shrinkage; At least not as easily. Just don't do this at home! It's very easy, not to just damage the part, but in inexperienced hands you might burn your house down. So please don't do what we do, be safe. Use heatgun or more traditional methods. We work in industrial workshop, with multiple fire extinguishers nearby. Due to extreme temperatures etc. it also needs experienced touch not to overheat sections. It's an acquired skill through multitude of failures.

Now, how about PLA being biodegradeable, doesn't it degrade being an outside vent? No. PLA for fast biodegradation requires high temp industrial composting. Namely, exceeding 65C and preferrably anaerobic. Otherwise it takes a really really long time to degrade. It does, but does it matter if it takes 150 years to do so? Not really. Also since these are on open air, not wet, the typical biodegradation processes don't occur.

Dynamic Pricing In Action

Pricing pressure above right now. First line is the global adjuster, amongst all MD units, the totality of them. Each line below is individual model.

Some models are in high demand they go up, some models in low demand so they go down.

This let's You Decide what the server is worth to You. We are really poor judging what our offers are worth to you, we don't know all the use cases, or how things solve your problem and what it is worth to you, or how the landscape of offers have changed, how do you value our unique take on servers, or our network, what our production capability for new units is etc etc. There are simply too many variables at play to anything else than guesstimate the ballpark. So let an algorithm decide.

The factors currently considered are total number of units sold/unsold, when was last unit sold, what's the average 30day offer price, average price sold at etc.
We will probably add more variables as time goes on, but for now this works really well. We need to finetune the algo now and then, for example we've been out of 4TB models for a long while now, once we got even 1 we decided to manually nudge the price higher, but the algo lowered it back down way too fast via the "Draw to the mean" portion. We had to lower it's power level to allow more flexibility in pricing, and still this resulted in too low price compared to demand :)

There was also a bug which made new model price 0€ because there was no price history :)

Why Power Meters on every set?

Curiosity! First and foremost. But in reality? This is valuable technician tool, you can at quick glance see if there's something obviously wrong, and during final setup the power consumption shows if all nodes booted up right. These show a lot of data, including temperature and total kWh consumed in the meter's lifetime, power factor, voltage etc.

Further, we get manual verification of a sets average power draw over time. This helps us more precisely calculate the Operating Costs (OpEx) of each model of units, which allows us to set lower starting pricing eventually! Once we reach equilibrium of producing more servers than we sell, having precise OpEx data is crucial to set the lowest possible starting price for a new model.

A lot of operators just goes like "350W PSU == 350W Consumption" or "180W TDP CPU + 4x3.5" == ~220W consumption" -- neither is even remotely correct, not anywhere near.

For example, we just noticed a Ryzen 3900X with 6x3.5", 4x DIMM Modules only consuming ~110W in production despite 105W TDP CPU and 6x3.5" -- Meanwhile a 65W TDP Xeon with 4x 3.5" consumes 180W !! Both are 1RU.

We would prefer to collect this data automatically into a database, but the power meters while they have both MCU & serial interface on the MCU -> This serial interface is not exposed. Maybe we will spend the time to see if the serial interface outputs data, but we don't expect manufacturer to disclose any information nor be willing to make custom firmware. We asked for customized version where this port is more easily exposed (other than less than 1mm diameter pads for pogo pins on the PCB) and the MOQ was 10 000 units.

Besides, someone will eventually make that version regardless, it's way too obvious of an value add for so little cost! Only a little bit of design time + extra port / pins on the board.

Also these power meters we got for a really fair price at quantities. Adding one does add a little bit of assembly time, and does take a little bit of power (tiny fraction of 1W, the screen is BRIGHT), and does cost a little bit of money and space on the platform, we think the tradeoff is more than worth it. All it takes is a few times saving technician time on troubleshooting.

Plus it looks cool! ;)

This is constant, and ever lasting. This month it's been many more or less significant costs being increased.
Some wrap it as "Good News! We only raised price by 4.5% since that's the official inflation figure!". 4.5% annual price increase = 55.296% over 10 years. That is not small.

Electrical prices continue to be high as well, the 2023 annual average electrical price was somewhere around 300% of what it was previously.
YES, there were a lot of days with even negative spot pricing; But this does not come all the way to us, we do not own our DC building nor the transformers there in, we merely lease the space and we are at the mercy of the buying skill of the real estate corporation (bank) which owns the actual building. They seem to be abysmally bad at this, when the prices started skyrocketing turns out they had no upfront plans, they had no protections in place, and were solely on spot pricing 100% -- Then they made arrangements to protect 40% when the prices peaked at their highest, for not sure how long.
Not only that, this corporation is also bent on "Saving The Planet Earth", only rent increases come in paper for example "because of Co2!" (I would argue regular print paper might be net negative Co2. Is Co2 actually an issue? Another topic for another date!); They spend more effort on recycling programs for the building than actually maintaining the building, well it seems like that

So what they do when prices are the highest? Move to "100% Green Energy!" of course!

Net result is that our electrical price lowest month was still ~+100% of what it was previously, typical month is more like +200% with much higher peaks. No, This is NOT A JOKE.
This despite the total annual average pool spot (energy exchange) pricing was not that insane last year.

Despite this, we still have the situation better than many others. Some have made 100% lock-in price for years to come at prices 4 to 8x what they normally are. Imagine those people and businesses who locked in for 50-60cnt/kWh PLUS Grid Fees and Taxes! There are large office buildings right now at that situation, no one wants to rent or buy from those buildings, and companies are moving out of the building because management did something that silly as to lock in the highest price in decades.

At this point the only cost that has not increased is transit/bandwidth costs, and select hardware components.
Even weirder; We see that HDD pricing has stagnated to same level as 2020 roughly, new models get released, bigger ones, but per TB cost remains the same year after year. It's like the price was fixed, remains within few percentage Year Over Year!

What Does This Mean To You As Customer?

Not much at this time, we have no plans for immediate price increases. New service order prices has been increased, and much to our surprise that has not affected new service sales much.

However, we need to set much larger growth goals in revenue growth to retain the same profit margins, AKA healthy business. It does not help that Seedboxes has been a dying niche for a decade now, and to this, we are now expecting a pivotal moment this year with the MD Platform; Projections show that if successful; This year marks the point we have pivoted mainly as dedicated server provider instead of seedbox provider.

Our current revenue is approximately 23.50% dedicated servers, out of which MD beta testing and development units already accounts for 40%. Projection shows that we will increase quantity of MD nodes by 5x by end of the year at this pace. There's been a small hiatus on new nodes as we've worked on tooling, testing, and fixing issues. All of that background work enables us to work much more efficiently in the future, paying back dividends for decades to come. We are now at a point bringing up a new MD node takes significantly less than 1 human hour of labor, and we are now starting to analyze operational effort required (repairs, replacements, fixes) soon as the platform is starting to near feature completeness.

The pivotal moment is when from our revenue is 50.01% or more in dedicated servers. We expect to reach this point by end of this year. It might take until the very last day of the year, but we are expecting to reach. We shall see how that goes.

Subscriptions are back! You can now make new subscriptions for pre-existing services as well.

This has direct Credit/Debit checkout option as well. You can pay with card without signing up to paypal.

This should streamline your overall payment experience when using Paypal.

The Saga Continues
Just when we thought our maintenance story was coming to an epic conclusion, it turns out the plot is thicker than Finnish rye bread. It seems the forces that be have decided we need a bit more suspense in our lives. And who are we to argue with destiny?
In the latest episode of 'Maintenance Chronicles,' it's not the missing gear this time but a water pumping station snag that's keeping us on our toes. Who needs TV dramas when you have real-life data center maintenance sagas? Contractor had not realized it needs to be powered on at all and not preparated for that; Otherwise the building might flood during the maintenance period.

Our Apologies
We understand this rollercoaster of maintenance dates has been a bit like trying to catch a greased-up server in a server lab. It's been tough for us too, not knowing when the big switch-off will happen. Please accept our sincerest apologies for the constant edge-of-the-seat experience.

Black Friday Silver Lining
But hey, every cloud has a silver lining, right? Our team, ever-ready for data center marathons, will now pivot to crafting mind-blowing Black Friday deals. Think of it as our 'plan B' turned into 'plan Awesome'. With this postponement, we can finally shift our focus to the exciting world of Black Friday deals! Yes, we've been holding off on those juicy specials, fearing that our team might be too busy dealing with post-maintenance gremlins. But now, it's all systems go!

What's Next?
While we don't have a new date for the maintenance yet, we promise to keep you updated with all the details as soon as we have them. Until then, let's enjoy the uninterrupted uptime and get ready for some amazing Black Friday treats.

Our servers, currently enjoying their uninterrupted dreams, are blissfully unaware of the maintenance drama. Let's keep their slumber peaceful a little longer, shall we?




As we navigate these maintenance twists and turns, we're more committed than ever to ensuring your digital experience is as smooth as a freshly debugged code.  We're all in this together, and we're committed to keeping your data flowing smoothly, come rain, shine, or maintenance delays. Thanks for being part of our journey and for your understanding as we tackle through this unpredictable adventure..

Got questions or need a digital shoulder to lean on? Our support team is charged up and ready to jump into action faster than a server reboot.

Energetically Yours,
 Aleksi
 Chief Energy Officer, Pulsed Media Team

🚫 Breaking News: Maintenance Cancelled 🚫

Okay, so here's the tea ☕: The maintenance is officially cancelled. Why, you ask? Well, equipment supplier kinda dropped the ball and didn't deliver some of the crucial gear that was needed. Bummer, right?

What's Next? 🤔

We take this occurrence seriously and are committed to implementing additional measures to prevent such instances in the future. At Pulsed Media, we continuously strive to uphold the highest standards of service and reliability. We value the trust you place in us and are dedicated to ensuring the stability and integrity of our services.

For real-time updates and further details, please refer to our network status page: https://pulsedmedia.com/clients/serverstatus.php

We apologize for any inconvenience caused and thank you for your continued support and understanding. Our team is available for any further queries or concerns you may have.

With any Python application this is the reality, they keep breaking at random times, sometimes even daily. That's why updates has to be avoided at times.
There are no real packaging solutions neither which would be convenient to deploy and contain 100% of dependancies, 100% of the time. There are some (virtualenv and derivatives) but they too have their own set of issues.

Containers would be an bullet proof solution, but unfortunately that could drop some performance and for seedbox that is not very acceptable trade off for some.


ETA for fix is right now unknown.

Affected number of servers seems to be limited as well.

We want to share a recent incident that highlights the importance of our community in helping us provide the best service possible.

One of our users, S.Y., recently discovered that he still had access to a previous 10G seedbox after his service period had ended. Instead of taking advantage of the situation, S.Y. promptly reported the issue to us.

This act of honesty not only helped us identify a potential issue in our system, but also reinforced the trust and integrity that forms the backbone of our community at Pulsed Media.

As a token of our appreciation, we rewarded S.Y. with a ~1month service credit. We believe in acknowledging and rewarding honesty, and this incident is a perfect example of that.

So, here's a big shout out to S.Y. and all our users who help us improve our services every day. Remember, your honesty and feedback are invaluable to us. Let's continue to make Pulsed Media a great place to be!

Stay tuned for more updates and improvements. As always, if you have any questions or feedback, feel free to reach out to us.

Happy seeding!

• 1RU Form factor
• Node scalability: Up to 16 nodes per 1RU, 672 nodes per 42U Rack
• Power scalability: Up to 1000W per 1RU, 42KW Per Rack. PSU Options: 350W, 500W, 700W (Redundant), 1000W (Redudant)
• Industrial Mean Well PSUs with High Power Efficiency of up to 94%
• Built-in managed network switching, 10G Uplink per 8x1Gbit nodes (Upgradeable once new managed small form factor switches come available)
• Power management per individual node, 0.5W standby power consumption, max 3.84W
• Designed for "consumer grade" hardware without traditional BMC/IPMI features.
• Supports efficient cooling for standard mini-ITX platforms, height can be increased conveniently for high power draw CPUs with big coolers (No definite limit on height)
• Current version supports standard 100x100 VESA Mount and Mini-ITX mounting, future version will incorporate on as needed basis 4x4"/NUC, SBC mounts etc.
• Designed for rack and datacenter level super efficient cooling, bringing cooling costs significantly down. BIG Fans, BIG Efficiency.

• OpenVPN Support is back
• Docker Rootless
• WireGuard (via Docker Rootless) + Tons of other. Most LinuxServer.io containers should work.
• Performance enhancements
• Stability Enhancements

Here are some tips to help you avoid issues related to Docker containers and linuxserver.io images:

1. If you need to perform tasks inside the container with fake root privileges and UID mappings, you can use the rootlesskit bash command.
2. For linuxserver.io images, set the PUID and PGID environment variables to 0. This should give you the expected UID on the host. For example: PUID=0 and PGID=0.

Using PUID + PGID set, you can prevent potential issues with Docker containers and linuxserver.io images changing the ownership of your directories so you cannot access them anymore from outside the container. As always, please exercise caution when making custom configurations and seek assistance if required.

Once upon a time in the land of Pulsed Media, some clever soul somewhere decided to assign one of our /24 IP ranges to Moscow, Russia. Little did they know that this seemingly innocent act would set off a chain reaction of kumbaya-singing, virtue-signaling censorship!

You see, ever since the war between Russia and Ukraine started in February 2022 people have demonized Russia, all the people of Russia. Every single one of them. Now some overzealous users of Cloudflare thought they'd single-handedly save the world by blocking everything from Russia. They're like internet superheroes, right? Wrong. Instead, they're unintentionally making the world a worse place by limiting access to information and playing right into the hands of those who want to control it.

Well, we've fixed the GeoIP issue, but Cloudflare's still lagging behind with their updates. Their support team can be harder to reach than the remote corners of Siberia.

To make sure this doesn't happen again, we've taken steps to secure our GeoIP data tighter than the Kremlin's secret stash of vodka. But honestly, we were stunned at how easy it was to fake this information in the first place.

As defenders of freedom of speech, we believe that access to information is the foundation of democracy. And if we want to fight the good fight, we should be getting more info to the people of Russia, not less. So here's a friendly reminder: censorship only hurts the good guys, and it's easy enough for the bad guys to bypass.

It's crucial to ensure everyone's access to information, especially in times of conflict, because an informed citizenry can challenge and resist the narratives that fuel senseless wars and loss of life. By providing alternative perspectives and promoting open discussions, we can create a space for empathy, understanding, and diplomacy to flourish. This, in turn, can lead to the de-escalation of tensions and help bring an end to tragic conflicts quicker. The power of free speech and uncensored information is not just a democratic ideal but also a practical tool for fostering peace and harmony in global community.

In conclusion, dear customers, stay vigilant, stay informed, and remember to always question the status quo. After all, the internet is a wild place, and you never know when you might accidentally become a Russian agent!

Stay awesome and uncensored.
- Your Pulsed Media Team

Thanks to Docker integration, our high-end seedboxes enable you to run almost anything, including a remote desktop, with ease. Whether you are a gamer looking to set up a Valheim server or a developer needing a proxy servers for SEO, Pulsed Media seedboxes have got you covered.

Don't miss out on this fantastic opportunity to try our free seedbox and experience the benefits of a seedbox trial without any commitment. Sign up today, and unlock the limitless possibilities that Pulsed Media seedboxes can offer. Explore the world of torrent servers, dediseedboxes, and more – all at no cost for 14 days!

Check the V10G Seedbox series out: https://pulsedmedia.com/value10g-seedbox.php
Or for higher performance SSD 1Gbps Seedboxes: https://pulsedmedia.com/m1000-ssd-seedbox.php

Standard features include things such as:

• Deluge
• qBittorrent
• Rclone
• Docker
• rtorrent / rutorrent
• autodl
• wireguard
• Jellyfin
• *ARR, ie. Sonarr, Prowlarr etc.

curl https://pulsedmedia.com/remote/pkg/arr_installation.txt | bash

This combined with our support for Docker rootless makes Pulsed Media seedboxes a true powerhouse of flexibility with little effort. If you're interested in learning more about installing Docker rootless and Wireguard VPN, check out https://pulsedmedia.com/clients/announcements.php?id=587.

We are happy to inform that these features are enabled for ALL of our seedboxes, no matter which price level. However, we highly recommend that you experience the full potential of these features by upgrading to one of our higher-end services, such as Dragon-R or M1000 SSD.

One of these issues is the occasional and unexpected halting of some of our servers. These are complete freezing of the system, with no hardware issues or errors reported. A reset is typically able to fix the problem, but we are working to find the root cause of the issue.

In some cases, the servers may show high CPU usage, but not always. A common characteristic of the issue is that no input of any kind is accepted on the console, and network and disk I/O throughput goes to zero. Sometimes, I/O metrics show high numbers just prior to the halt, but not always. There is no discernible pattern in terms of usage levels of the affected servers, with some experiencing high loads and others having low loads.

We are currently monitoring the issue and looking for information from others who may have experienced similar issues with similar configurations. We understand that this is a frustrating issue for our clients, and we want to assure you that we are working diligently to resolve it as quickly as possible.

We want to stress that it is much better to solve this issue by finding the root cause and fixing it, instead of relying on a watchdog to reboot the server. Each time a reboot is required there is a small chance of data corruption and other issues.

We apologize for any inconvenience this may have caused and we will keep you updated on the progress of our investigations. If you have any further questions or concerns, please don't hesitate to reach out to our support team.

Thank you for your continued support and patience.

You can now add up to 1000€ with a single invoice and the maximum balance for credits is now 5000€. This will allow you to easily automate your invoice payments upfront and ensure that your services remain uninterrupted.

We encourage you to take advantage of this feature by visiting our client area at https://pulsedmedia.com/clients/clientarea.php?action=addfunds and adding funds to your account today.

As always, please don't hesitate to reach out to us if you have any questions or concerns. We appreciate your continued support and look forward to serving your hosting needs in the future.

As a reminder, if you ordered during the promotional period and are from Europe, you are eligible to claim your Give Away prize. This includes T-Shirts, HDDs, and other IT hardware, as well as an AMD Radeon RX GPU.

Don't forget to visit https://blog.pulsedmedia.com/2022/12/happy-holidays-2022-and-a-look-into-present-and-future/ for full information on how to claim your prize.

Thank you for your business and we hope you enjoy your Give Away prize!

At the same time general software update + new kernel for enhanced performance.

Pulsed Media is not immune to market conditions, and the market conditions have dramatically changed over the past year. Energy prices started to heavily increase a year ago, but are now wildly changing week by week. Market pricing indicates a significant multiplication to electricity pricing with some people already receiving upto 10x the previous rates and 5x has become common place.

Our electricity price has already nearly doubled and we are afraid it could more than quadruple from that, to potentially 8x or more from the price we've enjoyed for the past 6 years in our current datacenter. Electricity price is, and has been one of the larger costs, but now it is already the largest. Some people are already receiving quadruple electricity price of what is our current already increased rate, and the best deals you can find right now are way more than quadruple of our original electricity cost.

There has also been discussions about potential power outages during the upcoming winter. How long and to what level, if any is yet to be seen. Currently Finland is on electricity deficit and has to rely on imported electricity, so a period without wind and extreme cold we could see power outages. Our datacenter is in Helsinki and next to a power plant, but even that will not guarantee we will not see power outages.

Electricity has always been a big part of our cost structure, and competitive electricity price and efficient cooling has meant that we have leaned very heavily on that advantage we've had in the past. Things have now changed with the energy crisis, and we are looking at very high electrical costs.

Our electricity is being bought in bulk, in combination with many other companies and for the long term, but even that manner is not immune to the market conditions -- typically this way of buying is a little bit more expensive than exchange market spot electrical rates -- but was stable and predictable. On the other hand, when electricity prices start going down our cost goes down slower than the overall market. Current public offers for electricity are more than 4x we used to pay already, with some companies charging even vastly more. Over 0.50€/kWh priced offers are increasingly common by the week.

Europe is not alone being affected by the energy crisis. The whole global economy is affected, albeit to a lesser degree than Europe.

Inflation, exchange rates and supply shortages

Inflation is very high as well, all operational expenses (OpEx) have increased over the past 2 years. Some costs have increased dramatically, not just energy costs. Official numbers hover around the 9% mark, Euribor 12month interest rate is going higher at a very rampant pace.

USD-EUR Exchange rate has dramatically changed this year, meaning that all hardware costs now roughly 15% more. Every service with USD basis in pricing costs that much more now. The good news is that those with USD as their billing currency are essentially now getting a 15% discount.

Finally we have supply shortages for some components, some things simply are not available at all, at any cost. In some cases we can find those parts for highly inflated prices, more than 2x the typical. Sometimes you hope to even find a "scalper" with the product you need in stock.

Our target market of low end, extremely good value for our customers has necessitated very precise calculations of all costs. A very precise pricing structure to be viable and profitable business for future expansion and to create ever better value proposition services year over year.

Some of the cost increase highlights over the past 2 years

* Electricity +~89% Estimate for Q4/22-Q1/23 is 400-800%
* Staffing +~50%
* Networking/Transit: +~23%
* Fuel costs: +~60 - 100% (fluctuates heavily)
* Buildings/Spaces: +~90%
* Accounting/Admin overhead: +~75%
* Hardware costs due to currency exchange rates: +~15%

We have managed to find some efficiencies as well which has boosted revenue without accompanying cost increases, but these things are not able to counteract even for all the inflation alone, nevermind energy crisis combined.



Price increases for services

Sadly, due to all these factors we have to increase pricing all around, for new and pre-existing customers. These will come in effect immediately or within a few a days. New pricing will be in effect on renewals and new orders, no mid payment term change. If you have an invoice already open for renewal that will be unaffected. New signup pricing takes a little bit longer to change as we have to re-do all the expense maths for all of the service ranges, these will happen by end of September, or with rough estimates before that.

Relative to revenue most expensive accounts are the smallest ones. The sub 5€ group of services represent about ~31% of services, but only ~10% of revenue, all the while being the source of about 80% of uncommon events. Seedboxes are highly dependant on the number of users per performance domain (ie. a server, or raid array) catering to this group actually increases production costs as well. On the other hand, the higher end service group is the complete opposite, therefore we will not increase pricing for that group as much but actually significantly less than actual increase in the operational costs (OpEx).
Therefore the pricing increases will be linearly adjusted based on service price 35-15% on 10-30€ services -- lower percentages on higher cost service, but also another 3.5€ for all sub 5.50€ accounts which are the most sensitive to OpEx changes.

The final pricing increases will be 35% to 15% scaling from 10€ to 30€ per month. Examples:

* 10€ current price, +35%, final price from 10€: 13.50€
* 20€ current price, +25%, final price from 20€: 25.00€
* 30€ or above current price, +15%, final price from 30€: 34.50€

This price increase will go to cover the increased operational costs.



Future plans and energy crisis mitigation options for the long term

Meanwhile we are starting a program to really start heavily lowering our electrical consumption. Several plans are already being executed, and we will get outside funding to achieve these goals if we have to. Pulsed Media is net debt free company currently and has been revenue funded until now.

Some of the plans are still aspirational, so we'll keep it brief. So keep this in mind, that some of these are currently aspirational and it takes time to move things to a new direction. Think it as a freight ship, even if you turn the rudder all the way now it takes a lot of time for the freight ship to even start turning around. Some of these plans has been brewing for more than a year now, so progress has already started on them.

First we got our second datacenter keys in August, once we build this out we are planning to install solar panels and the building owner is interested on buying our waste heat during the winter. This new datacenter is not very large neither, we are expecting roughly 400kw final capacity. Solar panels need some bureacracy, and we are not yet certain the scale we can build, but aspirationally we would like to get to somewhere around 600kw (+50% over the load) solar panel installation over the upcoming years, depending on the net metering we are able to negotiate. This site will have extreme economizer airflow volume capable of cooling more than 200kw @ 22 Delta-C by itself alone during hottest days of the summer. This will take significant time to develop, with multiple contractors involved and will be built out in phases. We expect very small production volume with compute only centric customers to begin during this winter.

Second, and easier to achieve. Extreme cost cutting by upgrading servers. Newer platforms consume less energy, while giving higher performance. No news there at all, but we are measuring in some cases even 110W per 1U server savings in electricity consumption. Secondary, we are looking into advanced storage setups such as effective NVMe caching right now and already have first servers up and running, this would allow us to have bigger performance domains // more users per performance domain. So far the test results have been _very_ promising for both upgrading to newer platforms and NVMe caching.

Newer platform will directly cut electrical consumption, but will still take years to pay off. NVMe/SSD caching would allow us to have more users per performance domain, while increasing performance. HDD performance, especially random performance, does not increase as the capacity increases so only way to increase number of users per HDD is by caching, or smaller performance domains (next item).

We have already started this process, but will likely take years to completely achieve.

Third, we are looking to partially virtualize and use bigger beefier servers split into multiple virtual machines for performance domain isolation. Therefore saving a lot in electrical costs per unit, while increasing performance. These were already on our internal development roadmap since earlier this year, and several very large storage servers are already in order with one of our server suppliers. This ties with the above hardware upgrades to further cut electricity expenses. These will first rollout to the Dragon-R tier of services (20Gbps RAID10).

Fourth, some legacy services -- really old services where storage capacity for end user is small we are looking to upgrade to SSD based service. These will account only for about few percent of our total electricity consumption however. Will also require hardware investments, therefore this is a minor item and will be done at the speed of convenience.

Fifth, we are going to beta test using regular consumer motherboards for regular servers like some very well known large brands in the dedicated server niche. We will begin with 2U-4U size for easier cooling. Currently only consumer motherboards we employ as servers is the ZEN MiniDedi lineup used for M1000 SSD seedboxes. They are absolutely brilliant for the task, but saddly currently out of stock everywhere. This has a big potential for long term performance increases and therefore decreased electricity consumption.

Sixth, UPS upgrade. We use Eaton PowerWare UPS systems currently. These consume a lot of electricity even when idling. Upgrading to modern transformless hybrid inverters could potentially pay themselves off just in the increased efficiency, let alone using lithium batteries instead of lead acid. Lead acid needs to be replaced every few years at great expense, and can be trusted only for approximately 50% of the rated capacity. Further, this should greatly increase reliability. This move would not make sense without huge increases in energy costs. This could also help if power outages happen, but sadly currently the USA to Finland sea freight is rather slow as well so there is no way we can get the lithium batteries imported by this winter. Typical normal UPS has battery packs for mere minutes of operation, and looses some of the extremely limited capacity each time they are drained. Earliest we can do this replacement looks like will be during next summer.

We might be forced to take the legacy type UPS units completely offline during upcoming winter if we get a really bad electricity rate, there is no sense running UPS when it cannot last over a blackout, and consumes huge amounts of energy. Our own experience and other datacenter owners all report that these units tend to cause more downtime than they save from, by a big margin. Hence, we were already considering changing them -- but that does require shutting down some devices for the duration it takes to replace the units.

Finally, we have to change the whole modus operandi based on increased OpEx and the risks associated with current crisis' -- at least until we get our new datacenter running at capacity and with sufficient solar arrays. This means we are going to target services a bit higher end, semi-dedicated and such as this shifts from OpEx centric to capital expenditure (CapEx) centric cost structure. Unfortunately, that means the days of sub-10€ services might be gone for a while for the most part. Or we find new ways to control resource consumption to acceptable level on the entry level packages, which is not a trivial task on seedboxes which are really hungry for those juicy IOPS! (IOPS = Input/Output oPerations per Second, storage performance).

Rest assured we are working hard on finding new efficiencies and keep providing ever better services year after year. Necessity is the mother of invention.



Not all gloom and doom

It's not all gloom and doom; We expect performance to increase on our services gradually as these plans get into motion. As part of the course for the solution to this energy and inflation crisis a lot of old time users will get service upgrades along with the hardware upgrades as well. This has been standard operating procedure, but never advertised or publicly mentioned before, to upgrade old time users after several years threshold to a notch higher grade service as the hardware and/or software (distro) gets updated. We really like our old time users, hence we do that along with the magnificent bonus storage quota. Bonus storage quota is about 23.44% of the actually allocated storage capacity for our users, on average. In other words, average user gets 123.44% of advertised storage capacity. Old time users can have multiple times the original storage capacity.

We also have new service launches coming this Q4/2022, hopefully all the effort needed to cut electricity consumption does not derail or further delay these plans.


You can discuss this at our Discord, link (valid for 7days): https://discord.gg/buTTbezZ

Root cause has been isolated and steps are being taken to ensure this does not happen again.

The issue has now been fixed, and almost all hardware is up and running.


There are a couple of servers still which needs more attention but might take an day or two extra, as well staff is exhausted after getting an middle of night alert.
The servers known with remaining issues are: sentnel, walton, greenwood, dumper

• Increased maximum bonus to 300% from 150%
• Changed bonus % gain per euro paid from 28€ to 50€
• Increased global max limit of bonuses in GiB gain per day
• Extra ~250 000GiB or ~0.25PiB will be added for users over the next week
• Bonus is given if server has at least 3000GiB of actual storage free, not based on allocatable / new user provisionable anymore -- this will ensure much more even distribution

ETA for full resolution 3hours.

UPDATE 23:02 - Resolved, all servers back online. A Rittal PDU had failed. This is a known issue with this model of PDU, their fusing over time simply fails and cannot be reset anymore. Failure rate is low enough however not to outright replace all units in already in use.

Add to that bonus disk storage you may get for service lifetime or euros paid: http://wiki.pulsedmedia.com/index.php/Pulsed_Media_Free_Bonus_Disk_Policy
and then potential volume discounts https://pulsedmedia.com/resellers.php

You might end up with a situation where you are getting upto 20TB for just 11.55€ a month with maximum volume discount. Even without any bonus disk this would be 1.44€/TB/Month.

1. Download the .NET Core binaries: https://dotnet.microsoft.com/download/dotnet/thank-you/runtime-aspnetcore-5.0.4-linux-x64-binaries and extract it to $HOME/dotnet
2. Add it to your Path
3. Download the combined portal version of Jellyfin: https://repo.jellyfin.org/releases/server/portable/stable/
4. Extract Jellyfin and run the jellyfin binary inside.

Run it once, set it up and go to the Networking tab to change the port. Then restart Jellyfin.

No battery backup (currently) in the world can last for 10 hours for a datacenter. So once our batteries ran out, so did the servers.

We have staff on site, and getting well caffeinated for checking that everything gets back online normally. Once we either get power from someone else's generator or our own generator somehow miraculously on the site we will start restoring services as quickly as possible. We estimate that most servers will come back online without a hitch, but there will likely be a few last services lingering until saturday night for full restoration.

**UPDATE 3 Saturday 01:11** We got the word that power might be restored as quickly as just 1-1½ hours.

**UPDATE 4 Saturday 01:59** Partial power restored. Routing restored

**UPDATE 5 Saturday 02:18** All power restored, we are now checking all servers startup as expected. Battery banks are recharging.

**UPDATE 6 Saturday 02:47** Almost all servers back online. A few remain to get back online, less than 50 now.

**UPDATE 7 Saturday 04:36** Every single server should be up and running. Please open a ticket if you are still encountering issues.

During the past year the amount of network capacity being utilized has dramatically increased for various reasons, so despite this upgrade process was already started now roughly 8 months ago we have essentially got our links bottlenecked.
Rest assured, we are doing everything we can to hurry the transit upgrades.

Until that happens we are saddened to say that we must limit Dragon-R and M10G sales due.

We hope to have this situation fixed within couple of weeks.

Servers will run latest SMOS, and have already been optimized for what those particular GPUs are capable of doing. Speed of these systems is 325MH/s +/- 3% generally, some go as high as 337MH/s and lowest current system is 316MH/s. Delivered system will be picked randomly.
We will setup to the rig to your chosen pool + wallet address but standard fee does not include SMOS access for hardware qualification and warranty reasons, as it is quite easy to brick an GPU with full access. Since these are manually setup, please try to keep pool changes to less than once a month.
Each server runs typically an AMD B350 or X370 motherboard with a ryzen CPU, and 1250Watt 80+ Gold or Platinum PSUs, in a good airflow rackmount chassis. A small number might be running a Intel B250 chipset + Celeron/Pentium CPU.

Service includes full hardware replacements and hw support on NBD+5 schedule just like any other dedicated server we offer, but unlike other dedicated servers we will maintain the software, configuration and monitoring for these to ensure maximum uptime for you.

Pricing is ONLY for annual prepayment due to the nature of these machines, but renewal costs less.

Immediate availability for 3 systems, 3 more available next week. RX5600 XT based systems coming available by end of January.

Pricing is as follows currently, but may change even daily due to stock availability for parts and is always based on current market pricing for the hardware. Price includes electricity and SMOS fees, you only pay for the server rental and it includes everything. This is an excellent choice for people who do not want to manage their own hardware and/or datacenter.

6x RX 5700 / RX5700 XT 320MH/s +/- 5% pricing:

• Setup fee: 1000€, Renewal 2355,30€ annually. First payment: 3355,30€
• Setup fee: 2000€, Renewal 2055,30€ annually. First payment: 4055,30€
• Setup fee: 3000€, Renewal 1855,30€ annually. First payment: 4855,30€
• Setup fee: 4000€, Renewal 1655,30€ annually. First payment: 5655,30€
• Setup fee: 5000€, Renewal 1455,30€ annually. First Payment: 6455,30€ -2,5% discount on first year: 6293,92€

One of the owners of Rapid Seedbox is "[PRIVACY PROTECTION]", many of these subnets are owned by "[PRIVACY PROTECTION]". Googling this name yields many interesting abuse db results, with multiple claims of "hacking gmail", ie. probably bruteforcing passwords like on this attack.
Other owner is named only one of the subnets as i can see. [PRIVACY PROTECTION] is named on multiple ones.
Almost none have rDNS set as of few hours ago. Almost all IPs are hosted at Leaseweb, where Rapid Seedbox has their servers.

Giving them the benefit of the doubt we did not immediately release this, it could've been their whole network is compromised. However, if they stopped communicating and their supposedly 15 people full time staff neither responds, while they pride themselves on super fast help support response time -- It smells fishy.

Normal attack like this, done by botnet and compromised systems would not be so blatantly obvious. Ssomeone with access to a lot of subnets and can spin up on them a lot of VMs at will has to be done behind this. Normal attacks are typically globally spread etc. and typically all IPs have set reverse dns, most have major ISP IP ownership etc. But all of these are fake sounding names, mostly linked to this "greatworktogether" group, and few of the subnets with very similar style of information are clearly linked to Rapid Seedbox.

To best of our knowledge at this moment, no customer accounts were compromised.

We would like to be wrong, as we've just had lengthy negotiations if we would supply servers to them. First contact they asked if we would sell Pulsed Media to them, but that was a very firm no. We proceeded to converse if vice-versa was possible but did not find common ground. After which we started looking for opportunities to work together, by us supplying to them servers. That neither proceeded anywhere.

There is total of 5977 suspicious IPs but we have to make sure no legitimate user IPs ended up on that list before releasing this whole list.

All of the relevant and important subnets are on the above list. Here are the /24 subnets:
104.144.10.0/24
104.227.145.0/24
104.227.27.0/24
138.128.29.0/24
138.128.40.0/24
144.168.216.0/24
185.126.66.0/24
185.164.56.0/24
185.99.96.0/24
192.156.217.0/24
192.166.153.0/24
192.186.151.0/24
192.198.103.0/24
192.3.79.0/24
193.23.245.0/24
193.23.253.0/24
193.8.138.0/24
193.8.215.0/24
193.8.231.0/24
193.8.56.0/24
193.8.94.0/24
194.33.29.0/24
195.158.192.0/24
209.127.127.0/24
209.127.146.0/24
2.56.101.0/24
2.59.21.0/24
45.128.24.0/24
45.129.124.0/24
45.130.255.0/24
45.130.60.0/24
45.131.212.0/24
45.131.213.0/24
45.134.187.0/24
45.135.36.0/24
45.136.228.0/24
45.136.231.0/24
45.137.60.0/24
45.137.63.0/24
45.137.80.0/24
45.142.28.0/24
45.145.56.0/24
45.146.89.0/24
45.152.196.0/24
45.152.208.0/24
45.154.244.0/24
45.154.56.0/24
45.86.15.0/24
45.87.248.0/24
45.92.247.0/24
45.94.47.0/24
84.21.188.0/24
85.209.129.0/24
85.209.130.0/24

To block these subnets just add them to your firewall as is, or without /24 and subnet 255.255.255.0

We have not moved to block these IPs on our production servers as there could be actual seedbox end users on these subnets and we would not like to be punishing Rapid Seedbox's customers.

We do not know yet if there is relation to the spam attacks lately as well.

If you have any relevant information, please do contact us.

UPDATE 3: Got a response from Rapid Seedbox very soon after releasing last update. Investigation continues.

UPDATE 4:  Rapid Seedbox says all of these IPs are connected to single customer, and requested personal information for privacy to be removed from earlier update. These names has now been changed to [PRIVACY PROTECTION].

UPDATE 5: Rapid Seedbox response was that 1 customer was behind this, and they have disabled access to pulsedmedia.com on their proxies.  -- A proxy service? That's all the information they gave us.

Further, this attack had some email addresses correct, infact many accounts got failed login attempts from these. How did these attackers have any idea what addresses to try? By random chance it's very difficult to get even 1 hit by randomly trying, nevermind in the hundreds. A lot of them are fairly (think decade old) billing profiles too. That makes us think that the attacker has some seedbox niche email address database to try upon.

In any case, we are still going through the list of these targets, and will reset some people's billing profile passwords to be on the safe side. It's quite a bit of manual work as we do not want to just blanket reset everybody's, but manually look at each of the profiles rather and add them to a list so we may contact them if needed. We want to emphasize that we have no evidence of any compromised account so far. To best our knowledge from looking at the logs, zero accounts were compromised.

Make your own conclusions on this.

Here at Pulsed Media we've always been quite conscious about environment and recycling. A lot of the servers we use are 2nd hand (where it makes sense), giving it another life cycle instead of becoming ewaste - and we try to hang on it as long as we can. We recycle all the cardboard packaging materials as well. We work really hard on optimizing power usage despite enjoying very low cost rate of electricity compared to many other european countries, and cold climate most of the year for cooling efficiency. Some of the cooling capacity we use is also used to heat rest of the building instead of being dumped into the environment.

• rTorrent version 0.9.8 / libtorrent 0.13.8
• ruTorrent 3.9
• OpenVPN: Sometimes routes were not setup after boot for a long time. This has been fixed
• OpenVPN: Cipher updated to AES-256-CBC
• Info tab now has a nice daily chart for traffic use
• Sox and nzbget added by default
• tcpsack vulnerability mitigations (disable sack, firewall small tcp framesizes)

• Resilio sync added (rslsync)
• Sonarr/nzbdrone installing reworked
• BTSync defaults now to 2.2
• Syncthing added
• rclone updated to 1.47

You can also access rutorrent directly by appending rutorrent/ to your access URL.

Wiki

Wiki will be restored soon, we have full backups of that as well, but will require some config etc. and we are hoping the old server comes back online so we can have the refresh database backup of the wiki. Our backup of the wiki database is missing some latest changes to one of the wiki pages.

So Wiki will be restored within 48hours.

Final words

This did not affect any of our core systems, only these 2. "Static" server for static seedbox gui files, and wiki. All services remain online.

We will update these servers now, and will probably separate static and wiki for future at the same time.

When you combine all of that with our amazingly efficient datacenter operations and network you have some truly capable servers at your disposal! The environmentalists among us can also rest at ease, we do operate one of the most green datacenters in the world, constantly making great strides to reach ever better PUE! (PUE=Infrastructure efficiency) We have discussed this before in our blog, and we continue to make new records every year.

This pre-order comes also with MAGNIFICENT DISCOUNTS BUILT-IN! All pre-orders will be 20% OFF from list pricing using the below promocode.  You get to keep that discount for as long as you keep the server. On top of that, there is 10% additional discount for annual payments. These two combined can make for a SUPER LOW COST NVMe server at just 27.69€ per month! oO; :O On the higher end, You can have a 2x 2000GB NVMe, 32GB Ram, 2400G for just insanely low price of 50.36€ per month! :O :O This uses the brand new Intel 660P QLC NVMe drives, but you can also opt for more usual Samsung 970 Evo TLC NVMe drives for even higher performance.

• Chassis: ASRock A300 DeskMini  - AN Amazing super small form factor motherboard and casing measuring ONLY 155x155x80mm!
• M.2 x4 + M.2 x2/x4 slots!
• 2x 2.5" slots for additional SSD or HDD!
• 1Gbps Unmetered Volume Network
• Any Linux or *BSD Distro of your choice
• Full ROOT Access; Install anything you wish!
• SLA Gold Level!
• 14-day moneyback guarantee*
• Hardware technical support: NBD+5 (ie. OS reinstalls, component replacements).

There may be several outages during this period of time, please be patient as we work.

This saw the Dragon Mushu, the entry level Dragon to increase to 1.5TB (1365GiB) of disk space, at just 13.99€ per month! Where as the largest option is a incredible 12TB of RAID10 Disk Space! In some cases this means you get a dedicated server for you; 4x8TB RAID10 = 16TB, and if you have 12TB + bonuses there won't simply be space for anyone else. Thus also availability is very limited.

On the SSD series we turned the slider from bandwidth heavily towards space, in reality it is so few people who needs those extreme traffic levels. That being said, it still goes upto 60TB on traffic. Biggest available SSD plan is now a whopping 2TB! Best of all, it is still quite affordable at just 59.99€ a month. Series starts from just 5.99€ a month for 150GiB, but we recommend the 300GiB plan at 9.99€ a month.

Sorry for the delay in repair.


**Issue Resolved

NzbDrone installation issues has been fixed for Deb8 based servers now.

Usual rolling updates are taking in place. If you want your server to skip ahead on queue, please contact support.

There is an emergency relating to these services, and we are handling it as quickly as we possibly can and sorting things out.

Affected services are following series if your server is located in France and setup/upgraded/migraded to in the past 2 years:

MDS Series
PDS Series
Storage Server Series

Got small amounts of data you need to get in hordes out there? What better choice than our SSD Seedboxes.

Measured since 2013/01 we had both the best response average time and closure time average so far!

Excellent! :)

We implemented everything we could think of, contacted all the biggest causes for concern, checked all the possible blacklists we could find for any clues.
The good news is, this time there is atleast an error message. The issue used to be that the e-mail was seemingly accepted, but silently dropped and never delivered to the user in question.

What does this mean?

We are forced to change our outgoing e-mail server in short order - this will cause another myriad of delivery issues. Big e-mail providers like Hotmail don't like at all new senders, first month will have severely limited reachability. We aren't even sure if we should e-mail our monthly newsletter next month! We had severe issues getting it delivered this month as well.

However, we are working on it, and will make a solution happen as quickly as possible.

In the meantime, you can always check out your e-mail log by logging into the billing portal and under account submenu there is a mail history link, direct link: https://pulsedmedia.com/clients/clientarea.php?action=emails

If your server has been updated your welcome page will have download link to the config package and very short windows install guide.
If you'd like your server to be updated immediately, please contact support.

Examples
Case 1:
Super20 at 5.99€ a month, running for 12months, no previous services, no refunds.
71.88€ paid in gives 1.44% extra, 12 months gives 14.4%. Bonus diskspace: 15.84% or 36GiB (3GiB per month)

Case 2:
Super100 at 10.99€ a month prepaid for the next 6months, already running for year and half, previous services total sum 200€, no refunds.
Paid sum total: 9.27%, service running time: 21.6%, total: 30.87% or 228GiB (12.67GiB per month)

Case 3:
Super100+ at 16.99€ a month, prepaid for the next 3months, already running for 2 years. Previous services total sum 300€, 50€ refunds.
Paid in sum: 14.17%, service running time: 28.8%, total: 42.97% or 635GiB (26.46GiB per month)

Case 4 - Paid in sum "multiplier":
3xSuper100 at 9.99€ a month, prepaid for next 10 months, already running for 2 months. Previous services total sum: 300€, no refunds.
Paid in sum bonus for EVERY service: 7.20% - each service receives this for the paid in total.

What speeds are slow? FTP, Torrent, SFTP?
Multi- or Singlethreaded?
Since when has this happened?
What kind of speeds are you experiencing and do these vary over the day?

More information is the better, simple "i have slow speeds" doesn't tell us anything at all.

We will jump right into that and more we got information the easier for us is to check the reason and potentially fix it.

When can you get prorata?
When you UPGRADE plan value (price) wise. No other circumstances.
Not when opening a new service with a coupon code, or downgrading to smaller service.

Example:
You are on Value Small @ 9.99€ per month, and you take Value Starter with 10% discount code: No prorata, this is a downgrade.

Example 2:
You are on Value Starter @ 7.49€ per month, and you take Value Small @ 9.99€ a month for upgrade: You can get prorata. Just request via billing if this is what you want.

Example 3:
You are on Value Starter @ 7.49€ per month, and you take Value Starter @ 6.99€ and 20% discount for 5.59€ monthly pricing. No prorata, this is a downgrade value wise.

Service pricing changes

Service prices change all the time, today we also change it based on available free slots, not just what servers, bandwidth etc. cost today.
General trend is downwards per Gb of storage, less so on bandwidth. However, there is a absolute maximum of how many users can be put on a single server.

We have a lot of users. This means if we change the price lower by 1€ for the most popular service - you will not get auto-adjusted, because this would mean quickly VERY big changes in overall sales in a month: We simply would not have a budget for this.
Further, the server you are on has not become any cheaper, the newer servers might be cheaper - the cost of operation for your service has not changed, but for future ones it may have changed.

Same goes vice-versa, if we increase the pricing, almost no one would accept this if auto-adjusted, many cancellations would occur, many complaints etc. So why would you demand lower pricing when prices fluctuate, when you are not ready to accept increase in pricing for the very same reasons?

Hence, all pricing is grandfathered. It will remain the same as when you ordered the service.

Most of the time lower pricing and special campaigns etc. are mostly about getting new users in, more we grow, more we can afford scale of mass kind of benefits, ie. our own DC, and more internal traffic will happen there the more users there are.

These are also some of the reasons that prorata is not available on downgrades.

• Automatic account creation
• Account creation: Server selection, based on free resources
• Account creation: Strong random password generator
• Account creation: welcome e-mail sent automaticly
• Account creation: username picked and sanitized automaticly
• Service details page: Direct link to login to seedbox automaticly
• Account suspension
• Account suspension: E-mailing the reason of suspension
• Account Unsuspend
• Account Unsuspend: Warn staff if unsuspension failed
• Automatic recheck on server resources

temperature: 24 °C -- humidity: 33 %

IT'S ALIIIVEEE!!!

It's a milestone achievement in the development process - it's good to continue from here and add the required features.

The bottomline is that almost no caching software has any kind of sane algorithm behind it, it's basicly random blocks in the cache unless you have SSD caching in the range of 5%+. When it's the correct blocks the SSDs go into crawl level because no caching software implement TRIM/DISCARD functionality. They mystically expect the server admin to do that for them (?!?). This is true for the convenient methods suitable for our use, we couldn't use something like bcache which needs a format to take into use, potentially leading to reliability issues. Reliability is a big question when you are running potentially hundreds of end users from that single array (20+ disk with big SSD cache arrays).

SSDs are good if you can dedicated it fully to a task, like a RAID10 array of SSDs for DBs, or desktop computers, they are absolutely brilliant on that use scenario.

We tried ZFS L2ARC, EnhanceIO, Flashcache among others. Lastly we tried autonomous tiering but CPU utilization + latency became an issue with that, performance until the core was fully utilized was simply brilliant, but since the software is not properly multi-threading and apparently inefficient CPU utlization it didn't work for a big array on random access pattern. For sequential loads ZFS performance was absolutely the best we've seen, however ZFS falls flat on it's face on random access and linux implementation on reliability as well.

Further issue was that all but L2ARC wanted to do all writes to the SSDs, wether or not on writeback mode. When you are expecting continuous loads of upto 300M/s+ on writes that becomes an issue.

All but the tiering method suffered from wear leveling issues, it was left to the firmware which meant you could only utilize about 75% max of the SSD because activity was so high that 10-15% was not sufficient for wear leveling. Firmwares as well struggled to do proper wear leveling, even the best SSDs did occasionally fail at this. We tried most of the major brands. With the tiering method we went overboard and put of 30% of the SSD capacity for wear leveling, so it's not entirely certain is it because of the tiering software, or so much taken off that performance doesn't degrade over time, but there's no easy way to test this neither, and we prefer to err on the safer side when it comes to production systems.

ZFS: Anything but what it's advertised for

ZFS L2ARC warm up times were ridiculous, the warmup could be 2 weeks (seriously). Further, ZFS linux implementation fails on reliability very badly. On our first, and last ZFS box we used the best feeling and looking premium SATA cables we had and solid feeling power connectors, but the premium sata cables were actually worst i've ever seen: Those sata cables had failure rate of 80%+, further this batch of HDDs had a 60% failure rate, along with some of the power connectors failing miserably.
This lead to intermittent connection issues to the hard drives, sometimes causing a 10-20sec "freeze", sometimes the drive would disappear from the OS.
Unlike any RAID array we've ever encountered, ZFS didn't go to read only mode. W.T.F?! It continued writing happily on the remaining disks, with only reading manually the status of the array telling it's degraded.
When the drive was reconnected: It would start resilvering (resync) with now faulty data. Most data was readable (80%+) up until that point, but due to the resilver/resync process ZFS linux implementation would corrupt the remaining data, and there was no way to stop or pause that process. We ended up loosing a huge amount of customer data.

ZFS Random access performance was ridiculous as well. Our node had 13 SATA drives + couple latest model big SSDs for L2ARC, and it could peak only at 3 disk worth of random access. We were baffled! Afterall, big vendors were advertising ZFS as defacto highest performance around. After a little research and lurking in the ZFS On Linux mailing list, it was found out that the way ZFS is designed it doesn't work for random access *at all*. Every single read or write would engage every drive on the array.
Only way to mitigate this was to have multiple VDEVs, ie. Mirror, ZFS Raid pools. This meant that highest random access performance (our load is 95% random) would be 50% of the hardware performance, while only having 50% of the capacity in use. This is a total no go in environment like ours: Customer demand is high storage with 1-2 disks worth of performance. We would need to double up the disk quantity which would mean huge cost increase, not just outright HW purchases, but also in electricity consumption, failure rates etc.

Further, the failure in reliability was a total no go. A raid array should be by default sane enough to stop all writes if drives are missing.

L2ARC was not usable performance wise neither: The maximum each SSD drive, the fastest on market at the time, would benefit us was slightly over SATA 7200RPM disk speeds, occasionally peaking to almost 2 drive's worth. The good thing with L2ARC + ZFS was that certain things were actually insanely fast. Things like our GUI loaded up as fast as end user internet connection + browser could. In that sense, ZFS L2ARC caching algorithm is by far, orders of magnitude better than anything we've seen since.

iSCSI Backend

We started utilizing ISTGT because it was the default on some big name vendor's products, which were based on *BSD. However, after a while it was clear ISTGT is anything but ready for major production: On Linux we couldn't online load new targets etc. but had to restart the daemon, and it was going to be hard work to hack ISTGT for multiple daemons to increase performance etc.
Every time we added a new vnode or removed there was high probability that another or all other clients would go into read only mode due to the restart time taken. This obviously is a no go. ISTGT has a method for reloading the targets, but this does not work under Debian 7, after researching into this, it looked like that functionality was not finished.
ISTGT as well lacked any proper documentation

Hindsight 20/20: ISTGT has been so far in the end the most stable and highest performing one and easiest to manage.

We moved towards LIO, using targetcli, since this was built into Linux kernel and had some great performance promises and features were all there what we needed, including thin provisioning. That was on paper. For a new node thin provisioning doesn't really matter, so we didn't look into it initially, but after a storage node has been running for a month or two, it's a must have. There was also some performance regressions. One would expect that something built into Linux kernel would have even semi decent documentation and everything open source.

Looking into the documentation: There's not a single open piece of documentation about thin provisioning, explaining how to enable it. Looking from the software it requires some obscure seeming parameters which are not explained anywhere.

Same goes for the management API: Something we def need since as is, targetcli/LIO requires HUGE amounts of manual typing, mostly repeating yourself. Something a script could do with one line command, ie.: ./setupVnodeIscsi.php vnodeId vnodeUser vnodePass dataStorageInGiB osTemplate
Right now we have to create the files manually, and type something like 30 lines or so for each vnode manually, a little of this can be copy pasted reducing the risk of typos, but typos still happen. There is no exampls of how to use the API, only documentaiton is reference manual of commands, nothing really how to utilize it. The documentation is at: http://linux-iscsi.org/Doc/rtslib/html/
Hindsight: Looking into configshell and it's code might have helped in figuring it out.

Don't be fooled by the URL. It say's Linux-iscsi.org but really it was RisingTide Systems LLC site, now owned by Datera. As a business man, i congratulate them for their brilliant marketing work: They have marketed themselves as the de facto Linux ISCSI open source solution - despite reality is that "community edition" users are just beta testers for them.

We approached RisingTide systems and later on Datera, before Datera acquried we didn't even get a reply. Finally when Datera acquired them we got a reply, we were curious where the Core-ISCSI files had gone, only links i found directed to kernel.org but they were not there anymore: Turns out they had turned Core-ISCSI paid only closed source, part of RTS Director/RTS OS. Datera didn't even bother to tell us the price. I doubt the price would make any sense.

RTS OS, without the cluster management features of RTS Director that is, would have had price tag of 950 euro per node. Considering that most of our storage nodes would be 24 disk arrays with max 8 SSDs, the combined maximum cost being around 6000€ per storage node, that would mean roughly 1/6th price increase even on that case, and bulk of our storage nodes were at this point 10 disks max with 1-2 SSDs the cost is unacceptable. Never mind no proper info would it benefit at all, would we receive support, could we utilize SSD caching etc. what features it includes etc. Combined to the fact that a reply i got from them i considered having kind of attitude, the reply made me feel like the attitude is "Muah! You are so scr**** now that you are using our products, you are just forced to pay for it!". I also got the sense that only way to get proper performance out of LIO was with the Core-ISCSI, they wouldn't bother to optimize for the way Open-ISCSI communicates.

By suggestion of a friend we looked into Open-E, but this has a price tag of more than 2000€ per node per year, but would at least have evaluation option for 60 days. We decided against it, as it would force us to use minimum 45 disk pods and that is too risky to put so much business in a single storage node at our current scale. But it uses SCST in the backend, and apparently has very high performance charactestics, we decided to look into SCST. We never iSCSI authentication to work due to lack of proper documentation, and this is the point where we are now at.
SCST has a multitude of ways of configuring it, and the official documentation is basicly snippets of code copy pasted, so really no help at all. Not once in my life have i been confused and dizzy after reading documentation, SCST has been the first one. Obviously a no go, such a high maintenance endeavour with custom kernels, compiling tools from repo etc.

Managing storage nodes: Not as easy as it sounds

There are many dangerous aspects, just last week we had a disk failing while raid was rechecking the array, resulting in completely broken array since the failed disk was different from the one which was being resynced. Caveat of RAID5.
Resync would always cause the storage node to crawl, and without forcing it to do high speed (end nodes crawling) it would take weeks of no redundancy(!!!).
The obvious conclusion form this is that Legacy RAID methods don't work for what we are using.

Looking into the future: Ceph

We are waiting for some developments on Ceph to start testing that on the side, the features should be released by Summer 2014. However, there are serious performance considerations with Ceph. By the scarce benchmark it looks like Ceph performance is very bad for what we are trying to achieve, but we will not know until the time comes.

We are hoping that by then the tiering software has become better or there is better SSD caching solutions available, if Ceph management is good, we would develop gateway machines with RAID10 SAS 15k drives + SSD caches and huge amounts of RAM between Ceph and SAN.

This would mean we would have 2 discreet performance and reliability domains: Ceph cluster and Gateway. Ceph cluster would provide the bulk storage at a premium on performance, and the Gateway machines would provide the performance. SAS 15k drives provide stable performance at 2.5-3x SATA 7200RPM Random IOPS performance, and smaller models don't cost *that much* and with stable high performance they will far outperform SSD with the downside of larger electrical draw.

If a proper algorithm and sane logic SSD caching software comes along, they could proof to be the key to drive the ultimate performance for end users to where it needs to be. We want to be able to offer to every single node 4 SATA 7200RPM worth of random io performance. Currently we achieve more like 1 disk worth.

With the management features of Ceph we can have multiple failure zones and storage domains. By defining "Crush maps", we can make it so that the redundancy backups are on other server room, or even on entirely different building to ensure maximum data reliability. With the upcoming Erasure coding features, we can have say 10 disks for redundancy from 100 disks, and failure of 10 disks simultaneously would result to no data loss what so ever. Due to the automation of Ceph, it would immediately begin make new redundancy copies without waiting, so in a day or 2, from the remaining 90 disks 10 are again as redundancy, only drawback being less free storage.
Complete freedom of storage capacity and redundancy will ease management by a lot. In theory, with sufficient backup, when disk failures happen, we don't need to react it to at all, we can just go and swap the failed disks once a week, instead of immediately, taking the burden away from our support staff.

Further, we can then add storage as we go, for example, just racking 10x24 disk chassis's ready to accept disks, put 1 disk o neach and have a 10 disk array ready for use, and as storage is getting used just go and add the quantity of disks required. This would make things so so much more efficient.

For example, in that scenario, we could just rack all the 100 or so nodes waiting for assemble + testing + racking right now, and start putting them online, as nodes get taken into production just go add more disks.

Since performance is made on the gateway machines, the Cluster disk quantities don't affect that greatly on the end user performance for it to matter. We could simply maintain a healthy 30-60TiB buffer on storage and utilize whatever disks happen to be on hand or at the shelf at our vendors.

*This is what we wanted to achieve to begin with!*
But too many unexpected issues arose that we never got around to coding the management system and too many caveats were discovered in key pieces of software to make it a reliable reality. As a small business we simply lack the resources to hire developers to develop something like Ceph with the performance characteristics we need ourselves. That would need a dedication of several thousand man hours at the very least. ie. 3-4 person development team to get it achieved in sensible time frame, and these 3-4 persons need to be all A players in their respective development fields.

Other hardware issues

We had plenty of other hardware issues as well. Some of them funny, most of them totally unexpected. For example, failed power buttons. Duh! One would never expect a *power button* to fail :)

Some motherboards would fail network boot, unless hard power cycled. This is a totally wierd issue, happens every now and then on all types of motherboards. Power cycling solves it. They would simply not even see there is PXE firmware on the NIC.

Pierced Riser cables: Occasional networking failure, disappearing NICs etc. This took a while to notice what was causing it. Since we are currently stacking the nodes close to each other, sometimes the upper node motherboard's bottom component pins would pierce riser cable of the lower node, causing the NIC PXE firmware to load, poor networking performance, random crashes etc. This one was hard to troubleshoot, as they always looked just fine, and required a bit of luck to even notice this was happening, some of the mobos were just 2-7millimeters too close to each other. Since then we began putting hot glue on the riser cables as extra protection and not assembling the mobos as close to each other.

Poor networking performance: Bad NIC chips or overheating. Some RTL chips have very weak performance, a, b and c models of RTL 1Gbps NIC chip is a no go. Only e is now accepted. A few times poor networking performance was the result of a nic chip overheating. Duh! one would expect that if they use such a big process node it creates significant power draw (we found out that upto 4W !! ) they would put even a small heatsink on it. Ever since, we've been adding tiny heatsinks to the chips which don't have, even the latest model Intel GT NICs which are "Green" and very power efficient.

Overheating chipsets: Some mobos don't have heatsink of the chipset, since some came used to us, they were all brown around the area of chipset huge chip. The odd thing was: This issue was first noted on *Intel* motherboards, duh! Initially we were "OK! They have designed these for passive cooling on cramped space and decided that no cooling is required". Eventually tho, we started adding heatsinks on all  and every one of them.

Heatsinks: Since those 2 above, we've been adding heatsinks to some chips despite no obvious need. With some motherboards we put them under load, and used a infrared thermometer to gauge the chip heat levels and added heatsinks to the chips heating beyond 40C. Some ran as hot as nearly 60C after just 5mins or so!

NICs, NICs, NICs.... We had to go through 10+ models until we found what we want to use... The best Intel has to offer. This too took months to realize that despite RTL advertised specs, they are not up to the task at all. Now we use only 2 models of Intel cards for bulk of our mobos: Intel PRO 1000/GT and Intel PRO 1000/MT. These cost multiple times more, but less headaches, better performance makes it all worth it. We still use the latest RTL 1Gbps chip on the PCI-E adapters, but most mini-itx motherboards have a PCI 32bit connector.

MOBOS: Initially we assumed that all mobos delivered are working and useable. We found out that is not true, we had as high failure rates per stack as 60% ! even new Intel mobos tend to have significant failure rates initially, but then again, with those they were an experimental model to begin with, and many of which were engineering / tech samples which were delivered. Eventually intel stopped the production of the motherboard DN2800MT. Shame! We really really really liked this mobo, only thing it lacked was second 1Gbps network adapter.

Network booting (PXE booting): To add to the per stack failure rates is a bizarre Linux network booting bug: After loading initramfs Linux would use DHCP to get IP address + network boot info (iSCSI target info). On some mobos this fails, on any of the nics, some mobos it fails on the other NIC. Fortunately, this issue rarely happens on addond network adapter card and is usually a symptom of other issue (see riser cable above). Certain MOBO models would claim PXE, but fail at various stages of the boot up stage. Some mobos would refuse to load addon NIC PXE firmware. Most often these issues were with Foxconn brand mobos, which is wierd since Foxconn MFGs so many of the mobos which are in production!

RAM: This mostly pertains to actually acquiring the models, since most ATOMs still utilize DDR2, and DDR2 memory modules are in very scarce supply nowadays since MFG has been stopped or nearly stopped on these. This means 2G modules are *very* expensive on retail. We have acquired A LOT of the memory modules via eBay since our local vendors dried up from supply really fast, and we need 2G modules for the bulk of the mobos. Most of the memory ads in eBay are a scam in one way or other, they go beyond and above any sane effort to scam you into buying server ram or 2x1G kits instead of the seemingly advertised 2G modules. Now we have useless 1G modules in the *hundreds* on shelf, and useless server memory modules in the dozens. Duh!
On the other hand, we have found out that second hand memory modules have freakishly low failure rate. We've had higher failure rate with brand new modules than used. I think the grand total of failed used modules is in the vicinity of 4 or so, while new modules at least double of that!

SATA CABLES! Huh! Don't get me started on this topic! Bottomline is: The cheapest looking red cables seems to be the most reliable. If it's premium cable, it's trash, just cut it with sidecutters and throw it away. You don't want that pain. I'm serious. If it's a normal cable, infact, cheaper the better, locking clip or no locking clip, the more reliable it seems to be. More expensive it is, that much more likely it is to be crap. But hey! That's science for you! That's why Mythbusters so love to do what they do (apart from getting to blow stuff up): The totally unexpeted results. This is one of them. We didn't do proper science of these (ie. didn't write down the exact failure rates) BUT the verdict is beyond obvious: Now we throw certain types of SATA cables away immediately. For example those black ones with white strip with metal locking clips: Almost none of them actually work.
If you have periodic disk slow downs, or disks disappearing: Swap the sata cable first. If it persists, then swap the power connector and issues are very likely gone. if not, the disk is failing probably, seagates fail grafecully like this, giving symptoms before eventually failing. Other brands tend to just go off.

Hard drives! Out of many models and brands we've tried, Seagate Barracuda offers the best performance, price and reliability ratio. Despite our first batch being screwed up with 60% failure rate, which was obviously messed up in freight (obvious physical damage on connectors etc.), remainder has been very reliable. Also, performance is greater than that of WD anywhere near the price. Only WD Black is able to compete on today's drives, it's just sligtly faster, but the power draw is so much higher and the price is 50-60% more it makes no sense to get them. Also, it looks like Seagates are the only ones failing "gracefully", ie. giving symptoms before failure. Also on, RMAs, we have concluded it is enough to say that these don't work on our workload and they get swapped, no hassle, no fuss. If we say that these are having symptom X, even if they can't recreate it, the disk gets replaced.

There are tons of hardware issues i've now forgotten, these were the most memorable ones, there has been dozens upon dozens of little gotchas, like how to route power wiring, power led + button wiring and their attachemnt (we now use hot glue to put them in their place). We also need to modify all the picoPSUs for the right connectors, and all the PSUs we use for the connector types we use etc.
There's been network cabling gotchas as well. Btw, we've spent thousands on wholesale network cabling alone! :O There's just so freakishly much cabling going around, many colors, many lengths, managing them etc. It's hard work to put network wiring for a stack of nodes!

Software issues has been plentiful as well, many of which has been described above with the storage issues. One of the most stupendous and annoying ones is Debian installer bugs tho! Since we still have to install Debian on so many nodes with local drives, it's been annoying as hell.
Debian installer is nowadays so full of bugs it's insane it never passed into distribution! First of all, you need to make sure you are installing on SDA, or at least your boot sector is going to be on SDA. You can forget installing debian on 2Tb or larger disks, pretty much guaranteed failure. UEFI bios as well causes quite a bit of grief. The partitioner sometimes don't allow you to make the partitioning you need to make a bootable system. Installing on RAID1 seems to be a crapshoot as well.
For some one with deeper knowledge, it's just easier to manually partition by hand in rescue system and debootstrap the system than even try using the installer.

So what now?

We are going to take our time to develop the smart server system, we tried to achieve too many things at once, too many things are being in flux and moving constantly, that no one is able to keep track of things. Management has been a total pain in the storage system, we've been in such a hurry to get produciton up and running, but all the little problems have sucked all the time we've got etc. We are basicly putting delivery of these on a halt, and going to traditional setups with local drives etc.
If you have an open order: Don't worry, your node will eventually be delivered. We just don't want to make any hasty mistakes for the final storages as in order to free up the maximum number of disks.

We would have LOTS of capacity to deliver LOTS of suboptimal smart servers, but we don't see that as a good choice: Bad manageability, performance not upto our standards, bad reliability. 

So we are moving in short order to the local disks variety, since this can simplify the setup by removing second nic, and each node with their own local drives creates what i call smaller reliability zones, for disk failures, less nodes are affected (just the one in question, unlike right now 10 might fail in a bad occurence). This means we need to develop some things before we get upto speed with that, we want to finish our DIY blade design for one, and we need to do a PXE installer setup as well. Fortunately, for PXE server installation there is readily available software both open source and closed source commercial applications, but all of this takes time.

In the meantime, we have enterprise varierities you can order, Dual Opteron and Dual Xeon options, these have a known lead time based mostly on freight time from the USA. All of which you can get IPMI/DRAC for remote management, this means you can opt to install the OS yourself etc. It's essentially a barebones leased server option, where everything is tailored to the customer, 1 to 4 disks on each system, of your chosen type and size etc. We also have a limited supply of 20 and 24 disk options and a few 16 bay DAS's for utilizations. You can opt for SATA or SAS, HW RAID etc. 100Mbps or 1Gbps, Guaranteed and Bulk volume varieties.

Just contact sales if you need such.
We still also offer leased servers from 3rd party DCs just like we used to, just contact our sales with the specifications you need.

Custom DIY Blade design is expected to be finished around march or so, and at which time we will not offer an preorder, all orders would be deliverable within 1 business day.

Therefore, we unfortunately had to raise the price for now for new signups to the Super100 series. This increase in pricing will allow us to get new servers more efficiently for the range of services, and eventually have it available at demand.

2009+ Series: We actually have some free spots *right now*. Turnover a few business days until setup, once a week done.

Both OVH servers were obviously shaped, Leaseweb was shaped to oblivion, despite each of these servers being a low network usage server.

We are already planning on Jabba4, which is a bit different due - this one will experiment with custom storage chassis in the backblaze pod style. For Jabba5, chassis is already on order and on sea freight pending delivery at end of november. For sea freight arriving at end of January/Early feb, we are planning to get 3 20-24 disk storage chassis, and couple Rackable Systems 3U 16disk arrays for testing.

If the Jabba3 software is a success, we will start utilizing 4Tb disks with even heavier SSD utilization than before. That will finally start realizing the level of operational cost we have been targeting, while more than realizing the performance targets.
We are also planning to bring online 2 more Solo type storage nodes, but with a few more disks than usual.

We are also toying the idea of building almost SSD only storage node for OS data only, it would feature several Tb of SSD, and smaller size magnetic drives in RAID10 to maximize performance.

We got 27 nodes waiting to be brought online, 25 of which are waiting for new PSUs, for which we will probably source locally overpriced ones to get them online ASAP. We will roll them into production slowly to see how the new storage model copes. All 27 of which has already been sold and pending delivery to customers.

Some 20 nodes are already purchased and waiting for delivery, and at end of month we will order a bunch of the AMD E350 mobos + latest gen Atoms (~30 total), along with a quantity of older model Atoms for the 2G series. We are targeting to bring up next month up to 50 nodes online - depending upon the storage progress.

We also have inbound on sea freight a high end Dell Dual Quad Xeon, 72Gb ram server for testing - if the power to performance to ram ratio is good, we might start offering this model of server with 1Gbps unmetered at beginning of the year.

Right now by far the biggest expense is the storage - to maintain high performance we need to seriously overshoot the performance characteristics, but once we get the software and hardware designs honed in we are hoping to reach a sensible storage vs. nodes cost ratio.

Right now the operational costs are FAR FAR more than the revenue from Espoo DC, we are trying to ramp up the production schedule of new nodes as much as we can. This is not easy work, there is so much things which needs development in both software and hardware, and we actually need to come up with new hardware designs to be able to compete with big budget DCs, thinking outside of the box is a must to be able to come to a sensible cost for the services. Some big budget DCs even have almost free electricity, and their space costs are a tiny fraction per m2 compared to ours, and to overcome these obstacles requires a lot of creative thinking.

Current bandwidth usage is very low as well, we are happy to say that we are operating BY FAR uncongested network: http://i.imgur.com/8X95di8.png
In the graph they are vice-versa. so inbound is actually outbound. Measurement is from cogent side port.

In question is a SQL Injection where any string is passed directly to the DB without sanitization if user input is prefixed with certain string. Sounds almost like an intentional backdoor.

WHMCS sent a security advisory last night, but this flaw was exploited before we could react to that advisory.
We had to restore a backup just prior to the attack - a few transactions may have lost, we will manually input them by tomorrow, but if you don't see your payment by monday, please open a ticket.

As a precaution, all WHMCS passwords and service passwords will be reset over the weekend. This is purely a precaution, we do not have evidence of any password leakage, but it's better to be safe than sorry in this instance.

You will receive a password reset e-mail for your service(s) and billing portal via e-mail soon.

We have managed to solve issues with our old supplier, which provided suitable servers for the Super100 series. Some new servers will be acquired with them to keep the queues shorter. Main focus is still getting these up & running on our own hardware however, so what this means that the queue simply gets shorter, and sometimes even eliminated as we work on getting more of our own running, and get only servers from the 3rd party supplier when the queue is becoming too long or no new servers in sight.

We might have days with more than 200 tickets, some of which might take an hour or longer to solve. and when there is tickets which do not explain the problem, a multitude of them etc. it can become extremely irritative.

First of all:

We are human beings as well, and we are a small team, we do need some sleep as well, time off, have dinner etc. which means that we cannot cover every single minute of a day always.

During weekends we work on skeleton staff - only critical stuff is handled, usually no service provisionings and definitively no billing support etc. which is not time critical.

Making a ticket seems to be hard. Very hard at times.

Many of you send us tickets like "not working" - what is not working? What needs fixing? These tickets get largely ignored because testing everything is just not possible every single time a ticket like this is opened, usually the same people open multiple tickets: It severely slows us down.

So please, let us know WHAT is not working in detail, what error message you are receiving, what you are not seeing or what you are seeing which is not supposed to be there.

We do not provide support for autodl-irssi or other power user features - if we don't have a KB or Wiki article about it, you seriously need to google and learn how to use it. This is irssi in general (Autodl-irssi is actually a plugin script for irssi), shell usage, screen usage etc. We provide support for the web gui, web client, pulsedBox, FTP, SFTP etc. the normal, every day features everyone uses.

We are not teachers. Goes with the power user features - we cannot spend hours per user wanting to use these but do not know how to, to teach them the most basics of irc, irssi, shell, linux, terminal, screen etc etc etc. These can become lifelong topics of learning. What we do however, is try to create KB and wiki articles for the most commong questions.

Do not ever make multiple tickets over the same issue - ping the existing ticket if you need to. Keep it short and simple.

Do not send in a ticket 5 mins (or even 3hrs) after service purchase "where is my server?" - We are working on it, but since we do it manually, your ticket just wasted the time which could have been spent on service provisioning. And we usually do service provisionings in batches to sort them out fast and efficiently.

Do not ask us to provide trials or free seedboxes - we have 14 day moneyback guarantee and free seedbox offering which is given on a random basis - read the terms instead.

To open ticket via e-mail: You can just send us an e-mail - but do it from your registered e-mail address, otherwise we have no idea what you are talking about if it's account or service specific question. Also e-mails sent to support e-mail from nonregistered e-mail address will be rejected completely.

What is IRC, how to use it?

IRC is not 24/7 live chat with immediate response. It's not a support channel with a person constantly waiting just for you. It's a hangout, you can get some support for generic questions, but no account specific support.

If you come to IRC and ask a question, please wait patiently, likely someone will reply to you once they check up on IRC. IRC is a lot like Skype, MSN etc. you simply keep it open on the background and check in now and then - in some cases, people might be away for weeks.

Do not PM or /msg staff, and especially if you do, don't then just quit IRC after 1min waiting. Nothing is more irritating than getting 20 PMs to close every time you check on IRC, and if you reply to them, 80% of them have already left and you just wasted 1hr replying to things which never reaches whoever made the question.

IRC can be a nice hangout and for general chat - but if you need priority support, the only proper way is to open a ticket.

Several users are now going to get a ticket about the issue - if people don't start paying attention we need to take harsher action than just notifying and at maximum limiting upload slots.

Everyone on a shared service has equal right to the bandwidth available for the server - and it's simply not right for a single user to be using 90%+ constantly and not letting others get any bandwidth.

Super100 series is already being planned on having traffic limits in near future when the series starts migrating to Espoo to our own servers when the series gets upgraded for 1Gbps downstream speeds.

Infact, we have a few spots remaining over for immediate setup.

Next week the hardware router module should arrive, and couple storage nodes are ready to be taken into production, along with couple stacks of nodes next week.

We are waiting for a module for the new switch, no exact ETA, but looks like max 2 weeks for it to arrive. In the meantime we will continue prepping more hardware to get online.

Currently there is couple cages racked and ready to be deployed, two storage nodes completely empty and waiting. Next week we should receive a new 20bay chassis for which we will use Adaptec raid cards with BBU + SSD Cache, and likely do a RAID6 on 20 drives plus 5 or 6 SSD drives on RAID6 and tier the storage utilizing btier. It's a little bit of an experiment but one we are hoping to be very wortwhile our efforts, on that node production will be started up slowly to see how it goes.

We also have on order a significant number of new nodes.

In the meantime, we will still try to solve the software router, but not keeping our hopes up for the moment.

A LOT of disks arrived, which means after new switches has been configured we can roll out a lot of new nodes - however, we will not do that until network speed is good, then we will roll out at once tens of new nodes and tens to follow in a week.

By end of the month we want to reach 100 nodes at least online, ie to double our production.

We will be looking into it again, and potentially changing to hardware router afterall.

By the looks of it we will be spending Saturday afternoon onwards fixing the router and getting new nodes online.

Currently we have racked waiting for delivery:
8x2G
8x4G
2x8G

Waiting assembly:
10x4G Atom
5+ Misc Atom

On order:
15xAMD E350
9xAtom D410
Few misc atoms
24x3Tb Seagate Barracuda Disks
5xMicron SSDs for booting
5xOCZ SSDs for caching
etc.

We are going to swap the routing software for a commercial application if solution cannot be found within the next few days - there shouldn't be an issue like this with Linux, the router itself sits at about 99.9% CPU idle but won't route more traffic. Ultimately, we will swap to Dell Powerconnect gear if no other solution can be found.

Upstream provider issues resolved

Last of services has been brought online, except for a very few dedicateds which got cancelled or no user communication still.
95%+ has been up since sunday, just a few accounts remained to be handled during the past couple days.

PDS + Dediseedboxes are pretty much complete, a few servers remain a question.

2012 series is half way there.

That provider changed the terms once again during the weekend to make matters even worse. Very nice of them. Fortunately, only few months remain until we can move production out of there, and this kind of stuff is exactly the reason why we've been moving out from there slowly for a year now.

We are also approaching this from the point of leasing some of the hardware since the need for new hardware does exceed greatly what we can get on our own budget alone for the next month.

For the next month we have already scheduled to get online at least:
15x2G
10x4G
15x16G

Most of these has already been sold and assigned for. We need to build the storage to go with these as well, which is approximately 110Tb and 10k IOPS+ constant.

If we find the development time we will be testing on small scale some distributed storage options as well, but how they work are not ready for prime time due to the need to triple to quadruple the amount of storage we have, so it will remain small scale testing just to see how the performance works out.

We are also going to test some new pieces of hardware this upcoming month to see if those will drive up the IOPS capabilities how much higher, along with some new software solutions for automatic storage tiering.

So it's going to be a another busy month.

We are also working finally on our own blade chassis and estimate the first prototypes will arrive by end of the month.

Infrastructure investments has been much much larger than anticipated, storage expenditure has been much much larger as well than expected. It also has come to our attention that offering a 2G model might not be in our best interests - RAM costs are so low right now, and RAM will provide such a boost to I/O performance, so we are considering the options in the upcoming weeks.

We are working now on bringing remainder of dedis and 2012 online.

NOTE: This does not affect all our services, about one third was affected. We are really sorry over the past 12hrs, and this is exactly the reason we are working to get to our own hardware - stability and reliability is what we strive for.

Some servers will altogether be replaced with new ones as that is the best way to do things right now. We are sorry for all the inconvenience, rest assured, we are working constantly to solve the situation.

This situation is solely caused by the French upstream provider giving us no notice what-so-ever of the changes in terms, and giving us no flexibility on solving this. Infact, last time we called them they were quite angry, it felt like the customer care rep wanted to attack through the phone.

All these issues is why we are working towards getting to our own hardware - so that quality and reliability is on our own hands - not someone else's, especially someone as far away as France.

It seems like we have no option to renew under these terms for just 1 month, but will find that out.

PDS/PDS13/DEDISEEDBOX/1GBPS/100TB:
If you make an early renewal for server, please use add as credit if no open invoice exists, then when the invoice is created you can apply the credit or ask us to apply it - helps in managing it. It's faster to manage that way and assures everything gets correctly accounted for.

Migrations
We are doing a lot of emergency migrations right now for 2009+ series, you will receive couple e-mails if yours is on the emergency migration. For 2009+ we happened to have plenty of stock, next we are checking up 2009+ Starter, X2, and finally 2012 series. Value, Super100 is on another DC without these issues.

This time in question is the renewal terms, our process has worked always on a weekly basis - check weekly for cancellations, renew those servers weekly, this has helped to minimize the price of service for our customers as there is less extra time we need to pay for, less things to check up on etc.

Last night was time for weekly renewal of servers - but this time around the minimum term was 30days. Since we didn't know this upfront we didn't reserve that kind of sums on account, as we try to keep balance quite low for security purposes, on top of that, most dedis are due for renewal well before 30days is past, many servers are to be migrated out etc etc etc. making this extremely complicated. A bunch of servers we did renew for a month, dedis which have due date 18th of September or later, multiple shared servers etc. but a large portion remains to be handled still.

We will be communicating how this issue progresses.

Pretty much, the upstream provider customer care is not interested in solving this issue neither - but we will keep contacting them if an resolution has been found.

We will do an software update and see if this solves it, giving for all nodes good speeds on single connections, if not we will upgrade to Dell Powerconnect gear by end of year.

Even if that particular node cannot achieve good single connection speeds it has been noted that with enough effort it can exhaust it's link no problem.

Solo4 (Storage): Getting built in the nearby days, this is a new config which might require extra days to test. This will have 10x3Tb Drives and a Samsung 840 PRO 256Gb SSD Cache drive.

Solo1 (Storage): SSD caching enabled in write around mode -> seems to give roughly 1 magnetic drive worth of performance boost when utilizing eio. Not happy with the performance increase. Planning on tiered storage for future.

Network infra:
2xExtreme Networks Summit200-48 switches purchased for 100Mbps.
2xCisco Catalyst 48port + 2xGbic switches acquired for 100Mbps, and the according Gbic modules purchased.
Still got 1xExtreme Networks Summit200-48 waiting to be deployed.
1xBrocade switch arrived at mid warehouse for the SAN infra (48x1Gbps + 4x10GbE)

Contemplating on acquiring a Dell Powerconnect 6224F -> 24xSFP+ 10Gb ports, and same series 48xGbit + 2xSFP+ switches for december delivery from the states.

Purchased a lot of network cables, over 600euros worth, now we've spent some 900 euros on cables alone! :O On top of that all the accessories for cable management.

3xCore i3 is waiting to get into production. 4xAMD E350 is waiting to enter into production. 4xAtom D410PT waiting to enter into production. Misc cage of 2 & 4G Atoms (D510, D525, n2800), 8 units, racked and waiting to enter production.

Still waiting for racking & wiring is a cage of 8xAtom D510/D525 with 2G each, next cage of 8 Atoms is waiting for misc accessories to be built.

We are starting to run out of ports on internet access side -> adding one of the Summits during weekend.

Shipping to warranty 3x3Tb Drives + 2xCore i3s today. Waiting from warranty 3x3Tb drives. Suspecting 2 more drives waiting for rechecking before sending to warranty. If they indeed are failed, that is already 8 drives sent to warranty (yikes!)

Waiting from the states:
1xBrocade switch, few SSD drives, about 20 nodes, lots of ram, couple highend seasonic PSUs, Adaptec highend SAS/SATA Controllers (16port each!) etc.
One vendor today shipped a batch of 10xAtom D2500, 10x92mm cooling fans, 3x120mm fans and 6xPSUs.

Currently on stock about 20 more pico PSUs, 40 NICs (each node needs 2), 10+ 3Tb drives, few 2Tb drives (spares), and a bunch of 1Tb drives (will probably enter produciton ever), 18x120mm fans, some 30x80mm Papst fans etc etc. Today receiving a shipment of correct plugins etc. accessories for assembly.

Contemplating on a purchase of a lot of 16x AMD E350s. There is cheaper models available as well, but they are out of stock right now, so considering these more expensive ones.

End of the month we should receive first 20bay hotswap chassis for review, anxious to receive it. Contemplating tiered storage for it. To do that however, i think we need to intentionally wear some SSDs first so that their failure times will not be close to each other.

We are also wondering how far will the cooling last for us, but fortunately, during winter is scheduled upgrade for secondary cooling unit. Our current cooling unit is just 9.4kW cooling capacity, but since it's the only one we do not dare to load it much beyond 6kW, after adding the second one our capacity should be at around 15kW or more. It does ease however that we run the DC "hot", at just shy 30c ambient (fluctuates 28-29). It's nowhere as hot as Google does however, and our metric is disk temperature -> keeping them below 40c is what we target.

Lots of software work remains to be done, most of the monitoring, new distro templates, better identification of physical units and better management of them, automatic per node monitoring + reboot (along with the hardware!).

Storage is stable and performing. We are seeing daily higher peak BW usage.

Immediate backlog is about 30 nodes, and 10 migration nodes.

8G and 16G models got changed for AMD E350 and lower price!

Storage addons were added!

See all the details: http://pulsedmedia.com/personal-dedicated-servers-finland.php

You can view it at: https://pulsedmedia.com/clients/networkissues.php

For now we have scheduled Jabba1 migrations.

Let's see if this way of informing is better than using announcements for this :)

Letting the storage to resilver for a while before onlining everything again.

Work has begun to bring solo1 online.

UPDATE

Jabba2 array1 decided to resync as well, ETA for completion at current speed is about 5 hours.

However, we will just increase min sync speed and online things far before that.

UPDATE2

Solo1 is building it's array, ETA ~7hrs

Jabba2: We are using Qlogic/Dell NetXen card for it's 10Gb, and a driver issue makes it unavailable every now and then on reboot, took a while to recognize that it's yet again failing to load and that's making us take longer than expected in bringing things back online.

Jabba1: Solo1 will be imported to Jabba1 for migrations, we are unsure will we update the OS images for new IP or will we keep it like it is, going through Jabba1 for a while. Probably the latter for first couple of weeks.

UPDATE 3

Some nodes are up, for some reason nodes from Jabba1 ZFS pool are not getting online, and we are seeing some worrying things from that pool (3 disks resilvering while parity 2??). We'll know more shortly, might be just so badly overloaded due to the resilvering process.

Solo1 Array1: very huge variance in disk performance, swapping 3 of the 5 disks in an attempt to get more similar performance drives in the array.

Solo2: Being installed.

Jabba2 Array1: Still resyncing (resilvering), with limited speed, all nodes utilizing that are up & running tho.

Jabba2 Array2: Is doing initial sync (5x2Tb).

Recap: Storage entering online is two 5x3Tb arrays and one 5x2Tb array.
Solo3 might get built today as well (5x3Tb array)

UPDATE 4

We are manually running fsck on those machines which did not boot automatically - some of them have some corruption but so far every single one has booted up fine after doing manual fsck.

We are soon fully back in production.

Migrations will begin by tomorrow on the newly built arrays.

Solo1:
5x3Tb 7200RPM with small SSD cache drive, RAID5
Jabba2:
5x3Tb 7200RPM with small SSD cache drive, RAID5
Solo2:
5x2Tb 7200RPM, RAID5

Jabba1:
Will import the array from Solo1 to migrate remainder from the initial ZFS pool which is underperforming.

This will vacate a bunch of 3Tb drives, which will be used to create a RAID6 array on Jabba1 with a 256Gb SSD cache in writeback mode. We are likely going for 12x3Tb in RAID6 or 2x5x3Tb in RAID5 for the Jabba1 after that in couple of weeks.

We have reserved 6 of the small "Solo" type of storage servers, these have each 4G ram for cache and 2xGbit links, able to sustain ~220M/s disk speeds (which is ~triple of what an array like that can sustain in heavy random I/O), and each have 6 drive spots, 1 of which will be used for a SSD boot + cache drive, and remainder for a RAID5 array.

Each of these "Solo" arrays will be used for max of 8 nodes, depending upon the setup and target, most likely for 5-6.

The initial Jabba1 pool has suffered now 38% disk failure rate - all of them were new drives, so it was a very bad batch of drives which we got for the first array, this combined with ZFS basicly being falsely marketed as outperforming RAID5 has been the culprit for most of the issues.

Istgt as the iscsi target will be dropped, we will be testing soon can Risingtide Systems iSCSI target support the special characters in the passwords used, or should we manually fix all the vnode OS images for new passwords in order to migrate to it.

The past month performance and stability is completely unacceptable, unsustainable and incomprehensible. Murphy's law at play here, when it rains it pours. None in our team has ever experienced such a degree of wierd issues as we have had to endure during the past month, starting from failing new fans, power y-cables randomly loosing connection, sata cables giving issues, 38% first month disk failure rate from a single batch (while other batches have 0%), almost none of the software used working as expected and advertised causing random issues and the level of logging being so inverbose that they are useless in debugging issues in question.

But we have learned a lot during the past month, and are heading to the right direction, several key portions of the infrastructure are now very stable and we can build on that.

As a backup, we are setting several systems with local disks, just on the high end and just a handfull.

I'm fully confident that we reach full stability and mass production during august.

It will lead to some nodes having downtime as we look into it, but we are hoping to keep it under 2hrs.

We will be adding several arrays today, and hopefully bringing a bunch of nodes into production today as well.

Servers are booting back online, and we are now targeting that only downtimes for the next several weeks are data pool migrations.

FS is giving for couple of nodes error of data corruption, but most likely fsck will handle that without data loss.

the first pool is right now resilvering but FS is warning that data corruption may have occured, it will be brought back online shortly along with the nodes.

Exact ETA is unknown, but everything will be back operational within several hours.

Almost all espoo nodes will be affected.

We've worked on the software side of things, and finally we have found a way to deliver both performance and flexibility, and bigger scale stability testing will ensue soon.

We've managed to get raw device performance over the network without any special "hacks" etc. and in a stable manner, infact, with the new software on the backend some nodes has been working and in use for a while now.

We are starting now to build up the arrays, which are initially going to be 12Tb each.

We had to downgrade our redundancy level significantly to get the performance required for this to be pulled off, but we will increase the same redundancy level by doing smaller pools, so if a pool is lost it will affect a maximum of 20 servers or so.

Eventually we will also provide optional backup services on a per node basis.

There will be a maintenance starting today as we upgrade one of the SAN gateways to faster network access. The time when this is done is not exactly known, but the maintenance is done so that we will not need to do any maintenance on that particular machine for a long time to come. This will affect a fraction of the nodes running currently.

All vnodes will be migrated to the new arrays to vacate the current one so we can vacate the 3Tb disks from it, and do a nicely performing array from them.

Most initial arrays *will* get SSD caching backup, later on we will add SSD cache on a *as needed* basis.

We are working to build a sustainable solution for the future, and working to build a cluster of storage servers, which we initially build out, before building up, this means we will keep adding storage cluster nodes with minimal disks initially, before starting to upgrade them to bigger machines.

redirecting efforts to build new storage bricks.

The primary router will be changed to more suitable hardware, this will result in approximate 45mins downtime. Basicly we are downgrading it. Yes, downgrading. We over configured the gateway by vast margin, which results in more expensive, more power hungry than required, we will do a tiny downgrade on it, CPU processing wise it will remain the same which is the only thing that matters. Even during heaviest load so far we noticed less than 1% CPU peaks due to the routing.

Storage will be reworked today, due to flaws in ZFS design, it will cause intermittent downtimes. We will move a vnode at a time, which will mean this vnode is down for the duration of the copy operation, we will restart it once copied to another raid array, after which we build a new storage brick, and do another array, to which we copy from this moving array. Again, causing intermittent downtimes.

The storage work is likely to last 8+ hrs due to the sheer volume of data to be moved. After this maintenance we should expect to see multiple times the storage performance what we see currently, and this should be a setup to stay with for a long period of time and we start to rollup the storage network and cluster after we verify this gives us the "holy triangle of storage": Cheap, Fast and Reliable.

This time around we are going to rely on older, proven, well known FS: EXT4 + Raid5/6. Sure, we lose snapshotting, checksumming, inline compression, but via bcache we should achieve better usable cache for our systems and *solid* performance.

The problem is with the way ZFS was designed with only one goal in mind: Data integrity. This means that a RaidZ[1,2,3] volume only *has* single disk performance - despite being advertised by many (Including one big old time enterprise which has been always in forefront of industrial computing) to be even faster than Raid5/6. This simply is not true. Your only other option is to have weaker redundancy and loose 50% of your storage by doing mirrored pairs, or do multitude of RaidZ, and thus loose much more than 50% of performance. Neither of these scenarios are acceptable. If i have understood correctly, even with mirrored pairs you still loose 50% of performance. So this translates to roughly 2 options: 50% storage + 50% performance loss OR 20-25% storage + 80-90% performance loss.

Bottomline is: ZFS is good for backup and streaming loads, it has magnificent write speeds, but poor read speeds, cache warms up too slow, and cache expiry is vastly too long for our type of loads (hotspots change frequently), and very poor IOPS figures in RaidZ and cache wise. Also there is absolutely 0 pool reshaping features apart from expanding by swapping larger disks. The nice features such as inline compression, checksumming do not outweight the lack of performance unfortunately. In any case, on our load data volumes have 0% compressible ratio.

EXT4 + Mdadm RAID5/6: Raid5 achieves near hardware read speeds while suffering on write, Raid6 is not much slower than that, we expect to see 75%+ of raw hardware performance, while loosing 25% of the storage initially. EXT4 is overally well performing, especially with a little bit of tuning, EXT4 features are limited but what it does, it does well. Mdadm is slightly harder to manage (ZFS failed disks are very easy to replace!), and while write speeds suffer, write is not a heavy load for our use: Generally 1:25 write to read, meaning 25 times more is read than written!

Further, ZFS on Linux fails to compile on latest kernel, there is some hacks to get it compile but we haven't attempted this, while EXT4 is built-in the kernel. ZoL documentation is also extremely weak, and it still has some regressions which are worrying. EXT4 has no regressions, and documentation while not needed, is vast.

BtrFS: It's getting more mature by the day, reading up on the BtrFS site the only major drawback we noticed is the lack of needed redundancy levels, we'd like to use Parity 2 or 3. BtrFS is in the kernel, so we will look into it someday.

After we validate the storage is performing top-notch we start to roll out more machines again, we got tens of nodes in standby to be built, tested and configured, and several already up but not being utilized.

Also we are about to purchase a big lot of hard drives within several weeks. Initially for every disk we put online we can put 1-1.5 servers online, and add the missing storage on as required basis. This should quickly rise to 1:2-3 ratio as we grow the pool and add more SSD caching. Finally when storage starts to be utilized, we estimated the total need of 1.25 disks performance per node, but the scale out architecture we are planning for allows us to trivially keep on adding storage on a *as needed* basis, allowing us to offer these extremely competitively priced services.

1.25 disks performance per node means it's relative to modern 7200 RPM SATA disks, whether the performance (IOPS) is delivered via SSD or Magnetic drives.

New hardware for new storage nodes are on order and will arrive by late next week, until more performance is achieved no new nodes will be provisioned.

Things are still so much work in progress that we cannot setup new machines any faster :( We already maxed out parts of the infrastructure and have to beef it up before new machines can be setup. That is going to happen sometime during next week by all likelyhood.

I think we get new storage nodes up & running by end of next week, it'll take several days of building, testing etc. before that is ready, and then we will put online a lot of servers each day.

We have currently such a great demand that this situation will remain for months to come by all likelyhood, we got more than 60 servers on backorders alone right now, and we need for our own needs a total of 300+ servers, it's likely we setup these on 80:20 ratio, 80% customer dedicated orders, 20% of our own.

A lot of development remains to be done on the management backend as well, remote reboot code is still largely done, identifying nodes and their image counterparts, managing the network storage etc. all needs a lot more effort put in.

We only got a handful of customer dedis operating right now, and this has given valuable feedback on where to improve our infrastructure, and the most important bits has been developed now in regards of building new servers.

In the end we reverted back to old config, as iscsitarget was not providing the performance excepted and several authentication issues.

We will be looking at alternative options, but for the time being, storage will remain in configuration freeze, now changes will be made for a while to it.

Everything is prepared so hopefully the downtime is only less than 30mins.

We need to reboot all machines. Reason is on the storage side several drives have dropped in speed, which means the whole array is crawling right now. We have no idea why the SATA link is being renegotiated for lower speed, ALL sata cables are high quality and new. Further, Samsung 840 PRO SSD drive is again giving grief with hugely degraded performance, doing only 28M/s read peaks!

At the go, we will likely also update kernel, and likely finally change over to IET iscsi target.

All of this will take several hours and occur over multiple reboots etc. production will be restored occasionally momentarily as we prep the next change.

These storage maintenance breaks *will end* as soon as the cluster is up & running. We estimate that to happen early august.
At that point we can work on individual nodes without interference on the user facing side, as long as it's not something to do with the SAN gateway machine.

Both system show full bandwidth utilization, and the AMD one also heavy inbound traffic.

despite these, the GUI has *never* been this fast. I've never seen the filemanager load up so fast, nor never seen ruTorrent load up so fast. It's insane!

Tomorrow is some fine-tuning work to be done and some broader changes as well, from istgt to iet and getting the only performance drawback fixed: Latency. Sometimes there is high latency on system which is not utilized, on data which is not utilized. Intent is to fix this, trying out different disk schedulers etc.

Plus, using IET BlockIO mode should enhance things further as well.

Disk array is working fast as well, we are seeing very nice speeds with barely any load what so ever on the array, the SSD caches sit also mostly idle writing up new cache data -> Cache is never really getting warmed up yet either!

Let's hope as production goes forward and we start our work on the storage cluster things look this good!

Also any connectivity issue very easily causes client node to remount as read only, for unknown reason.

Therefore, we are possibly changing the iSCSI daemon to another one within the next 48hrs which was noted to handle errors much more gracefully in tests done earlier.

istgt was chosen because it was the daemon of choice in some enterprise solutions and we already had some experience with istgt.

Things are still a bit work in progress, and i'm sorry for the issues, this is quite a bit of research and development as this kind of usage case isn't just happening elsewhere the way we are doing things.

The file system itself seems to be going really fast right now, we are seeing 1.6G/s (YES, Gigabytes per second) writes and reads, with random reads still above 540M etc. with iozone -a -s 16g -r 4096 -T test.

This needs more diagnosis, but by the looks of it the culprit now is the iscsi daemon istgt.

There's been reliability issues as well for the past day, one volume got corrupted and several machines required reboots to regain access to storage.

All vnodes are back online, and we are prepping to bring online a lot more during this weekend.

This will happen today, approximately from 18:00 GMT, and will take couple hours.
We will try to keep client nodes up & running during this period but we cannot guarantee that. Downtime is likely to occur.

There is now also first dedicated server delivered, and some servers for seedboxes for a total of 5 servers in production now.

We are already seeing peaks of about 300Mbps and 30+ kpps on network side, this was with just 3x100Mbps servers on production. Our gateway is performing admirably, at those rates CPU idle sitting nicely at very near 100% or at 100% :) Wonder what happens when dynamic routing is enabled, but even then this gateway should be capable of sustaining 20Gbps+

Storage server there was a suprise, CPU was maxing out, and this is not a slow CPU either, but it was quickly figured out why and fixed.

Currently there is 16.5T allocated from the storage. New storage nodes + drives are already in order.

Tomorrow several more nodes will be setup, and we are expecting a shipment of several nodes this week.

Hopefully next week we reach 20+ servers online, with a lot of the preorders already delivered.

Initial production system testing on the SAN:
Core2Duo
Intel Gbit adapter
Single connection, Single Session
No jumbo frames

108M/s throughput on first attempt and 3200IOPS.

Comparison: WD Black using the same tests achieved only 160IOPS.

We are days away from production.

We ended up on decision to build a big cluster of small machines, and keep on upgrading those on a as needed basis. This will give much better redundancy as things are relying on many more individual machines and scale out is easier on our budget.

This doesn't mean any changes on the current offerings, just how they are going to be provisioned in a month from now. After this initial expenditure this allows us to put much more budget upfront on the nodes and then weekly storage upgrade routines.

We are targeting a year goal of 120+ HDDs on the cluster with 40+ SSD caches, with steady throughput capacity of 40Gbps+, likely to be upgraded to quite fast to as high as 400Gbps, depends upon the needs.

Also, we are going to experiment sooner than expected with local SSD caching, and purchased a lot of mSATA SSD drives for testing purposes.

Our IP Block has been now assigned, and tomorrow people are working towards getting things online.

Netboot/iSCSI boot is still undone, but first storage brick is now stable and final configuration.

i3s require 19VDC power input, a oversight in design, it was assumed they have wide input as well, so these will be delayed until late next week to get up & running.

There is circa 15x2G, all sold already, and some 9x4G models, most already sold, and 3xi3 8-16G for the first nodes to be setup on these first days, more will be setup weekly, wednesday or fridays.

After several days of issues with a storage brick we noticed the SATA Raid controller was not compatible with this specific SSD model and make. Then basic SATA power connectors failing one after another, a need for not just Y-cables from Molex to couple sata power connectors, we actually soldered our own tri-cables as no shop was open (past 9PM) and wanted to get it up & running ASAP.

Then we get to the fun part: Switches. Those mystical black boxes. We are starting to understand why people pay a premium for support contracts and keep a tight check that their model is still supported. A bunch of our switches are Brocade / Woven Systems 48xGbit + 4xInfiniband 10G, and whoa, has it been a job to get them funciton at any level!

After tedious search with very little results finding a supposedly correct manual it turns out half the example commands and half the documentation is completely wrong - but not totally off the base. Hours to get the switch even to respond to it's IP via management etc.

Just to notice that our HP Infiniband adapter is not compatible with this type of switch, the HPs are compatible with the Cisco Core switches (those beasts of a switches with 960Gbps fabric), but not with the Brocades. Fortunately, we had some Myricom cards which are not really even Infiniband - sure they utilize the same switches, the same fabric, but only support IPoIB, but these atleast work with the Brocades :)

Lots of things like this, even small pieces like rack nuts can give you a headache: We had a bunch of nuts which used smaller bolts, and huge pile of nuts which were way too loose to be used oO;

Personally, i've never used fiber links anywhere, yes, i've seen that stuff, but actually using -> a little bit of research and wondering What the ??? am i supposed to do with this connector :) Thing is, i like to learn this stuff, even tho we got people in our team who knows this stuff, but as "The Man"/"The Boss", i ought to know at least the basics :)

An all to the way of Brocade being very sensitive to cable quality, and not bringing up interfaceto 1Gbps if the cable in question is not top notch. Well I guess that's a good thing! :)

Plenty of progress, long nights at the colo room but things are moving, expecting that our transit link is up & running in a few days.

Br,
Aleksi

Cogent assigned today our 10Gbase-LR port and are waiting for demarcation information, after that, PtP connection between the buildings is done and we can setup the public network.

Further, we developed our own temperature & climate monitoring system based on Raspberry Pi in order to further lower costs. On this system we can add unlimited number of sensors for roughly 3€ each.

- order and pay the new server
- Once server is setup we will adjust your service due date accordingly
- You may now migrate over to the new one, or we can attempt a mirror copy
- Once migrated, you may opt for prorata refund or days transferred from the old server to the new one, cut off weekly wed-friday. Open a ticket at this point couple days early.

The cutoff is based on our payment cycle, we usually pay the servers weekly, and we will provide prorata according to that date.

The final specification of our own dedicated servers, with a lot of hardware already acquired and ready to be put online as soon as the network infrastructure is finished!

We estimated first servers delivered before mid-july.

Pricing is also very competitive, setup fees are also waived on annual or longer orders!

See them at: http://pulsedmedia.com/personal-dedicated-servers-finland.php

The hardware will be "standard specification" Core i3/Xeon E3, 16G Ram. We will put more users per node, but provide vastly more disk I/O resources, we are thinking about 15 to 25 users depending upon the performance. As traffic limits are also introduced at this time, we believe there will be no performance issues. If there is, we will upgrade the SAN connection, SAN setup or do local cache disks.

After that is operational we will be building nodes up as fast as we can, and new orders for Super100 series will be setup on our own machines.

First migrations will be 2009+ Starter as for that we can utilize 2nd hand hardware. Next up will be Super100, and likely these migrations will go in parallel.

We are doing Super100 in batches of 3-4 servers at a time.

Tomorrow we will finish setting up the necessary access cards, and after mid summer festivities we will start to rack some servers.

Cogent link is expected to be up by 1st of July if timetables hold, there is a 3rd party which provides the transport from the teleroom we connect to, to the equipment Cogent has which will ultimately make the schedule.

A lot of hardware is arriving from a lot of different places, couple storage servers has been built so far, and misc servers are lying around.

Decided to do a nice upgrade to the SAN plans, we acquired several Cisco switches to act as SAN core, each switch has a unblocking 960Gbps fabric. We will have couple of these in production and a spare.

Also, for the SAN side we are going to use way over specced cabling, Cat5e is able to do 1Gbps, but we decided to go for Cat6a and Cat7 S/FTP, which both are way over spec, Cat7 being primarily for 10GbE. This will ensure the best speeds possible, as there will be a lot of cables bunched together and it's important to eliminate as many as possible sources for interference. The shielded cabling will also have way lower crosstalk etc. Cat5e is barely able to do 1Gbps - That's not something you want in your SAN.

The next step in storage servers will feature 60 to 80Tb raw storage in each of them, before finally stepping up to the 135 to 180Tb models.

It'll be after midsummer festivities before we can rack them etc. and we might have production capabilities by start of July.

We are getting keys to the place finally next week and are going to start setting things up.

Routing and networking setup has to be done prior to 10.7 because the network expert is about to leave for vacation then, so we are hurrying up our schedule somewhat.

A few is being setup right now, but that doesn't cover the full backlog yet.

We estimate thursday for all current orders fullfilled, but this is dependant upon the DC getting new servers set up.

From now on, we will try to keep few servers at hand ready for fullfillment of new orders to ensure DC delivery delays does not hinder the provisioning.

This all despite our contract with the DC has a delivery within 24hrs clause, but we do understand their situation.

Storage server:

FX-6100 AMD CPU (6 core)
16G DDR3 ECC Kingston
4x3Tb Seagate Barracuda 7200RPM (Integrated SATA3 controller)
1xSamsung 840 PRO (Highpoint SATA3 RAID Controller)
Integrated 1Gbps network connection

Via 2xHome 1Gbe switches (cheap)
Using subspecification network cabling (poor condition)

Testing server:
Intel QX9770
8Gb DDR2
1Gbe integrated network
Background load level 10% CPU, 64% RAM, 3% network

Unrar performance: roughly 60M/s
I/O test operated with 50% of the time a background unrar going on the file server.

Despite adding up this many factors to negate performance and cause issues first results were promising:
4workers, 32kb 84% random, 85% read, 95% of workload + 256Kb 50% Read 50% Write 95% Sequential 5% of workload.
Which resembles 4xrTorrent instances + 4xUnrar instances, loads 95% rTorrent 5% unrarring.

1620IOPS (8 magnetic drives worth)
52.19M/s  (More than double the monthly average of 100Mbps servers)
Average response time: 8.9ms  (Less than to be expected from magnetic drives in real world)
Maximum response time: 337.5ms  (Most likely caused by the suboptimal network)

Runtime: 120 seconds, ramp up 10secs

More results to follow.

Therefore the upcoming 2G plan will be very restricted supply and not part of the main offerings.

Also, we intend to invest into the best of the current generation for the level of hardware, this means latest Ivy Bridge CPUs, only largest low voltage memory modules which can be utilized, only 80Plus Platinum PSUs etc etc. The buy in might be steeper but the savings in electricity and prolonged life span should be far worth it.

Further, we have decided to delay the launch of dedis a little bit back in order to ensure seedboxes, internal server etc. are migrated first, but the delay we are talking about is only a few weeks, basicly it's about prioritizing the workload we are under right now. Not once in Pulsed Media's history has there been such a high work load for such a prolonged period of time, but by the fall things should start settling down and getting back more to a routine.

We will also rework the pricing with updated electrical consumption figures from testing to be done during June and with the updated lifespan expectations.

Pre-orders will be opened within several weeks, if not earlier. More details of pre-order will be released at that time, along with updated schedule estimates.

Usually new nodes will be ready within 24hrs, but the DC ran out of nodes so we must wait for new ones.

I know you guys are anxiously waiting, and we are hoping that backlog will be cleared by end of this week.

The two testing nodes are minimally configured. AMD 6core 3.3Ghz CPU, 8Gb DDR3 ECC, 5x3Tb Cuda, 1x256Gb SSD Cache, 2x32Gb SSD Boot drives, Highpoint raid controller for cache drives, basic RAID controller for the boot drives.

Final configuration will be 32G DDR3 ECC, upto 14 magnetic drives, 4 cache drives. The two configurations we intend to have in launch are: 14x3Tb + 4x256G and 14x4Tb + 4x256G. Both will operate in Parity 3 mode, allowing 3 drive failures before data is in danger. By my maths, during first 6months there will be 1 disk swap in either of the nodes, and during 3 years designed lifespan there will be 2-3 total failed drives. If failure rate exceeds 4 drives over the two nodes during the 3 years, then cooling and anti-vibration needs to be enhanced.

Now we are testing with total of 10 drives, both nodes running parity 1. Each drive are mounted with rubber insulation, and the drive caddy is mounted with rubber insulators for anti-vibration. Then we put all the drives next to each other to maximize the heat build up on the drives, to drive them to the edge during testing.

At our colocation room the storage servers will be next to the cold air vent, getting the best cooling. Target temperature at the hottest spot @ ~1.5meter height in the colocation room will be 30C. So yes, we will be running the hardware hot, but as testing by google has shown this does not necessarily translate negatively into defection rates. If it does, we will drop the temperature 1C at a time until defection rates are at acceptable level. In the end, we have built in redundancy by design, and it boils down to financial maths, is it better to let 1 extra unit fail per 6months, or is it better to lower temperature and pay X amount extra in cooling.

This is why we test things, for one we need to get extra short lockable SATA cables, better SATA power cables, the ones used in Corsair PSU take up too much space with SSD drives, and modify the cases to accomodate extra 120mm fan or two. Maybe some additional 80mm fans too. Also, someone (me) forgot to purchase low profile video cards as well!

Thankfully my apartment has a spare room to fiddle with these before we get the colocation room, it takes up surprisingly much space when assembling these!

Some pictures: http://imgur.com/xlmRlpO,kn15CWr,BZRXtN3

Now the testing may begin, things we are testing are heat build up, PSU efficiency, performance (ofc), reliability. Our intention is to push these to the limits, and test all kinds of malfunction scenarios. Just building up these nodes showed couple drawbacks in the design, for example the last 5 hard drives with standard size ATX motherboard are ridiculously hard to get into their slot or out, so we might need to drop the storage server size down to just 8 drives.

No infiniband tests yet.

- Aleksi

If you still want one, and can assure us it's not to be used as a seedbox, please open a sales ticket.

With our own hardware we can provide things which were not previously possible.

Reliability will be vastly increased, some will be realized immediately upon transfer, some will be realized over time.

The instant benefits are:

• Data redundancy: 99% of data loss cases should be eradicated, all servers will have redundant storage
• HW redundancy: We can simply boot you onto another HW node if available, hastening recovery from CPU/RAM/Mobo/PSU failures.

This will delay the launch until July.

This gave us a sigh of relief as the schedule was so extremely tight previously that not all hardware would have arrived in time. The 10G switches are expected to arrive first or second week of June for example, and remainder of the storage server drives and components are not expected before mid june.

We are slowly rolling out this release. You may now manually install autodl-irssi if your server has /etc/autodl.cfg file.

Has been tested to work on PMSS with minor modifications.

Namely some packages need to be installed as root, and depending upon the usage /etc/autodl.cfg created. Nothing fancy is really required.

Autodl-irssi will be added on the system when traffic limits are in effect, and only for series with traffic limits. The reasoning is very simple: Autodl-irssi will quickly lead to massively higher loads due to constantly adding torrents, especially when configured too greedy. On unlimited traffic seedbox this would quickly lead to a situation where there simply is no BW for those not using autodl-irssi, and the overall experience is much worse for everyone.

We will likely make an "enable" button for autodl-irssi, and then provide a quick tutorial how to use it, and how to use it for normal ircing as well. We might even go as far as add web based terminal to connect to it.

It feels like insurmountable task with so many different things one needs to take care of. Many would think "Oh, just throw in a router and do the BGP", but infact, there is a plethora of other things you need to account for.

DNS Servers: These need to be properly configured recursive servers, 2 are required. Has to be secure, hardened, and contain filtering for possible DNS attacks (poisoning, amplifaction etc.)

DHCP/PXE: In our case also a DHCP and PXE server is needed, with dynamic on-the-fly idempotent configuration.

Monitoring, web: Web server for displaying monitoring data, out-of-band, discreet connection via separate physical links and on separately routed network. This same server can serve multiple purposes with multi-ip setup, such as inbound VPN etc. However, for security we will propably have all tasks separately.

Monitoring, calculation: A server which does the actual calculations and drawing the graphs for monitoring, which are then synced for the displaying one. Due to the sheer volume "simple" calculation will take a lot of effort, and in this manner there can be multiple nodes to just calculate the graphs to keep up. Munin will not work here neither, and there needs to be things ensuring data corruption does not happen (Munin is extra happy to corrupt your data at high volume)

Management: There needs to be a management server which is primarily connected out-of-band.

These are just some servers to configure, of course there are other concerns as well as the electrical infrastructure, UPS, racks itself, building and testing the servers etc. Never mind all configurations need to be thoroughly tested before taken into production.

Business wise these are annoying tasks, as traditionally none of these tasks are actually marketable, these are so called "hidden" features people normally does not care about, unless they break down, but it's the marketable things which brings in the business to pay for all of this.

See all the juicy details at http://pulsedmedia.com/super100-seedbox.php

One of the so far unanswered question is upgrades for the dedicated servers. Answer is yes and no.

We need to have the HW configuration as identical as possible, so that limits the upgrades and additional options.

For the 16G model you can upgrade to 1Gbps at 50-60€, which will likely require a reboot to another hardware node.

You can also upgrade storage, at this moment not sure can we expand the base storage, or does it need to be delivered as another virtual device. It's likely we can create a system where the existing file system can be expanded, but might result in some downtime as the data is migrated.

Additional storage upgrade likely prices are going to be:
1Tb: 12€
2Tb: 20€
4Tb: 30€
8Tb: 50€

At that pricing you could have 10T servers with as low as 75€ per month! Making it absolutely the cheapest high storage solution in the market. With the 16G it will make 16T at 105€ a month, or 1Gbps 16T at 155€ a month!

You can also migrate directly to better hardware at your will, say you got the 4G model and want to upgrade to 16G model, that will result in a reboot and at worst case a little bit of downtime as the storage is expanded. Of course depending upon do we have hardware nodes available at that point how fast the migration happens.

Initially we will have very limited quantities of servers available as we already have reseller requests for more than 300 servers.

We have been giving thought also to new seedbox offerings, and these are promising. With introduction of strict traffic limits on torrents (FTP, SFTP, Shell not counted) we can do something quite special. The base offer will likely be hosted on a Core i3, 16-32G Ram.

The entry level plan could be:

750Gb Storage (non-burstable)
3Tb traffic limit (4xStorage)
1Gbps Up + 1Gbps Down
Fully redundant storage, optional backup features
9,99€ per month.

Storage upgrades priced at: 750Gb 5€, 2Tb 10€, 4Tb 15€
Traffic upgrades: 3Tb 9€, 6Tb 15€, 15Tb 30€, 30Tb 50€

These are just preliminary maths, and we reserve the right to change these at any time.

Initial offering will be monthly with setup fee, and longer cycles without setup fee. Initial offering will also be limited to 300 accounts.
With the upgrade options it becomes flexible as of what you need most.

Shared server seedboxes will be load balanced quite efficiently. We want to develop a transparent migration system from server to server to make sure performance stays top notch for all users.

Semi-dedicated offers (4 users max per server) are not yet verified, but likely hosted on Atom N2800, 4G Ram, with following specifications:
1Tb Storage
5Tb Traffic limit*
100Mbps Up + Down
Fully redundant storage, optional backup features
9,99€ per month

Actual traffic may vary depending on the level of congestion on that particular volume network switch on the semi-dedi offers (based on upgraded PDS14-4G).

We will have likely have configurations like these, or very close to these, with the following prices.
All of these servers will come with unlimited* traffic. Please note that when we offer 1T of storage it's actually 1024G not ~917G.

PDS14-2G
Atom D410
2G Ram
2T Storage
100Mbps Bandwidth
25€ per month

PDS14-4G
Atom N2800
4G DDR3 Ram
3T Storage
30-64Gb local ssd cache
100Mbps Bandwidth
35€ per month

PDS14-8G
Core i3
8G DDR3 Ram
4T Storage
30-128Gb local ssd cache
100Mbps bandwidth
45€ per month

PDS14-16G
Core i3
16G DDR3 Ram
6T Storage
60-128Gb local ssd cache
100Mbps
55€ per month
1Gbps upgrade +50-60€ per month

It is likely we will have the 2G, 4G and 16G models from start. We intend to keep as many units as possible on certain types to drive efficiencies of scale into play.

The local ssd cache we'd like to do but might not be on initial release, and rather added later. There is security concerns in cleaning the data from ssd cache drives and thus their endurance. We need to develop a secure erase system which does not unnecessarily wear the drives.

Bandwidth has been set so that the fair amount per server is 30Mbps+ at 100Mbps, and at 1Gbps 150Mbps+ at minimum. We will maintain at least this amount of bandwidth capability per server on datacenter scale.

It is also possible that monthly and quarterly signups will entail a setup fee initially.

Scalability?

Initially we will have room for roughly 1000 servers, with an IP block of /21 (2048 IP addresses). We can rather easily acquire extra space for upto 5000 servers at this location.
We will initially have only 1 fiber pair connected, installations of new fibers will take several weeks or upto a month.
We are acquiring an AS# so we can rather easily add peering and multi-homing during 2013/2014 winter.
We intend to acquire connectivity separately clearly intended for nordic and eastern (russia, asia) connectivity, this is already in the plans.

IP addresses will be the biggest limiting factor, if nothing else, we have the potential to use PA blocks from our upstream providers.

Scalability storage wise is infact constantly more and more efficient, as we reach more scale we can drive even more performance, even more reliability and even higher cost efficiency through the system.

Launch?

We are expecting to be online by 15th of June. Initially we will bring in our webservers, our own seedbox servers etc. and limited scale testing dedicated servers. We are hoping that by 1st of August we can start delivering dedicated servers at a scale, and hoping to fill this room by Spring, 2014.

Resellers

We will also do a clearly defined reseller guidelines at this time.
These are what we are considering right now, and will closely resemble what we intend to do:

Bronze level
5+ servers
300€ in monthly fees
5% discount

Silver level
20+ servers
1000€ in monthly fees
10% discount
Free SMS server monitoring

Gold level
50+ servers
2000€ in monthly fees
15% discount
Free SMS server monitoring

Platinum level
150+ servers
4000€ in monthly fees
20% discount
Free SMS server monitoring
Direct phone line concierge support
Skype personal support
Reseller personalized statistics monitoring page

Sometimes the rsync only runs for a few seconds before DC firewall drops the connection.

DC has not done anything to this so far.

We have now stopped new migrations, and those we are still attempting to transfer data might be cancelled and back to old servers. This situation is completely unacceptable by the new DC.

This isn't the only issue we are having with this DC, there is a myriad of billing and management issues as well. We are hoping that when our own DC is completed migrations out will be fast and effortless.

Simply several rsyncs are causing their over zealous anti-dos measures to cut the connections. According to them. According to the DC inbound data rates were much higher than could be.

This is their explanation, and we are baffled how could 100mbps server send that much faster, and especially over TCP connection.

This is completely new kind of issue, and it's most likely the cause of the new DC.

We are working on this, and new migrations have been halted until this can be solved. It does not always happen, but lately there is now 3 different servers from which data migrations are near impossible.

If you have been migrated, and still not seeing all your data, please open a ticket.

We have reduced the price for long term signups on the Value series significantly. You can view the new pricing at http://pulsedmedia.com/value-rtorrent-seedbox-100mbps.php
All monthly rates remained the same. This price change reflects the acquired lower pricing with upfront investments.

All pricing is grandfathered and this is in effect only for new signups.

We have accepted bitcoins for years, initially manually then via OKPAY, but now aswell via Bitpay for faster and more efficient service.

You will see a Bitcoin payment option on invoice views and checkouts.

OVH has decided to add Apache2 into the Debian 6 template in a manner that it cannot be cleanly removed. Init files, binaries, configs everything will remain even after removing the debian package.

So in order to install PMSS you need to first:

/etc/init.d/apache2 stop
killall -9 apache2
rm -rf /etc/init.d/apache2
rm -rf /etc/apache2
.
.
.

To clean apache2 out of the way. Otherwise the apt line with Lighttpd will fail, and subsequently rTorrent compilation.

There has been widespread discussion about speed capping, throttling and shaping in the OVH network as of late, they silently introduced new rules and regulations, which basicly says 'great bandwidth, unless you use it'. They have specifically mentioned that seedboxes are free from all bandwidth guarantees etc.

It looks like some are already being hit with this. However, this does not affect Pulsed Media and is not expected to affect. Value series doesn't have a single server in OVH DCs for example.

Never the less, we are putting haste in to our plans to make certain there will be no OVH seedbox servers by end of the year remaining. This process has already started months ago, and we will keep going on with our long term plans.

Now you can opt for PMSS to be setup on your dedicated server FREE OF CHARGE!

This includes full torrenting system supporting multi-user, and even pulsedBox is included!

Read more about PMSS at: http://wiki.pulsedmedia.com/index.php/PM_Software_Stack
Ready for a server? Check out PDS series at http://pulsedmedia.com/personal-dedicated-servers.php or 1Gbps lineup at http://pulsedmedia.com/1gbps-dedicated-servers.php

Everything has been solved now, and should function as expected. DNS Flush for some might be necessary.

EU customers, those who have to pay VAT got their service price lowered just enough that with the new VAT price remains the same, no change.

In future orders however the price will not be discounted.

I'm really sorry about this move by Finnish Goverment to raise the VAT price, but ultimately, we cannot do anything about this.

I'm happy to announce that migration out of 2011 servers is pretty much complete now :)

Last few data transfers are going on, but otherwise completed. It was heck of a project to do all this, and many thanks to the users who made their pick upfront and thus easing our job.

It got costly to us - more servers and more in cost, while almost all users moved to cheaper plans, but such is the life of a provider. We are just happy the troubles are now over and we can concentrate again making our offerings better for our users.

For a rather low fee you can get a dedicated drive seedbox with 1Gbps connectivity!

See all 2012 offerings at: http://pulsedmedia.com/1gbps-seedbox-2012.php

The providers on which 2011 servers are is extremely hard to keep online, because providers with 4 disk 1Gbps 100Tb/Unlimited server offers tend to extremely hard to work with, whether from NL, UK, Germany etc.

This time some servers are offline due to couple DMCA claims and this provider is not allowing us to even check up on the server and demanding we file a counter-claim or terminate user(s) in question - in the past many of these DMCA requests has been bogus to start with.

This is an european company called Swiftway - we have been arm wrestling about which law takes precedence within the Europe: US or European law for the past year.

We've always handled all DMCA and copyright notices as per European and Finnish law as we are a Finnish operator and services provider. You can read about Finnish copyright laws from http://en.wikipedia.org/wiki/Lex_Karpela and http://www.finlex.fi/fi/laki/kaannokset/1961/en19610404.pdf or by googling "finnish copyright law".

Because Swiftway does not give us legal options to handle these requests, or at all we do have to drop them - Unfortunately this is the last provider with 2011 series type of servers, and replacement of the servers is very hard as typically only 2 disk systems are available from providers at the price range we are looking for. Some places do offer servers which could work as direct replacement but does fall very short on other areas (such as basic maintenance of servers) to provide the quality we seek for. With 2011 series we simply cannot lower the disk quantity, nor raise the cost per node.

Some of the trouble with the providers is that we require that we can use the service to the full extent and they are not faulty - it's odd to find that many providers are unable to make basic server installations to the spec with software raid, and other oddities which we find to be ultimately basics.

The cost limitations are another limitation to the choices of providers - which tends to favor those providers which compete with price - not service.

So if you are still on 2011 plan, we do urge you to choose an upgrade, up-to-date service. More information will be available within the week how long the 2011 series will be available etc. and i do hope we can amicably resolve the current issues to bring servers in question back online.

Payments made before 1.1.2013 will still go with old VAT rate, but VAT will increase at the change of the year.

Unfortunately there is nothing we can do about this tax increase :(

We will inform later how we intend to handle the VAT increase regarding to existing accounts and payment subscriptions.

This has been now fixed, it was a simple human clerical error compounded with some server crashes along with supplier management system maintenance preventing us from taking correct action. All sorted out now.

However, if you had an payment during this period (Paypal subscription) please check it got accounted for, if not please open a ticket.

We are really sorry for any inconvenience this may have caused.

New range of dedicated seedboxes has been unveiled!
All servers are now 100Mbps Full-Duplex dedicated link and unlimited traffic!

All have dual drives, 7200RPM S-ATA. Optional one time setup fee for lower monthly rate.
Now always with full PMSS license, including pulsedBox and yet to be unveiled features.

Prices ranging from 47.99€ to 74.99€

Check the new servers out at http://pulsedmedia.com/managed-dediseedbox.php

The new line of PDS offerings has been revealed! Check them out at: http://pulsedmedia.com/personal-dedicated-servers.php

Now starting from just 16.49€ per month, and first dual disk option for just 28.99€ per month!

Next PMSS update will do some major, long overdue refactorings in key parts.

If you have your own server utilizing PMSS there is several things you need to keep an eye out for:
Cron jobs are all together changed and this may result your cron jobs being overwritten during the update, so please setup your custom crons to /etc/cron.d directory.

If it is an OVH server with Debian 6 your kernel will get swapped to Squeeze repo kernel as not all features works with OVH kernel. You can change it back by editing /etc/default/grub and setting default to 0 and then executing update-grub. If repo kernel is present no changes to grub will be made.

Some major backend upgrades are present in terms of logging and ensuring quality of service.

NOTE: This update has not yet been released - this will happen soon. For Pulsed Media series a usual type of rolling update will be done.

As usual, you can see the full changelog at: http://wiki.pulsedmedia.com/index.php/PM_Software_Stack

PDS series will update this week, this is last chance to get higher CPU spec models 16G and 24G.

The new offerings will come live by end of this week and the higher end models will have a CPU downgrade, no 24G model will be available anymore. New 16G model will be cheaper, and with dual 1tb drives but with just a i3 CPU instead of i5. The new 16G model will be the best available.

On other hand, low end gets an addition with mini 2G featuring just a 500G drive, but at a lower price of 13.99€ per month using Celeron or ATOM CPU. Standard 2G will be an ATOM cpu always from now on and with 1Tb drive, same price as before.

These are changes in our supply chain and we do not have power over those decisions.

We had to unfortunately stop sales of the current dediseedbox offerings.

New options will be introduced soon. We are still considering the range of options we want to offer on the renewed managed dedicated seedbox series. We have options to consider from 30€ a month all the way to more than 1000€ a month per server.

A very high value seedbox has been made available! These use very strong AMD Opteron servers with a sweet 16Gb RAM allocation!

Extremely high disk quota makes this a very sweet deal!

For just 11.99€ Per month you can get 230Gb disk quota and unlimited traffic! But when you look at the high end, you can get 610Gb for just 20.99€ per month! Now how sweet is that?

Check the new offers out at: http://pulsedmedia.com/value-rtorrent-seedbox-100mbps.php

A lot of chances will be happening in the next couple of months, with new suppliers and new datacenters, along with planned software upgrades. Our plan is to do a overall update in everything Pulsed Media, lean out the operations, make things better and more stable, thus able to offer ever greater services.

New basic seedbox plans are to be released during this month - exactly when we are not sure, still pending some final checks on new supplier and new hardware, and fine-tuning everything for the optimal BitTorrent experience.

After that we will concentrate hard on our 1Gbps offerings line, planned updates are to stabilize and lower operational costs, while providing better performance.

More about these will be in upcoming newsletter, with some more information.

We do not allow account sharing, unfortunately, some people do not listen and still shares accounts. Due to that we have to now build monitoring for it, and a custom tailored malware scanner, along with global blacklist.

In the past weeks there's been several instances of malware destroying customer GUI, it's always specific user, and did occasionally reoccur after rebuilding user space. It was finally determined this is due to account sharing in some of the cases, seeing tens of different IPs accessing those accounts, leading to leaked FTP password.

The index.php is changed via FTP, and that is most visible, but also few other files are being changed to inject counter.php malware.

If you are sharing your account with others, please stop. Contact support immediately to change your password if that is the case.

NEVER post your login details on any forum or chat, and NEVER give them to any 3rd party.

People should familiriaze themselves with IRC etiquette and how the medium functions.
Good places to start is:
http://www.livinginternet.com/r/ru_chatq.htm
http://mriet.wordpress.com/2012/06/21/proper-irc-etiquette/

So in short:

• Be polite
• Be patient. IRC is not an instant medium despite it looks like it (See IRC Idling)
• Do not send personal messages to obviously busy people (ie. PM Staff), they tend to hate it
• Don't ask your question and drop off 3 minutes after: You will never get an answer like that (See Be Patient). Most people do this, we are not watching 24/7 every single minute of the day if somebody happens to ask a question.
• Ask your general questions in public chat, if it's a private/per account question, ticket is the only proper way
• PM Staff gets constantly the same basic questions again and again, worst of all: In private messages. Ask in channel and/or see KB/Wiki. SEE the two points above this.
• IRC is to hang around and to chit-chat with fellow PM users, not meant as the official support channel. Altho, most likely you will get your questions answered there if asked in public and you are patien. SEE ALL above points.
• Most people keep IRC open 24/7 and checks what's going on a few times per day, or maybe even just every other day.

As usual you can check the full changelog at: http://wiki.pulsedmedia.com/index.php/PM_Software_Stack

We have now updated ruTorrent and added new plugins, highlights:

• Screenshots plugin added and ffmpeg added to pkg lists
• getfile plugin added
• Welcome page quota parsing has been fixed for LVM based setups

This is exactly the reason why we do rolling upgrades - as usual there are problems.

The problems are not showstopper level however - mostly inconvenience, and patch is scheduled to be developed and begin distribution tomorrow, 21st of September.

We have upgraded PMSS to utilize newest official release of rTorrent yesterday. Few servers have already been updated, and we are doing a rolling update as usual to catch problems our QA didn't catch. As per usual, there was some new things broken with rTorrent on this upgrade, but these were deemed to be so minor relative to the enhancements that we decided to go ahead with the upgrade.

This new version does support magnet links without a problem, loads them without a problem and not causing a crash. Earlier version magnet links worked only occasionally.

Please report if you notice any trouble after rTorrent version update on your server. You can notice when the update has been done when ruTorrent bottom bar will update to rTorrent 0.9.2/libTorrent 0.13.2.

If you haven't noticed you can choose payment method by viewing the invoice and using the drop down to choose which payment method to utilize.

We currently accept: Paypal, OKPAY, IBAN Wiretransfers

We also accept Bitcoin payments manually, just open a ticket with billing if you want to use bitcoins for payment.

In question is too high MTU value, this is quite easy to solve with simple single command in command line.

Unfortunately this cannot be set server side, and we will be looking at how to fix this directly server side, but for now a quick workaround is to correct the MTU value to 1250.

Click start button and type in the field "cmd", you will see a black console icon, right click on it and choose "Run as administrator".

You will then see the console and there type in this command:

netsh interface ipv4 set subinterface "VPN Connection" mtu=1250 store=persistent

Replace VPN Connection with what name you used for the VPN connection, you can also view the connection with:

netsh interface ipv4 show subinterfaces

Due to OKPAY making it practically impossible to get our account verified with ridiculous demands, only way to get registered with any sensible cost would have been to incorporate in United States.

The verification process is so far the most hardest which is impossible to satisfy as a Finnish company. Verification process which we thought to be hard before seems child's play compared to this one.

Therefore we are dropping OKPAY support for that reason and several others:

* Hidden premium on Bitcoin payments
* WHMCS Module is broken and automatic accounting of payments does not work
* The actual fees are rather high

On the plus side their customer service is actually reachable, shame they choose not to be helpfull.

New alternative payment methods will be introduced in near term.

One such alternative was Bit-Pay but their customer service is nearly inaccessible and we found within a quick review multiple potential fatal flaws in their WHMCS API implementation. We are not sure does these problems extend to their internal system. As a customer we noticed several usability flaws in their system. Due to the security concerns we opted not to utilize Bit-Pay.

Payza, formerly known as AlertPay we are going through the verification process and will implement after we ensure complete usability.

Today's attack was an rather old method: Mail bomb.

Curious part is that how it passed google filters, as we use for support emails and such google for proper spam filtration. Not only did it pass Google's filters, search could find the threads but in-built custom filters didn't? Thousands upon thousands upon thousands of e-mails in very short period and neither Google nor Yandex had any filtration for such an occurence.

Further it stuns me how WHMCS lacks any protection what-so-ever too, WHMCS not only checks on these bombed mails, then sends a reply, but POP processes keeps on piling up: There is no locks, there is no "alive"/"already running" methods, nothing to check if the e-mail import job is already running nor even rudimentary filtration, ie. threshold of e-mails from single address/domain/ip.

We'll need to code our own layer on top of WHMCS e-mail import to add these checks to avoid those processes from piling up.

The end result of this mail bombing was that the system was overwhelmed with the quantity of processes, which in their turn overwhelmed mysql. All the errors and e-mails then resulted in system partition filling up and further stopping from anything working.

What stuns us most is that this 90s style attack can still work today, with all the work against spam, with all the tools available to us, it still gets through and is viable attack vector. To stop things like this we specifically chose Google as the e-mail host for pulsedmedia.com domain accounts, thinking their filtration system is so excellent and they have a lot of hardware online to stop even the largest attacks of this kind. This attack was rigged for exponential growth by solely utilizing yandex, google and our resources.

The situation is now alleviated and some failsafes has ALREADY been implemented, but more will be implemented in near future.

We are very sorry for the inconvenience this may have caused to you.

We do not usually show these kind of metrics, but they are indeed followed and monitored :)

We took a small random selection of 2009+ Starter and 2012 series servers, and here are the graphs. As you can see the very high value of our services and the stable performance. Some of the variation is due to users changing on the servers, and these graphs do not show short bursts as they are per week or per month.

2009+ Starter

2012 Series

That's quite a great earning rate, especially when it's recurring and also from further sales of the referred customer.

So try it out now!

Minimum withdraw amount is 40€/AUD/USD, which you may get as credit or as a cash payment. You can thus utilize to renew your services with this or take it as a cash. Affiliate payment delay is only 15 days, so you get your earnings quite fast after a payment.

Please note that if you are using SEPA wiretransfer to pay your invoices our bank account has changed to OP Pankki, and you should use this account in future.

We offer an affiliate system with 10€ signup bonus, 40€ withdraw limit and 7.5% commission from recurring payments!
It doesn't even need to be your referred sales, as long as the customer signed up via using your affiliate link all his or hers further orders will also give you 7.5% commission! That's quite cool!

Assume you refer total of 10 guys, which buys servers and high-end seedboxes for an average monthly charge of 30€ that gives you 22.5€ per month efforless, recurring income!

But if you are a serious website owner with say 500k unique monthly visitors, if even 0.5% of these guys purchase, even more from the lower end for 20€ average that is 2500 sales for a wicked awesome 3750€ monthly recurring revenue! Now *HOW COOL IS THAT*? Just imagine if your website also happens to be for target audience, techies, P2P afficionados etc.!

Payouts via PP or SEPA wiretransfer upon request monthly or twice a month.

Extra support for serious referrals, such as creating banners, campaigns, promotions etc. to further increase your earnings.

We currently have plenty of PDS-24G servers with 1Gbps network connection & speed in stock, all of which also has upgrade on Xeon series CPU.

Price is 79.90€ per month.
Get yours now!

We are now at final steps for testing of next rTorrent version to be used - we are choosing between official 0.8.9 and a few updates newer repository version with some bug fixes, trying to choose the most stable version.

8th of October, 2011 version had issues with autotools, while official release from 4 months earlier has these working, but there is important bug fixes between the two versions. You can see the change log at: http://libtorrent.rakshasa.no/log/

Final decision will be done by end of week, and some Debian 6 specific updates will be done at the same patch. Servers with rTorrent 0.9.0 will be prioritized on patching, and it will be a rolling upgrade, few servers at a time only updated like usual.

We are starting to deprecate Debian 5 support and looking forward to targeting Debian 6 only. It's not big changes, just few paths, and couple extra steps during installation which most be done for Debian 6. We are targeting that by mid April Debian 6 is the primary distribution for PMSS.

During last month's patch we opted to use the latest rTorrent version available, 0.9.0 with libTorrent 0.13.0. What else would make sense to choose?

Little did we know that many trackers do not accept the latest rTorrent version, autotools are broken with it among other things.

We have attempted to change it back to 1-step older version, which does compile just fine, but does not work. It is missing some code required for it to work, thus upon start crashing on an error.

We are now considering the option to downgrade all the way back to ~1½ year old version, or start testing different SVN version to find a version which works.

If you can suggest a version combination which actually does work as expected, please do not hesitate to contact support and let us know your opinion. After we have chosen and done Q&A testing for new chosen version it will be distributed to servers within 72hrs.

The problems caused by something as simple as this is the reason why we opted not to push rTorrent updates for a very very long time. Past experiences with rTorrent updates has shown things get broken left and right in each update.

PulsedBox has been updated to work with 2012 series offerings now, so you can use it conjunction with 2012 servers.

During this time we had to do some DNS debugging, which resulted in a short downtime for pulsedBox logins. Entries not found resulted in redirection to search.com via adding .com at the end of the query, for certain tools only. Seemingly causing a DNS poisoning attack. In the end it's OS resolver, which without a search line in resolv.conf adds the .com in the end, not bind itself. This was not happening on any other machine so it was quite curious and took a while to find out why this was happening.

Some of our customers have a lot of IPs, and some of these require reverses set for all their IPs. Now you can submit to us a list of reverses you want to be setup, even for hundreds of IPs at once.

We have just updated the stock status for various services.

For some services we are considering adding a lot of capacity by early next week.

Deliveries currently are slightly backlogged as new servers are being setup.

2009+ Starter series all servers are full, same goes for 2012 (Except 1 vacant 2012 Large slot) so these services are backlogged until new servers has been setup and tested. Normal schedule for these should continue by end of week.

PDS series we got only a few spare before next batch of servers arrive later this week. We are hoping that by mid-march we can again deliver PDS-2Gs within 24hrs as in the past.

Ending US Premium VPS Line

US Premium VPS line of services was never with high demand, therefore it will be discontinued, and sales for the series has been stopped. Migration of existing US Premium VPS to the PVS series has already begun. Servers will be shutdown by latest 1st of April, if not 1st of March.

15 servers has been reinstalled, 18 has been confirmed to been affected in total, so only 3 servers remain to be reinstalled.

If your server was one of those affected according to our lists, a ticket has been opened already for you. You will receive e-mails for new login credentials etc.

Also, this does not affect seedboxes, vps, 1gbps servers, and newer personal dedicated servers. Potentially affected services are only PDS series and dedicated seedbox series, to a limit. Access was not granted to all servers with this tech administration account.

Only 15 servers has been confirmed to have been affected by this so far. 8 of these has been already reinstalled + new logins e-mailed. Remaining 7 are reinstalling currently. A few is having issues with netbooting so they are delayed and being inspected by DC technicians.

So according to currently confirmed status the reach of the breach was a tiny fraction of all servers, very serious, but fortunately just a small subset is affected.

Please contact support if you suspect your server has been affected and if your server was affected but you've not received new details within 2hrs.

We have an discreet account for tech admins for our dedicated server customers, this way our dedicated customers have limited access, discreet logging, and important messages can be easily forwarded to our helpdesk.

Unfortunately, this e-mail account also had e-mail forwarded to the same ex-employee who took action in late November against us, his forwarded was the only one. This allowed the attacker to change login password for this tech account. Access was made 14:03 GMT from IP 123.243.98.195, and access again limited at 15:50 and all servers taken off from this tech account by 16:30 GMT.

Unfortunately during this time he had sufficient time to put several dozen dedicated servers to be reinstalled. Restoration has already begun for those dedicateds which has been reinstalled, but this process is going to take a little bit of time. So far it seems roughly 20 servers were reinstalled.

Restoration will be done as swiftly as possible, but if in case is a dedicated seedbox or windows server, those are very slow to reinstall.

Please contact support if you suspect your server was a victim of this attack, the process is first to do a hard reboot if there is no reinstallation ticket on file, and to verify the state.

All users affected will be compensated for the loss of service time.

We have made an bug fix release today, fixing the most notable bugs.

• Torrent creation on newest ruTorrent
• Quota plugin on newest ruTorrent
• Recycle bin for ajax file manager

Dediseedboxes ordered after 9th February afternoon will be unmetered traffic @ 100mbps. Existing dediseedbox owners can open a support ticket to have their server updated to unmetered traffic at the new price.

Those dediseedboxes which have 1Gbps link will be capped at 32Tb upstream. Downstream and internal will remain unlimited.

View all the dediseedbox offers if one suits your needs! :)

Some of these are older, but results in tickets (patch planned), but due to ruTorrent + rTorrent update of yesterday there is several new ones. Only a small number of servers have been updated to new ruTorrent + rTorrent.  We are working on all of these, but fix(es) cannot be done per user, or within minutes, but takes development effort. Time for this is scheduled for the upcoming week.

rTorrent + ruTorrent is by far superior software for this usage type however, but maintenance hurdles are painstaking, and upgrades on our scale are a big hurdle, not only because usually broken backwards support but also due to introduction of plethora of new bugs.

• Browser cache has to be cleared manually due to updated JS in some cases
• Torrent creation from ruTorrent is broken for unknown reason
• Remove + Delete on ruTorrent doesn't work in all cases
• Traffic plugin is broken for some users, but not all
• ajaXplorer data deletion results in "Permission denied" in most cases
• Quota display plugin broken due to new ruTorrent

PMSS Has been updated as of today, with a great deal of new features and upgrades, some very highly anticipated ones.

• rTorrent + ruTorrent patched to latest release
• mediainfo added
• php5-geoip extension added: flags etc. on peer/downloader list
• New ruTorrent plugins: cpuload + theme and others!
• Resellers: Preliminary work for whitelabeling

So if you are experiencing packet loss, or lower than usual transfer rates currently, please submit a ticket with traceroute and ping results, along with what you were expecting to see. We will submit these to the DC for inspection. We also need to know who is your ISP.

Thanks.

Downtime shouldn't be greater than 1 hour on these servers, and affects only a really small fraction of servers (handfull).

• SSL Support for web interface, simply change from http to https
• Webdav via SERVERNAME.pulsedmedia.com/webdav-USERNAME/ ** Warning beta, your mileage will wary. Will not work with windows integrated client, for windows use BitKinex, Webdrive or some other application.
• SFV checking via cfv
• Removed SSL key creation from installed (Deprecated method)
• Update creates automatically a certificate for SSL

These offer far more per euro than the 2011 series, and 2012 series servers offer far more CPU power and RAM than the 2011 series.

See them at: http://pulsedmedia.com/1gbps-seedbox-2012.php