MD Platform Development Update 01/2024
We've again made some progress! Below a photo on the latest prototype reaching a production rack.
This is hardware rich development, iterate iterate and iterate in-production, each time making something a little bit better. There's an unbelievable number of little nuances in a system like this.
This just entered production last week, yet isn't even our latest model. That's still on the "healing" bench, waiting for some parts. We got the wrong parts accidentally in stock.
New Challenges Await
One of the models refuses to network boot without display present. Easy enough fix, just took a while to get the "dummy plugs".
Some models which have Realtek NIC has random seeming network performance issues with some targets, upload performance on average limited to about 60% what it should be. Easy enough, just need addon NICs when the onboard one is not usable. It seems a bit random which unit has which. Mounts still need to be designed and made before moving into production testing.
Bios Woes, once again. Bios configs for these can be very nuanced. Issues will be fixed once they arise, no sense to take hundreds of nodes out of the rack each time we find something which should be better or fixed, unless that issue rises for the particular node.
Approximately 30% of the MD models produced to date are currently down due to various little nuanced issues, some are having actual hardware issues tho; Namely we received a really bad batch of RAM and NVMe drives, all of these were brand new but with sky high failure rates. Most are config errors and such however, small human errors. Plan is to fix these during February.
We planned to use PC-ABS from PM for electronics enclosures etc. but unless you make solid part this material is pretty much unprintable, it cannot be bridged at all even within 65C chamber. Either you have bridging with no layer adhesion, or you have no bridging with layer adhesion. The sweet spot is way too narrow to make parts which are not mostly solid. We are probably being over cautious with material choices however. Since these do not need high continuous loads or high stiffness, we will be trying another material.
Burn In Testing; We need to automate this and make a process out of it. Some 7-8" displays are now on order to make many tiny burn in + bios config stations, something to work on after February. Does anyone know of a minikeyboard which simply has arrow keys + esc + F5 through F12 keys, and works universally? Let us know!
Progress Made Over The Past Couple Months
1) New units will have new power cabling, which is faster and therefore cheaper to produce AND more flexible with no specific model tie-in anymore. This power cabling takes couple minutes to assemble per unit, instead of the old one averaging to very laborous 20minutes or so per node. Old power cabling cost was about 3.50€ per node, new one costs about 5€ per node BUT saves about 15minutes of labor while providing much neater and nicer end product. This is a huge win!
2) Power Meter + RJ45 panel and also is better mounted now, earlier affixation was a little bit too weak (photo above)
3) Dual NVMe Drive Models will shortly be available
4) Bios updates (fixes some non-bootable units)
5) Automation is starting to get a long, some further tests has been made and first unit installed through new system done. Working on API for power management.
6) Some parts removed from the platform which were deemed unnecessary and potentially even detrimental to final product.
7) TOOLING! Can't stress enough about tooling. New tooling is pretty much operational and in use now. It makes some annoying portions now outright satisfactory.
8) New rack mounts are in stress testing right now. Just proof of concept, but we want to first see how well they hold weight before committing more resources into design.
9) MD Product Page; The little chart next to price shows if the price is going up or down, but if you hover over it you can see last 30D and 7D average for sale prices too.
10) Dynamic pricing is working excellent, EXACTLY as we hoped and planned for.
11) 19th model was just added, with at least 3 new models inbound during February if there are no surprises.
Tooling in action examples
When the marketplace doesn't have what you need? You make it. This time a HVAC ducting adapter which is eccentric and conical, this exact size was not immediately available and conical eccentric sometimes go for as much as 500€ a piece (seriously!) when they should be 50€ a piece. This cost 20€ in materials, 30minutes in design, less than 1 day in print time each. Immediate need was for 2, but later on more as fans tend to die eventually. Especially 315mm sized ducted fans, they have very weak bearings to their size, no matter of manufacturer. Couple of years per fan. Our first 315mm sized ones cost nearly 1000€ a pop, but now we adopted older tech and they are more like 250€ a piece. They last about 2 years each. The bearings can be replaced however if you have hydraulic press, and spend the time to arduously open the stamped shut casing etc. This job is better left for someone who specializes in HVAC fans tho, so we've just been accumulating failed fans over the years.
It is now quick and easy to make all kinds of custom tooling as well to make production faster. Here we see an very weird network cable organizer. Assembling an MD set takes multiple lengths, so varying width slots required.
Jigs and such has saved tremendous amount of work too.
We've got an 800x800x1000mm printer en route to make HVAC intake/exhaust grilles too. While those are very basic in their construction, the large sizes tend to be exorbitantly high price. We calculated a minimum of 8000€ just for a few grilles if bought from the market. CAD Design takes less than 1hour, materials for all those will probably be around the 150-200€ mark, and printing time in the 1 week range. Another ~7000-7500€ saved in costs with just a little bit of effort.
Material woes? PLA is UV safe with just a shallow paint coat, that stops UV rays. Also even if you don't paint, only the surface layer is affected, UV cannot penetrate the whole material. Temperature range is more than sufficient, intake never gets to 65C, and on exhaust side even with our 8M high DC and taking from the ceiling, we won't probably see 65C exceeded, even tho we are hoping the heat stacks up nicely. What if 65C is exceeded? The PLA anneals, and then it can take 150C, however shape will change slightly as the material crystallizes. Surface annealing is both easy and preferrable in post production, this takes mere minutes and is much simpler than most expect; Just take a blowtorch on the darned thing! ;D The extreme temperature (~1300C) not only eliminates the "whiskers" but actually heats up the very surface to beyond 65C causing some crystallization of the polymer structures, therefore surface annealing the part. Once the surface cools down, it can withstand 150C, and once temperatures reach over 65C the surface can keep the internal structure shape better, therefore more thorough annealing without loose of dimensional accuracy / material shrinkage; At least not as easily. Just don't do this at home! It's very easy, not to just damage the part, but in inexperienced hands you might burn your house down. So please don't do what we do, be safe. Use heatgun or more traditional methods. We work in industrial workshop, with multiple fire extinguishers nearby. Due to extreme temperatures etc. it also needs experienced touch not to overheat sections. It's an acquired skill through multitude of failures.
Now, how about PLA being biodegradeable, doesn't it degrade being an outside vent? No. PLA for fast biodegradation requires high temp industrial composting. Namely, exceeding 65C and preferrably anaerobic. Otherwise it takes a really really long time to degrade. It does, but does it matter if it takes 150 years to do so? Not really. Also since these are on open air, not wet, the typical biodegradation processes don't occur.
Dynamic Pricing In Action
Pricing pressure above right now. First line is the global adjuster, amongst all MD units, the totality of them. Each line below is individual model.
Some models are in high demand they go up, some models in low demand so they go down.
This let's You Decide what the server is worth to You. We are really poor judging what our offers are worth to you, we don't know all the use cases, or how things solve your problem and what it is worth to you, or how the landscape of offers have changed, how do you value our unique take on servers, or our network, what our production capability for new units is etc etc. There are simply too many variables at play to anything else than guesstimate the ballpark. So let an algorithm decide.
The factors currently considered are total number of units sold/unsold, when was last unit sold, what's the average 30day offer price, average price sold at etc.
We will probably add more variables as time goes on, but for now this works really well. We need to finetune the algo now and then, for example we've been out of 4TB models for a long while now, once we got even 1 we decided to manually nudge the price higher, but the algo lowered it back down way too fast via the "Draw to the mean" portion. We had to lower it's power level to allow more flexibility in pricing, and still this resulted in too low price compared to demand :)
There was also a bug which made new model price 0€ because there was no price history :)
Why Power Meters on every set?
Curiosity! First and foremost. But in reality? This is valuable technician tool, you can at quick glance see if there's something obviously wrong, and during final setup the power consumption shows if all nodes booted up right. These show a lot of data, including temperature and total kWh consumed in the meter's lifetime, power factor, voltage etc.
Further, we get manual verification of a sets average power draw over time. This helps us more precisely calculate the Operating Costs (OpEx) of each model of units, which allows us to set lower starting pricing eventually! Once we reach equilibrium of producing more servers than we sell, having precise OpEx data is crucial to set the lowest possible starting price for a new model.
A lot of operators just goes like "350W PSU == 350W Consumption" or "180W TDP CPU + 4x3.5" == ~220W consumption" -- neither is even remotely correct, not anywhere near.
For example, we just noticed a Ryzen 3900X with 6x3.5", 4x DIMM Modules only consuming ~110W in production despite 105W TDP CPU and 6x3.5" -- Meanwhile a 65W TDP Xeon with 4x 3.5" consumes 180W !! Both are 1RU.
We would prefer to collect this data automatically into a database, but the power meters while they have both MCU & serial interface on the MCU -> This serial interface is not exposed. Maybe we will spend the time to see if the serial interface outputs data, but we don't expect manufacturer to disclose any information nor be willing to make custom firmware. We asked for customized version where this port is more easily exposed (other than less than 1mm diameter pads for pogo pins on the PCB) and the MOQ was 10 000 units.
Besides, someone will eventually make that version regardless, it's way too obvious of an value add for so little cost! Only a little bit of design time + extra port / pins on the board.
Also these power meters we got for a really fair price at quantities. Adding one does add a little bit of assembly time, and does take a little bit of power (tiny fraction of 1W, the screen is BRIGHT), and does cost a little bit of money and space on the platform, we think the tradeoff is more than worth it. All it takes is a few times saving technician time on troubleshooting.
Plus it looks cool! ;)
Wednesday, January 31, 2024
