Electronics Weekly Electronics Design & Components Tech News
How big a deal is the ARM move on servers?
The Cortex A-15, which can run at 2.5GHz on a high-performance 28nm process, is aimed at servers.
The A15 can have up to four cores, and all cores in the processor are coherent in that the same single operating system can run across all four.
Changes to the A15 bus allow coherency to be extended outside the cluster of four cores to multiple sets of four cores.
To make it more attractive to server firms, the A15’s addressing range has been extended to 1Tbyte compared with 4Gbyte for the A9.
Then there’s Smooth-Stone, the Texas start-up backed by ATC the Abu Dhabi investment company backing Globalfoundries, plus TI, ARM and a number of venture capitalists.
Smooth-Stone aims to: “Completely remove power consumption as an issue for the data centre,” says Smooth-Stone CEO Barry Evans.
There’s a big incentive to do this because a typical server uses its own
value in power and cooling every two to five years.
Google has a million servers, Amazon and Microsoft both 250,000 plus each, IBM, eBay, YouTube, and the BBC have over 50,000 each, according to Future Horizons.
However, Future Horizons is pessimistic about ARM’s chances.
‘Dashing the hopes of many server-chip wannabes, we see the vast majority of
servers still powered by processors using the Intel instruction set despite the use of
languages such as Java and Python’ says Future Horizons, ‘whilst this sounds nice in theory, supporting multiple versions of server programmes across thousands of servers is simply far too complicated for most IT departments to manage.’
News just in: Google pays its engineers an average salary of £128k p.a. Loading on the 30% (from Bureau of Labor Statistics (2011)) for benefits & taxes, this gives a 2012 average salary of $166k.
As you can see from:
http://www2.electronicproducts.com/Salary_report_Google_pays_its_engineers_the_most-article-fajb_tech_salary_report_oct2012-html.aspx
Google pay the best. Admittedly this is for Software Engineers, not IT speciallists, but CW Jobs seem to thing software engineers are better paid than IT analysts, at least in the UK.
Very interesting discussion – which just goes to highlight the point made in the discussion of “Nice Core ARM; Pity About Intel MDF.”
Basically, the server market is a different beast, with a different business model, and a different ecosystem and ultimately a very different type of customer.
I question whether ARM really has what it takes to sustain a thrust into this sector. Even if it does, what cost to their core embedded/mobile markets?
Brave move – but just because Intel is (and can) chase after slices of ARMs market does not make the opposite thrust from ARM sensible.
Thanks Rupert, interesting stuff. At least they’re talking the talk on energy efficiency. But the walk has not – yet- turned out to be be that impressive and I always wonder, with all their resource, why not.
I was in a round table with Justin Rattner last week at IDF, and someone asked him a question about battery technology and portable devices.
Interestingly, most of his reply wasn’t about that. He concentrated instead on memory interfaces, saying (and I paraphrase):
“Memory interfaces today are dreadful. They consume way too much power. There are one or two orders of magnitude difference between on and off chip memory access. And it’s terribly inefficient, you access thousands of bits and you only need to check one or two. We have developed some early thoughts you’ll hear more about: we were hoping for the Solid State Circuits conference in February next year, but it’ll be later now. If you operate close to the transistor threshold – a few tenths of a volt above – you can get eight or nine times the energy efficiency. Can you build a whole chip that way? We haven’t built a processor that works that way yet”
So from that, I guess Intel is planning to at least extend the arguments about energy off-chip, and to make some moves towards new technology there, although whether anything practical will hit before 2020…
(Last time I went looking for millivolt logic circuits , I found some amazing things – including an abandoned proposal for a truely awesome cryogenic device based around tesseracts of Josephson junctions. Superconducting logic can work at the 10mv level, even if there are a few small practical problems.)
Mike – I have seen data from HPC contract tenders, but obviously can’t share it – sorry! A bit different from server farm overheads I’m sure. The rest is guess work as you have rightly sniffed out!
On the assumption that ARM consumes no power – no I have allowed for small sips of the stuff:
All the power going in to a processor comes out as heat. Assuming it is cooled by an air source heat pump (standard air-con) – which isn’t always the case, but is usual – you get 4x (self-builder’s rule of thumb) as much cooling as power you put in – so if it is a 200W processor you need another 50W of electricity to run the aircon to remove that 200W.
Because of the need for cooling that 33% power on processors becomes 41.25%. If I assume ARM to be 5x more frugal for the same processing performance we get 41.5/5% power consumption for ARM – 8.25% of the server farm power then goes to keep the processor running rather than 41.25%, which is a drop of 33%. If your 33% had included cooling it would have been 33/5% or 6.6% to keep the processor running, a drop of 26.4% which I rounded to 1/4 because everything else is a guess and false precision would only give the wrong impression.
Re total cost of employment – is the work so dull they have to pay so well?!
Really looking forward to seeing your analysis – I’ve only glimpsed the smallest part of the picture and it is amazing what a proper overview turns up!
33% is the percentage of power used in the processor. Aircon is obviously an overhead on anything that disipates heat.
The $220k is the total cost of employment of full time staff, not contractors.
On your last point you seem to be assuming ARM consumes no power. On equivalent actions, an ARM only saves a proportion of this power and indeed if the peak processing power isn’t high enough to meet the requirements and the task has to be split across two servers it can easily increase the power rather than decrease it.
I am currently researching exact numbers on the power used by Intel, AMD, ARM and SPARCs on a variety of server applications for publishing in December so if you have any data to add to the mix I’d love to see it.
Thanks for that Mike … I need to tidy the assumptions up:
When you say 33% of power consumption do you mean 33% accounts for the watts going in to the processor, or 33% accounts for the (watts going in to the processor + watts going in to the air conditioner to remove the waste heat from the processor)?
And by the same logic that Google won’t be paying $5k for a server, they won’t be paying professional consultancy rates for IT monkeys, they’ll be paying top tier graduate salaries + options, and ramping the ones who prove themselves worthwhile, so I think $150k is a reasonable mean. Or do they work differently from most of the tech giants?
A quick way to short circuit the need to these assumptions would be to find out what any computer centre operator pays for power, then divide by 4 – (if yout 33% includes getting rid of waste heat) or 3 (if it doesn’t) and that is approximately the cash they could save on power by going to ARM. Is that a bigger number than the cost of going to ARM? Torben seems to indicate that it would be a significantly bigger number for most customers.
As for whether it is a priority for ARM or not is a different matter.
If any one is paying $5000 per server please come and see me ! You can rely on the fact Google isn’t paying this.
The total cost of employment a good IT professional is actually around $220k. Salary is just one cost – you have to give him somewhere to sit, pay employment taxes and the like.
Obviously it is load dependent but in most server farms the processors account for about 33% of power consumption.
And as you say Intel have plenty of room to play on price.
One thing I will add is that ARM will be so busy selling A15 licences to other applications I suspect servers will drop off their radar pretty quickly.
I assume they’ve done the numbers at Future Horizons, but it doesn’t smell right to me:
Assuming a server costs ~ $5000 (I know – hilarious!), Google have 1M, which is $5B invested in servers. If we say they are efficient x86 servers & aircon systems and it takes 5 years to spend that $5B on power, rather than the 2 which is the other published guess. That is $1B per year on power.
Assuming A15 can reduce power by 5x for the same horsepower as the x86 competition (conservative if Atom v.s. A8 is the benchmark), and that the processor uses 50% of the power & produces 50% of the heat of the server (wet finger – I have no numbers to back this up) this would cut power requirements by 40%, a saving of $400M per year.
Say you need 1000 extra smart folk to cope with the difference between x86 and A15, at an average salary of $150k per year (pessimistic I think) that comes to extra costs of $150M, leaving a saving of $250M per year, or more than a $1B over the life of the kit, and the costs of running these heterogeneous data centres will naturally drop over time as it becomes the norm.
I think these numbers are conservative and I’ve not taken into account the effect of having another processor player in the market getting Intel to play fair on processor prices, knocking down overall data centre prices. You might say that just kills the A15 arguments, but the threat needs to be credible to cause price pressure on x86, and a big player could invest in the A15 just to make that happen. And anyway, the cost saving numbers are only for power, not for hardware. The power consumption doesn’t get less if Intel drop their prices, it just becomes an even bigger proportion of the cost of running a server farm!
All the names you mention are big players (except maybe Auntie Beeb, who will not be doing too much investing in technology for the foreseeable) and most have no particular loyalty to Intel – quite the opposite in some cases. Intel really needs these big server players though – it is like Keynes says of banks “If you owe the bank £100 you have a problem. If you owe the bank £1m, it has a problem.” These big server farm guys are such a big chunk of Intel’s profit that Intel’s cost structure is significantly different if they move on. Intel will have to drop their trousers on price if they are to discourage migration to ARM, but if they do this all the fat margin on server processors evaporates and the cost structure changes just as significantly.
Intel has a problem deciding whether to maintain margins or not. They are damned if they do, and damned if they don’t. Either way the economics of running a server farm just got a lot sweeter.
Well actually Mike I found it in Future Horizons’ latest newsletter but I rather assumed it might have originated with you. I will look forward to your analysis in December – didn’t know you’re planning an IFS for then. Good news
Yes, of course, Torben, you are right 250,000 not 250 million. Thanks for pointing it out and thanks also for that much more hopeful scenario than the gloomy Future Horizons prognosis.
Hi David
I see half of the above based on my last IFS presentation. Just to let you know I intend to present a far more complete analysis of the performance of ARM vs Intel vs AMD vs SPARC server approaches at the next IFS in December.
However processor dissipation is only one aspect of power consumption in a server farm so unless SmoothStone are addressing all the issues ($40m of investment says they aren’t) then the cooling towers will still be needed.
“Amazon and Microsoft both 250 million plus each”.
Don’t you mean 250 thousand?
In any case, I don’t share the pessimism of Future Horizons. My university department has regularly run server parks with various architectures and several Unix variants at the same time — they generally have kept the policy of buying what was the best value for money when they needed to replace servers. At the moment, this is x86, but in the past it has been Sparc, Alpha, PA-RISC, 68K, Vax and PDP-11. Many of these have co-existed at the same time running different flavours of Unix.
Compared to those you mention, this is a very small server park, and I think the main advantage of a lean server CPU is in immense server parks like those you mention that run millions of small jobs at the same time. If you can save $200K on electricity and heat per year, you won’t mind hiring an extra operator for the new servers.