Chatting with Ben Segal - Podcast Episode 12 - Evolution of Computing at CERN
David: [00:00:00] How did Linux become a thing at CERN? If it's a completely bottoms up...
Ben: [00:00:05] OK. It didn't. Linux came later to CERN. Well, Unix ...
David: [00:00:10] Well, Unix came for sure because...
Ben: [00:00:11] Unix had all its dialects and almost all the big development was done with different flavours of proprietary Unix.
David: [00:00:20] Right.
Ben: [00:00:21] Unix came in 1980. Let me see. I just came back from my sabbatical...
Ben: [00:00:32] Probably '82 we had the first Unix machine at CERN. I should remember because a guy was just talking about it. When I came back from my sabbatical in '78, Unix came a couple of years later. '80, '81...
Ben: [00:00:48] And we got a Berkeley flavored one because we had a Unix guru, a German guy. Very good. And he had a good sense of what Unix to get. And I believe, I was told by another guy who worked on this, that Bill Joy came as well early on and helped CERN with it. So that was just one machine at CERN and it was used for cross compiling, just an obscure thing. It wasn't a mainline thing at all. But Unix came slowly and the big drive was with the Cray.
Ben: [00:01:24] And at that point, because Unix didn't have features for running batch jobs that time. I remember very well when IBM was asked - because we were a big IBM shop. Do you have Unix - we could have a Unix? Yes, yes, we've got Unix. And do you have tape support? So the salesman went away and came back. Yes, we have tape support: "tar".
David: [00:01:48] No, no, no...
Ben: [00:01:50] I mean all that of course we developed with Cray. Cray came and Cray understood about batch jobs and NQS scheduling – job scheduling. And Cray understood you needed to schedule tapes and so on. So they, Cray, did a lot of development with the CERN colleagues.
Ben: [00:02:07] And it was in doing that development around Cray's Unix, they called it UNICOS, between '86, '87 and ... '89 that we understood enough about how you can put these features into Unix to make Unix work on a mainframe. And then the idea was as follows. In the early '80's we had obtained, CN had obtained an Apollo, Apollo DN10000. Have you ever heard of the Apollo? It's very important. The Apollo was the first serious workstation from a little company in Chelmsford - it later got bought by Hewlett Packard. I'll never forget this: the guy came in 19... must have been 1980 and he gave a talk with nice transparencies about this new workstation. And this workstation came on a ring. It had a token ring.
Ben: [00:03:04] There was a 12 megabit token ring called Apollo Domain and these Apollo machines were so advanced they could page memory over the network. So you had virtual memory: virtual network memory. In 1980, '81. And not only that. The guy was asked: "D'you run FORTRAN?"
Ben: [00:03:26] "Ah, yes". D'you run this, that?. "Yes. Yes".
Ben: [00:03:30] And people thought it was just just, you know, a snow job.
Ben: [00:03:36] So CERN... It was very impressive, the presentation. But was it real? So CERN sent one of their very best programmers to Chelmsford and ... to see if this worked. And he took his tape, of course, with the biggest simulation program of that time with him, and went off to run some tests. First thing he found out is he'd brought the wrong tape. He'd brought the IBM version instead of the VAX version and the ifdefs for the VAX would be almost right for the Apollo but the ifdefs for the IBM were not. So you had to edit 100,000 lines of code. OK. Now, you couldn't do that at CERN on any machine. Well, maybe with Wilbur. But it was super difficult to do. Certainly had no workstation of any sort yet or even the mainframe.
Ben: [00:04:28] So the CERN guy thought he was finished. But they said: "What's the problem? He said "I need an editor". "What's the problem?". And not only it wasn't a line editor, it was a screen editor. Yeah, it was a beautiful thing. And in half an hour or so, he'd done this. And then he could compile his 100,000 lines of FORTRAN which no-one thought it would ever do and it did it. And the thing worked.
Ben: [00:04:53] Then CERN got interested. Bitmap display, ... it'd never been seen. Okay it was expensive. So CERN bought a few of these. Anyway by the end of the '80's, they were very well established at CERN, the physicists liked them. It wasn't running Unix, it was running the Domain operating system - it was Unix-like. Later they did .. they were very clever. Later they put shells in.
Ben: [00:05:27] They put a BSD shell and a System III / System V shell as well. So you could run two Unix flavors or Domain - very cleverly done. So it was a great hit, this machine. Anyway, they announced their latest model, the DN10000 it was called: four CPU's and a one Gigabyte disk on it, I remember. And I had this idea: we'd seen these things could run serious computing and I had the idea that why don't we try and run batch physics on this thing and see how fast it would do it? The batch jobs were scripts basically, big scripts and they would need data from, you know, they would need to read tapes and things like this. Now the way the mainframes did this, the Cray in particular, they would "stage" the data. So you'd get a job and in the job control you would see that it was going to need tapes. So it would, if it had never seen this tape before, it would order that the tape be mounted. The tape would be mounted and the data would be read from the tape (or the files you needed, maybe all of the tape) onto a disk - this was called staging. Okay. And then the machine, then the job would go into execution with the computation done against the disk and the tape could be dismounted. Yeah. So it wouldn't be slowed down by that.
Ben: [00:06:55] This model of running jobs was standard in the computer center, like in many centers. So I said why shouldn't we do this with an Apollo? And all we had to do is connect it to the Cray. Now, the tapes, I should tell you, the tapes were all on the IBM. The Cray didn't have tapes. But the Cray had very expensive software and channel connections under the floor between the Cray and the IBM. So the Cray could order a tape to be mounted before a job ran, and that tape would actually be staged, mounted on the IBM and staged across, across a channel or whatever. So why not do this with an Apollo? I mean, my idea was to offer the users a service, and they wouldn't know if their job was being done on the Cray or the Apollo. IBM would be different because the job control was all different. But the Cray was Unix and the Apollo was Unix enough ... So HP liked this idea - we'd proposed this idea to HP that we would try this. And HP gave us the top end machine, probably a 100,000 franc machine, and they gave us one guy for three years of our choice. So we chose a physicist from I forget which collaboration, one of the LEP collaborations, whose contract had just expired, and who was the expert on Apollo because the Apollo machines were being used by the collaborations.
Ben: [00:08:27] And he came with us and he did the necessary liaison with the experiment - we worked with the OPAL experiment. I remember going to the operations manager, a guy called Jean-Claude Juvet, a Swiss guy. "Jean-Claude, I need a place to put this machine, this new machine, and it needs to be pretty near the Cray?". He said: "They've just taken all the big round tape drives away and now it's all on cassette tape. Look, all the middle of the floor is empty. Help yourself". So we took a table. We put it in the middle of the computer centre, right under the visitors' gallery. We put our Apollo on it, and under the floor we ran our Ethernet cable to the Cray. And all we had to do was to use "rsh", (rshell you know, get this tape or whatever). And so you could stage tapes just using the Cray. And it worked like a charm. So anyway, so this thing starts working. And when it was clear that it was working, somebody connected it to the accounting system at the centre. After a couple of months, the Director rings up my boss and says, "What is this?". You know what - 25 percent of the whole centre's cycles were from this. It was called HOPE. HP, Apollo, something or other. It was an acronym. "What's this HOPE thing? That's 25 percent of all the CPU cycles of the whole centre". Cray, IBM etc. So my boss says: "Well, that's er, that's our little project with the Apollo. So he starts asking questions. Then he rings me up and says, "Segal, I've heard your machine, you're connected to the Cray. I want to know exactly what resources you're using out of the Cray for this". He didn't like this idea at all. He thought we were cheating or something. How could one machine do 25 percent of all these mainframes? So I sat down. I worked it out. It was 1 percent or half a percent or something just for driving the network. And so I don't know if I called him back or something, sent him a message. I don't know. I didn't hear from him again. And so that was the beginning of the end of the mainframes. Now, OK, it was easy because we were emulating a Unix, a Cray mainframe. But so we sat down and wrote a proposal. Why don't we have a network of Apollos - or any sort of machines as Apollos were actually rather expensive because they were workstations. You don't need a bitmap display to do crunching. And we wrote this proposal. And it had a better history than Tim's proposal, because it was actually looked at seriously and accepted by the Division Head, David Williams, on condition that we could get at least one of the four LEP physics collaborations to work with us and to give us money. If so CERN, the IT Division, would match that money.
Ben: [00:11:32] So we went round... I've told this story elsewhere, but we went round the four collaborations. One didn't know what we were talking about. The other one, the richest one, L3 they said: "We've got enough computing at present". (They had their own IBM let alone, and a lot of Apollos). The third one said yes, as long as it's a VAX, a VAX...
Ben: [00:11:51] And the fourth one said "Yes". The fourth one had just got a new physicist, collaborator, a woman professor from the University of Indiana. And she had some money because when you join a collaboration you have to give some money. And she even, or she or somebody else , they had SCSI disks that they were going to attach to their own system. But we persuaded them to attach it to our system. And so we built a system - actually Silicon Graphics were the main machines at the beginning - and from starting from '91 it took off. And by '96 I guess we switched off the IBM. The Cray unfortunately got booted out before, even though it was our model and our beloved machine.
Ben: [00:12:38] I know what we were talking about. So let's go back to Cisco. So I ordered the two Cisco boxes to filter the Cray network. That was where Cisco came in. Certainly the first at CERN, maybe in Europe. And then later helped Europe, with USEnet etc... Right.
Ben: [00:12:58] So about the Cray and about security. We'd had to sign this agreement with the Americans. Now it turned out most of the big European centers signed the same things we did, but they didn't do anything about it, like Bologna and so on. They didn't actually implement anything. They just signed. Nobody checked that they were securing the machine or not. But we did. First of all we had the Ethernet filtering, the IP filtering, that worked.
David: [00:13:26] Couldn't you spoof ... it's easy to spoof an IP address in a lab. It has collisions. But you can spoof the IP and bypass that.
Ben: [00:13:36] If someone could spoof an IP address, they could get through up to a point, I suppose.
Ben: [00:13:45] I don't know how strict we were on that. We were never hacked, by the way. But then we had another thing. We introduced a SecurID card so that when you logged on, it wasn't just enough to put a name and a password. You also had to have a one-time password.
David: [00:14:08] Oh, wow!
Ben: [00:14:08] I don't know who it was that discovered this little company called SecurID in the US. They make this card. It was very clever. It would time out and all sorts of nice things. So we bought that code and I had to adapt it. What was so interesting was the kernel of it was beautiful code, the actual code that was doing the encryption, the rest was just rubbish code. Just crap.
David: [00:14:36] That's how it goes.
Ben: [00:14:37] Yeah. But anyway, we could adapt it and we lived with it. It was very unpopular with the physicists. First of all, in principle, because they hated the idea that Iranian colleagues couldn't have one of these cards and therefore couldn't have an account on the Cray. There were certain nationalities even then - Iran was on the blacklist - that we weren't allowed to give accounts to, so that the administration of giving out these cards was very strictly done by CERN.
Ben: [00:15:04] The first time CERN ever did restrict things like that. But anyway for CERN it was very strict. And in fact before the Cray came, there was no physical security in the centre. You could just walk into the CERN Computer Centre with your briefcase and put it next to the biggest machine you wanted that would blow up. No problem. It was when the Cray came that the doors were secured and you had to have little badges. Yeah, that was all due to the Cray. That was in the middle of the '80's. And we had a third level ... I don't know what it was... It was quite tight and... But we weren't hacked as far as we know.
David: [00:15:50] Yeah? Well, that's remarkable. Especially as this is such a highly visible target, especially in Europe.
Ben: [00:15:56] Yeah. There've been hacking attempts, successful hacking attempts at CERN, but not on the Cray. One of the very, very first hacking attempts was done on our VAX system with a famous case: it's mentioned in the book called "The Cuckoo's Nest". Have you ever read that? It's mentioned in there. A friend of mine, again on the VAX team, saw funny activity one evening and he followed the guy. And later he made friends with the guy. Online. And they got him because of that finally. He was hacking various systems. That was quite early days. On the Cray we'd have a look. Yeah, they would check, they would make checks on what what people were running, that sort of thing. It was fairly...
Ben: [00:16:49] OK, that was, so that's how Cisco came here. Another story about the Cisco box. So when the Cray left, (probably 1991 when the Cray went, '91, it went before the IBM, '92 probably) the two Cisco boxes were there. And I took them and I put them under my desk because they were my babies, right? And there they stayed for a long time until one day one of my friends from the operations group came to me. "You know, we've got this museum. Yeah. You know, I have a little museum. And can we have, you know, a Cisco router?". So I gave him - it was Cisco A and Cisco B - I gave him Cisco A, and some weeks later I went to see him, David, David Underhill, yeah, "Where's my Cisco box?" "Oh, it was too big to go in the glass cabinet. We're making a new cabinet for it". And then a day after he came in, he said "Ben, I've lost it. When we realised it wouldn't fit in the cabinet, it got taken and it was put too near the recuperation".
David: [00:17:58] Oh, no!
Ben: [00:18:00] It was just taken and thrown away.
David: [00:18:03] It got recuperated.
Ben: [00:18:05] Yes. So the second one was there. I was furious. The second one was there. So the second one, some time later, I gave to the CERN Microcosm (museum), in the days when the Microcosm did have computer stuff and it was there for a long time. Anyway now it's been... I got it before it was taken away from Microcosm. Now it's in the CERN archives storage ready to be put on show again. So that was the Cisco story.
David: [00:18:32] Yeah, I remember that museum. That was impressive. I visited there a decade ago. But there were a number of devices there.
Ben: [00:18:40] Well, it's less good now. There's a little museum in the IT Department. But there's a girl who's supposed to put that up again. They're very slow.
Ben: [00:18:54] So anyway, those were exciting days.
David: [00:18:57] Yeah.
Ben: [00:18:58] And at the end of that then was that I think we were the first to run serious Unix clusters instead of mainframes. Now you asked about Linux. Now, Linux did not come in early. We used proprietary Unix and we built our SHIFT system using different sorts of proprietary Unixes. For pure crunching you could use... HP had a very fast machine with what you didn't need for a disk server. The architecture was that you had compute servers, disk servers and tape servers. The tape servers would do the interface to the tape robots and stage tapes to the disk servers which were physically separate and the compute servers would compute against the disks. And for that you needed a very fast network. We used network attached storage. We didn't use NFS because NFS had a bad name. It wasn't very reliable. It wasn't fast and so on. NFS Version 4 would have been better. And AFS would have been much too slow. So we built our own network attached protocol. It was called RFIO. We just built file connections and all the tape management software that we'd built for the Cray, we just transferred it.
Ben: [00:20:16] So a job could be submitted, arrive on a compute server, give its instructions that it needs these tapes. It was a huge tape management system. A whole cache system, managed cache system on the disk servers. So you could easily check with the particular files you needed: were they there already? If not some of them, you would stage what you needed. And then the job would go into execution all across the network.
Ben: [00:20:44] Now, I was in charge of the network. Now the network is very demanding because .. At first I remember doing the calculations for this: if you have a system... Our target was to build a system - our first system would be three times the capacity of the centre, the existing centre, right. People would say "Three times! You're arrogant, you're stupid, you'll never do that. Your Unix is rubbish...". So we had this little proposal. And so I did my arithmetic. We have a job which is using ... needs to consume such and such an amount of data, and the data has to be first staged from tape, and then computed against, and probably only you know, once,.. it depends. There are different models. And so I figured out what this network had to be capable of. So obviously it wouldn't work with just one Ethernet. So I designed a network that was actually a hybrid network. So there were three levels of it.
Ben: [00:21:45] There was a very, very fast special network called UltraNet, which even had protocal processing on it. From the application it just looked like an ordinary socket. But actually your protocol overhead was done for you and you could do very fast transfers and they had interfaces for Sun, Silicon Graphics, a few other systems, and the Cray of course. So that was expensive, but we needed that for the core network. And then we had a 100 Megabit network - Fast Ethernet wasn't supported at CERN - they had a horrible expensive 100 Megabit network called FDDI. Very expensive. But there was a group in the network team that liked that and pushed it everywhere. So we had to use that for 100 Megabits. And we had ordinary Ethernet that was 10 megabits. Now in the end all that got replaced by Gigabit Ethernet and now faster than that. But at the time that was very advanced for the mid, mid '90s and early '90s because it wouldn't work without a fast network like that. You couldn't do it.
Ben: [00:22:54] So that was a success and so this OPAL experiment were the first people - they were very happy. And then the other experiments, the ones that had said they weren't interested - they came. And by the end of the '90s, everybody was on that system and there were no more mainframes anymore. And then it was at that point that we said, right, now this proprietary stuff is too expensive. Like for a tape server we used the VAX station or something. It's very expensive. Or like SCSI discs. We learned that SCSI disks were just the same as PC disks. It was the same disk unit with an expensive controller. So we didn't realize, we learned all of this. So we decided we were going to go to PC's. So the politicians said to us, "Yes, you can go to PC's, but you must use NT".
David: [00:23:50] Windows NT?
Ben: [00:23:51] Yes, Windows NT.
Ben: [00:23:54] So the smartest, one of the smartest programmers (who was the same guy who is now the Head of the Division, the IT Department), spent almost a year trying to make a system running NT. And he failed.
David: [00:24:09] Well, yeah, I mean it's not... nothing wrong with NT. That is not Unix like ...
Ben: [00:24:13] We needed Unix. But he tried...
Ben: [00:24:16] At that point we were allowed and then we went to Linux. We didn't know a lot about Linux. And again, the management had been quite rude about Linux. I'd maybe told you that the remark was: "Yeah, if there's a bug, you have to fix it yourself".
David: [00:24:29] All right. Yeah.
Ben: [00:24:31] As though, oh, we'd be spending all our time fixing bugs. They didn't know about communities on the network. We did know about that...
Ben: [00:24:36] And we decided we needed to use RAID, RAID technology on the disks, which all the expensive disk servers had. We found you could get a nice PCI interface with RAID on it, as long as you had the driver for Linux, OK. And not only that, but you didn't have to use SCSI. You could use commodity disks, exactly the same mechanically. And it cost a quarter of the price or something. And so, with a PCI interface and a good driver and a RAID interface and these cheap disks, we could solve the disk server problem.
Ben: [00:25:17] The tape problem. Well, the tape interface was some SCSI one, I don't know what it was. So we did all this. And for the network, we needed a Gigabit Ethernet interface. And there wasn't one, OK. So I hired an Internet guru, kernel guru, Danish guy called Jes Sorensen you may have met. And he wrote the first Gigabit Ethernet driver for a very nice Gigabit Ethernet board. A very very nice one. And we kicked off with those, as always. So we were really, you know, pushing on the envelope at that point and we got this thing going. So by the end of the '90s, we were all basically all Linux. And it's never looked back from there.
David: [00:26:03] Yeah, that was my experience in 2005. I came in and it was I don't know about it. It was based on RedHat, perhaps.
Ben: [00:26:11] RedHat. Again, they didn't have the real courage to go Fedora or something. They wanted support, wanted support.
David: [00:26:17] I think it does make sense ultimately - that's RedHat recently bought by I forget who, but RedHat's immensely successful as a company providing support for Linux. It's an example of a win-win situation. Right.
Ben: [00:26:29] So we've got the Linux expertise now. So that was 20 years - no, 25 years - since I would say things like: "We should run Unix on everything" and people would say: "Shut up, you're arrogant". So anyway, that's some of the more recent stuff.
Episode 1 -Who Is Ben Segal
Episode 2 - How to Get Started
Episode 3 -Los Alamos
Episode 4 - Fun with Tapes
Episode 5 - Chasing a Cheap IBM Computer
Episode 6 - California, then and now
Episode 7 - The Importance of Mentors
Episode 6 - California, then and now
Episode 7 - The Importance of Mentors
Episode 8 - Distributed Computing & Meeting Tim Berners-Lee
Episode 9 - Early Battle of Network Protocols After STELLA Project
Episode 10 - Connecting Europe & CERN to the Internet
Episode 11 - The Rocky Beginnings of TCP/IP in Europe