NUR NEWS DER SETI@HOME PAGE!!!
ONLY NEWS FROM THE SETI@HOME PAGE!!!
November 14, 2005
Tomorrow we are going to start merging the SETI@home classic master science database with SETI@home/BOINC. This will take several days, during which some back-end servers will be offline (splitters and assimilators). We stored up a large queue of work - we are hoping this will keep all BOINC clients occupied during the length of this outage. If we run out of work, we will have to stop the rest of the back-end servers. After we begin the outage, progress will be detailed in Technical News.
November 15, 2005 - 21:00 UTC
Today we started the big master database merge. This step is simple in essence: we are combining all the scientific data from SETI@home classic and SETI@home/BOINC into one big database. However, this is the culmination of many months of effort.
What happened during those months? Among other things, we had to migrate all the data off of one server onto another, find and remove redundant data, add new fields to old records and populate them, write and test software to merge databases while keeping all relational constraints intact… Basically a lot of cleanup, a lot of testing, and backing up the entire set of databases between every major step.
While this merge is happening, nothing can be updating either of the master databases. We shut off the splitters (which input new workunits into the database) and the assimilators (which input new signals). Over the weekend we created a backlog of about 2 million results, so this should keep the clients well-fed for most of this outage. The assimilator queue, of course, will grow significantly. When everything but the signals themselves have been merged, we may turn the splitters back on (at this point they won’t screw up any relational constraints by adding new work to the mix). When the merge is completely done, everything will be turned back on, and the assimilator queue should quickly drain.
Right now science being done in SETI@home classic is redundant to the science in SETI@home/BOINC, so this will be the last of the big science merges. The classic project will be shut down before the end of the year.
We do hope to eventually place the master science database on a faster machine with bigger/faster disks. This will mean another outage, but it will be a simple unload/reload of the data (as opposed to a unload/reload/correct/merge).
November 15, 2005
The old SETI@home Classic project will stop issuing work on December 15, 2005. Anyone still running SETI@home Classic should deactivate it and install SETI@home/BOINC (see instructions under ‘Getting started’ on this page).
November 16, 2005 - 23:00 UTC
Today we had our usual Wednesday outage to back up the database and upload directories, but also took the opportunity to move some equipment around.
We work closely (and more or less share the same staff) with a seperately funded project that does a survey of hydrogen in our galaxy using SETI@home data. They recently bought a new 3U server (a dual-opteron with 4GB ram and 3TB of SATA drives) which we are in the process of incorporating into our server closet. To do so, we had to move our BOINC upload/download server out of the way. In fact, it had to move out of the cramped closet altogether.
As always, this was no small task, as this server needs a network connection to our private ISP, as well as a connection to the gigabit switch to communicate with the workunit storage server. But our only choice was to move it into an office which had regular old LAN ports and nothing else. In short, we needed to invest in a bunch of long ethernet cables and move some plugs around on the lab’s main switches. But the move went smoothly and everything worked after we powered back up. It’s great when that happens.
Meanwhile, we’re still dealing with the master merge woes from yesterday (see previous post below for more information).
There is a major shell game when merging these two databases, as there is a chain of relational constraints that tie all signals (spikes, Guassians, etc.) to their result, which is tied to a workunit, which is tied to a workunit group, which is tied to a tape. These constraints must be kept intact, even though merging two databases means the ids of all the rows change in the process.
We developed and tested a whole bunch of SQL which did the job, but never tested it on the two tables that contain rows of user-defined type, which in turn contain lists of indefinite size. The Informix SQL engine balked at these, as it should.
Since we have a bunch of C++ code which already does inserting/updating into these tables, Jeff has been busily working today on a fix using C++ instead of SQL. We hope to have this finished and tested and perhaps try again with the master merge tomorrow (after we catch up from today’s outage).
November 22, 2005 - 21:30 UTC
We began sending out the mass e-mail yesterday warning SETI@home classic users that we are going to close down the old project on December 15th. It was sent to all 200,000 of the active classic users by this morning. Inactive classic users are being e-mailed at this point.
Due to the influx of new BOINC users (and the unfortunate timing of some googlebots and other web spiders) the load on our web server was extremely high for the past 12 hours. To fix this, we finally deployed a second web server to split the load. As DNS updates spread throughout the internet, the load on klaatu (the original single web server) decreases while the load on penguin (the new secondary web server) increases. Both are Sun D220R’s (2 x 440MHz Sparc, 2 GB RAM).
Part of the problem was that the web servers were configured to spawn more many clients than actually necessary, which left lingering, unused threads open on the database, which in turn lead to the database running out of connections. Some users saw messages to this effect when the load on the web servers was at its highest.
This looked like a database problem, when in fact we are currently enjoying a 10% performance boost on the database. Last week we moved some memory off the myisam tables (which contain web forum info and not much else) and slated it for the innodb tables (which contain user, host, result, workunit, etc. tables). The myisam tables didn’t need the excess memory.
By the way, the master database merge (see below) is currently on for the beginning of next week.
November 25, 2005
We are getting reports of problems attaching to SETI@home, and are working to resolve them.
If you get stuck in the Proxy Info dialog, make sure you’re connected to the Internet and that BOINC is not blocked by a personal firewall.
If you’re not getting email with your account key, make sure your ISP isn’t blocking email from SETI@home, and try again.
If the BOINC Manager fails to connect, try restarting BOINC (this problem will be fixed in the next BOINC release).
Please go to the Questions and Answers area for help.
November 30, 2005 - 22:30 UTC
So the master database merge is at a complete standstill. Unless everything suddenly works, we probably won’t embark on this adventure until after the December 15th cutoff date for SETI@home Classic. It has become a programming/database nightmare where each fix or workaround brings forth another unexpected show-stopper.
Our server closet is in flux. The SETHI project (which uses SETI@home raw data to study hydrogen in our galaxy) recently bought a new dual opteron system (4GB RAM, 3TB drives) which we wanted to rack up in our closet, but kryten (the BOINC upload/download server) was actually in the way. So we rolled kryten into our secondary lab. But first we had to route a Cogent connection and a link to our internal gigabit switch into that lab. Also in this lab are isaac (the boinc.berkeley.edu web server among other things) and jocelyn (the BOINC database server), which we hope to move in the closet shortly after Classic is shut down.
When this happens, we’ll be able to turn off sagan (the Classic data server) and get it out of the way, so we can remove a set of four A5000 (disk arrays attached to galileo which hold the now-defunct Classic master science database). And all this is just the beginning of what is shaping up to be a large-scale shell game.
We also updated DNS maps and URLs to continue balancing the web load as well as move the BOINC core client downloads off isaac and onto kryten. With the warning e-mails still being sent, the new core client downloads have been peaking out at 40 Mbit/sec. Since isaac, which handles these downloads, is on the campus network, this was adversely affecting others. So we moved all that traffic onto our Cogent link, which is now close to topping out at 100 Mbit/sec at any given time. All BOINC core client downloads, SETI@home science client downloads, SETI@home/BOINC workunits and SETI@home Classic workunits are all going out over our single Cogent connection. Of course, Classic activity will ramp down significantly over the coming weeks, so bandwidth constraints shouldn’t be an issue.
December 9, 2005
We have turned off work distribution for a bit to allow result uploads to catch up. We will be extending the deadline for returning results so that the troubles with the result upload handler will not result in lost credit.
December 13, 2005
We are now successfully recovering from a week of server connection problems. Some connections may still break, but most are getting through at this time. We will be extending the deadline for returning results so that the troubles with the result upload handler will not result in lost credit. More in Technical News.
December 15, 2005
After 6 years of operation, SETI@home Classic sent out its last workunit. We will clean up and synchronize the Classic stats with the current SETI@home project in the coming weeks. Thanks to all SETI@home Classic participants for their tremendous dedication to the project. You made SETI@home into something of lasting significance.
December 27, 2005
We have temporarily turned off the counting of workunits and results in various states (in progress, waiting for validation, etc) in order to give the database cleanup process more resources. Until we turn the counts back on, several of the numbers on the server status page will not be up to date.
January 9, 2006 - 21:30 UTC
There were problems over the weekend that caused internet connections all over campus to disappear at irregular intervals for minutes at a time. We believe this has been resolved.
We had another short outage today to further compress the BOINC database and back it up. Each successive compression has vastly helped the database server throughput. For reasons stated in previous tech news items, the database swelled up to 31GB, all because of results that were clogged in the last few queues of the BOINC backend system. It shrunk to 25GB by January 2, and 19GB by January 5. After today’s compression is was only 13GB.
So the backend servers are much happier. We’re pushing out data at 80Mbits/sec without dropping any connections. We might even be able to turn out server state counts back on once the initial rush clears.
Meanwhile, the master database merge is wrapping up. The spike and gaussian tables are already merged - all that is left are the pulse and triplet tables. We’ll still need to do the final cleanup of results/singals sent since mid-December. The size and scope of that outage (or whether or not we’ll need an outage) has yet to be determined, but it won’t be anything drastic (i.e. probably less than a day, maybe just a few hours).
January 12, 2006 - 18:30 UTC
We quickly wrapped up another database compression/backup this morning. It only took two hours as the database is now at its minimum size given our current operational status. Some extra time was spent rebooting the database server to change a BIOS setting (that was causing harmless but annoying messages to clog the server logs).
We started early as we were hoping there were batteries arriving to replace dead ones in one of our UPS’es, but they didn’t show up this morning. Had they been here we would have shuffled some UPS’es around during this outage. Maybe next time. At any rate, since we are nice and trim and all caught up we shall go back to weekly Wednesday outages for database compression/backup.
It should be noted we are currently processing 1.2 million results a day, which is far above our original hopes of being able to process 1 million results a day. But apparently our servers are pushing their limits, and certain events can trigger a month-long back-end malaise. Future SETI@home applications will have workunits that take longer to process, which will help. We also hope to acquire newer replacement hardware as well at some point. Like a database replica server so we wouldn’t need these weekly outages.
The main part of the master database merge is done. We are now planning the shape and scope of the outage for the final part of the merge. Unlike what was stated in the previous tech news item, it may take longer than a day to do this - we want to make this a “partial” outage (during which users can still upload/download work) so it will take some careful planning to minimize any “full” outage time.
In other fun news: We finally got around to adjusting Classic credit for users that showed obvious signs of cheating. In Classic, it was very easy to cheat the system to get credit without doing any actual work. We ended up partially or entirely removing credit for about 900 of the top 10000 users (all of which had about 20000 or more credits). Below that there wasn’t enough data to show obvious signs of rampant cheating (not to mention enough time and disk space to run the checks on the remaining several million users). These adjusted credits should be sync’ed up with the BOINC databases soon if not already. Case closed.
January 14, 2006
SETI@home Classic users: if you’ve been unable to link your BOINC and Classic accounts (with the error message ‘No SETI@home/BOINC account with Classic ID xxxx was found’) please try again; we fixed a bug in the script.
January 16, 2006
It looks like we may be experiencing local SSL network problems again. If so, this will likely be fixed tomorrow. Today is a university holiday. In the meantime there will be intermittent problems contacting the scheduling server.
February 6, 2006
BOINC lets you mix in other scientific computing with SETI@home. For example, check out SIMAP, a new project from the Technical University of Munich. SIMAP calculate similarities between proteins. It provides a public database of the resulting data, which plays a key role in many bioinformatics research projects.
Danke für den Hinweis, nur sind leider viele andere auch darauf aufmerksam geworden
Die BOINC Download Seite war vorhin nicht erreichbar, aber nun hat es geklappt.
Ein gutes Feature finde ich, dass sich der Client lokal konfigurieren lässt.
Somit ist man nicht auf die Profile (Default, Home, Work, …) des Servers angewiesen sondern kann die Ausführung des Clients lokal steuern.
Nur den Puffer für die WUs muss man am Server einstellen.
