tag:blogger.com,1999:blog-6079517387445025698.post6580031736787208376..comments2024-01-13T17:41:39.533+01:00Comments on Schaffhausen: Datenschutz ist den Usern kein Geld wertBeat Hochheuserhttp://www.blogger.com/profile/00412332728122566133noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-6079517387445025698.post-41505883283391217392015-08-20T16:57:29.191+02:002015-08-20T16:57:29.191+02:00Guter Artikel mit sehr nützlichem Kommentar auch) ...Guter Artikel mit sehr nützlichem Kommentar auch) Vielen Dank den Autoren!IQ Optionhttp://chiefbinaryoptions.com/de/iq-option-erfahrungnoreply@blogger.comtag:blogger.com,1999:blog-6079517387445025698.post-87164061258715566662013-08-06T17:04:22.113+02:002013-08-06T17:04:22.113+02:00Das hatte ich mich auch gefragt, vor allem, da Aco...Das hatte ich mich auch gefragt, vor allem, da Acoon ja vorher anscheinend über 14 Jahre mehr oder weniger ohne Spenden auskam.<br /><br />Die Hardware dürfte nicht all zu teuer gewesen sein (siehe Ausführungen unten). Das Teure war vermutlich der massive Traffic - der wahrscheinlich durch die Prism Debatte nochmals ordentlich zugenommen hatte.<br /><br />Die Page ist ja jetzt vom Netz, aber früher war dort noch ein Acoon Blog vorhanden, in dem folgendes stand:<br /><br />In this article I want to give you an overview about what Acoon currently represents, where the limits are, and how it's done.<br /><br />Current State<br />As of now the search-index of Acoon contains about 350 million web-pages. These contain a total of about 50-60 billion words, and there are 400-500 million different words on these pages.<br />A full search-index takes up about 400gb of space, but it takes about 15-20tb of incoming data to build this index.<br />Acoon runs on only 2 servers. One is the web- and mail-server, and also has the search-index, so all queries run on this server. The other server is responsible for crawling and building the search-index.<br />The crawler is a leased server, has a quad-core 3.2GHz CPU, 32gb RAM and four 2tb harddrives and has a 1gbit/s Internet connection. The crawler is able to handle incoming data at about 400mbit/s, which means a bit over 1,000 pages/s. It takes this machine about a week to build a new search-index.<br />The web-server is actually sitting right here in my home-office, has a quad-core 3.5GHz CPU, 32gb RAM and two 500gb SSDs. This server is connected to the Internet via a relatively cheap 128mbit/s Downstream, 10mbit/s Upstream connection.<br /><br />Current Limits<br />Currently the web-server can handle up to about 5-10 queries/second. Having the index on SSDs instead of regular harddisks is an absolute necessity. Both the higher transfer-rate and the higher-number of random-reads per second that an SSD provides are needed.<br />The software has an internal limit of about 536 million web-pages (2^29 -128 to be exact). More than that would overflow some internal data-structures.<br />While these data-structures would be easy to fix, it wouldn't really help the situation. The queries are still done single-threaded and that causes them to simply take too long in some cases. In practice I have found that 300-350 million pages is the limit before queries take too long to be acceptable.<br />Another limit is the crawler/indexer, especially early during the crawl while it is still collecting new links. The need to check about 50,000 newly-found links per second, and adding them to the URL-database, is slowing down the crawl during its early stages.<br />The parser takes up a lot of CPU-time too and the parser's limit on this server is at about 1,500 pages/second.<br /><br />How It Is Done<br />The entire software is written in Delphi which is a Pascal dialect. It is a surprisingly small piece of software with less than 15,000 lines of source-code.<br />But the software is highly optimized and tailor-made for its task. There is no big database-solution that powers this. No, it is all done explicitly in the software. This is the ONLY way to achieve this much with this little hardware.<br />At the beginning of a crawl the URL-database is seeded with about one million URLs as starting-points for the crawl. These are crawled, parsed for text and links, the links are added to the database, and then the crawl continues on until enough pages have been crawled. The resulting data is not yet searchable. That requires an indexing-step which takes about 8 hours for 350 million pages. It then takes another 10-12 hours to transfer the search-index to the web-server.Beat Hochheuserhttps://www.blogger.com/profile/00412332728122566133noreply@blogger.comtag:blogger.com,1999:blog-6079517387445025698.post-41915409848004990332013-08-06T16:40:54.128+02:002013-08-06T16:40:54.128+02:00Wo er Recht hat, hat er Recht. Aber 1000€ pro Tag ...Wo er Recht hat, hat er Recht. Aber 1000€ pro Tag für den Betrieb? Naja.Anonymousnoreply@blogger.com